This issue is triggered by #814990: node_mass_update() can't rely on a browser session, #556022: When a text format is deleted, some modules (like text.module) don't update their data, even though we say they do and #89181: Use queue API for node and comment, user, node multiple deletes, two critical of them currently marked critical.

I suggest we add support for a standard "worker queue" in the system module. This working queue will provide a way for modules to easily register jobs without having to register their own queue.

This should be as simple as the system module implementing hook_cron_queue_info().

To ease implementation of long-running bulk operations (like the ones identified above), we should also provide a way for tasks to define themselves as "immortal", and immortal task being re-registered automatically when it finishes.

CommentFileSizeAuthor
#1 job.patch3.6 KBchx
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

chx’s picture

Status: Active » Needs review
FileSize
3.6 KB

That's not a complicated thing to do.

aspilicious’s picture

srry chx

+  /**
+   * Add a simple job to the queue.
+   *

==> Adds

sun’s picture

Status: Needs review » Needs work

warning: n00b review follows. I'm not familiar with the queue system.

+++ includes/common.inc	2010-06-19 23:26:27 +0000
@@ -4632,10 +4632,11 @@ function drupal_cron_run() {
-    $queue = DrupalQueue::get($queue_name);
+    $queue = DrupalQueue::get($queue_name, !empty($info['reliable']));

Is the 'reliable' property documented already?

+++ modules/system/system.module	2010-06-19 23:26:10 +0000
@@ -2845,6 +2845,24 @@ function system_cron() {
+function system_cron_queue_info() {
+  $queues['system_workers'] = array(
...
+  );
+}

1) The internal name sounds a bit fuzzy... 'system_simple' or similar might be a better name?

2) Missing return $queues;

+++ modules/system/system.module	2010-06-19 23:26:10 +0000
@@ -2845,6 +2845,24 @@ function system_cron() {
+ * Simple job queue worker.
+ */
+function system_queue_worker($item) {
+  return call_user_func_array($item->data['callback'], $item->data['arguments']);

Guess it would be a good idea to document the special return value behavior as function description here.

Powered by Dreditor.

yched’s picture

That's possibly where the interesting convergence between Queues and Batch API starts :-).

Batch API processing is notably irrelevant for long-running processes triggered by API functions, since it requires a UI context and a client browser to loop on the requests. Having a generic way for any module / function to dump a series of jobs for automated 'later' execution is a great step.

I've been wondering around this idea for a while : does the future of batch API lie in being a mere UI consumer of queued jobs, that can be consumed otherwise in other ways (cron, CLI/drush ...) ?

On top of the system this patch puts in place, it could be great to have a message of some sort, 'there are pending jobs on the site, you can wait for cron to execute them progressively, or click this [link to Batch progress page] to execute them right now'.

Then Batch API primary use case is consuming the system_worker queue - in addition to a couple other business specific cases (hook_update_N, possibly update.module...)

/end patch-derail mode

As for the immediate step in this patch, my only remark is that addJob() looks a bit weird inside the DrupalQueue factory class - it's a business function relevant to one single specific queue ('system_workers'), can't it be a regular flat function instead of a static method ?

moshe weitzman’s picture

subscribe. we *really* need this

Anonymous’s picture

oh, hell yes.

i'd love to see something like waiting_queue in core, or at least, breaking the queue-cron coupling we have now.

neclimdul’s picture

You know this has been dead in the water for a while now but moshe is right. we really need this. And hopefully ysched is right too and this is where batchapi can start becoming sane. So here's a little bump post and subscribe.

Lars Toomre’s picture

I would very much like to see this concept implemented. I have written a private deferred-action processing implementation and would be happy to help implement any merged Batch API/Queue functionality. One of the features that I would like to see implemented via this functionality is the regular execution of what I refer to as 'check' actions (such as checking for orphaned field data or files).

franz’s picture

Although this was not ready here, it was already used inside #89181: Use queue API for node and comment, user, node multiple deletes - that kind of confused me.

moshe weitzman’s picture

Status: Needs work » Closed (duplicate)