This issue is triggered by #814990: node_mass_update() can't rely on a browser session, #556022: When a text format is deleted, some modules (like text.module) don't update their data, even though we say they do and #89181: Use queue API for node and comment, user, node multiple deletes, two critical of them currently marked critical.
I suggest we add support for a standard "worker queue" in the system module. This working queue will provide a way for modules to easily register jobs without having to register their own queue.
This should be as simple as the system module implementing hook_cron_queue_info().
To ease implementation of long-running bulk operations (like the ones identified above), we should also provide a way for tasks to define themselves as "immortal", and immortal task being re-registered automatically when it finishes.
Comments
Comment #1
chx CreditAttribution: chx commentedThat's not a complicated thing to do.
Comment #2
aspilicious CreditAttribution: aspilicious commentedsrry chx
==> Adds
Comment #3
sunwarning: n00b review follows. I'm not familiar with the queue system.
Is the 'reliable' property documented already?
1) The internal name sounds a bit fuzzy... 'system_simple' or similar might be a better name?
2) Missing
return $queues;
Guess it would be a good idea to document the special return value behavior as function description here.
Powered by Dreditor.
Comment #4
yched CreditAttribution: yched commentedThat's possibly where the interesting convergence between Queues and Batch API starts :-).
Batch API processing is notably irrelevant for long-running processes triggered by API functions, since it requires a UI context and a client browser to loop on the requests. Having a generic way for any module / function to dump a series of jobs for automated 'later' execution is a great step.
I've been wondering around this idea for a while : does the future of batch API lie in being a mere UI consumer of queued jobs, that can be consumed otherwise in other ways (cron, CLI/drush ...) ?
On top of the system this patch puts in place, it could be great to have a message of some sort, 'there are pending jobs on the site, you can wait for cron to execute them progressively, or click this [link to Batch progress page] to execute them right now'.
Then Batch API primary use case is consuming the system_worker queue - in addition to a couple other business specific cases (hook_update_N, possibly update.module...)
/end patch-derail mode
As for the immediate step in this patch, my only remark is that addJob() looks a bit weird inside the DrupalQueue factory class - it's a business function relevant to one single specific queue ('system_workers'), can't it be a regular flat function instead of a static method ?
Comment #5
moshe weitzman CreditAttribution: moshe weitzman commentedsubscribe. we *really* need this
Comment #6
Anonymous (not verified) CreditAttribution: Anonymous commentedoh, hell yes.
i'd love to see something like waiting_queue in core, or at least, breaking the queue-cron coupling we have now.
Comment #7
neclimdulYou know this has been dead in the water for a while now but moshe is right. we really need this. And hopefully ysched is right too and this is where batchapi can start becoming sane. So here's a little bump post and subscribe.
Comment #8
Lars Toomre CreditAttribution: Lars Toomre commentedI would very much like to see this concept implemented. I have written a private deferred-action processing implementation and would be happy to help implement any merged Batch API/Queue functionality. One of the features that I would like to see implemented via this functionality is the regular execution of what I refer to as 'check' actions (such as checking for orphaned field data or files).
Comment #9
franzAlthough this was not ready here, it was already used inside #89181: Use queue API for node and comment, user, node multiple deletes - that kind of confused me.
Comment #10
moshe weitzman CreditAttribution: moshe weitzman commentedLets call this a dupe of #89181: Use queue API for node and comment, user, node multiple deletes