Problem/Motivation
Currently when a queue is suspended with SuspendQueueException in cron, the queue is skipped entirely and the next queue (if any) is processed. The queue is not revisited until the next cron run.
In some cases a worker may be able to determine when the queue can resume processing. It would be helpful for the cron queue runner to revisit the queue when its ready to process again.
Drupal already has \Drupal\Core\Queue\DelayedRequeueException which can currently specify how long to wait, however this is on the individual queue item level. Its not efficient to process all queue items and throw DelayedRequeueException for each.
- A worker should be able to specify how long in seconds, and less than seconds, when to retry the queue.
- We should keep the existing behavior where you can resume a queue before it's ready: queues should remain stateless.
- A suspended queue should be non blocking: continue to process other queues while a queue is suspended.
- If there are no queues ready to be processed, then sleep until the queue is ready. Except if the queue wont be ready in a timely fashion, see next.
- A configurable timeout: in most cases we want the cron runner to exit in a timely fashion. If the queue wont be ready for a while, then skip the queue and dont resume until the next cron run.
- API for suspending a queue for a time should be optional (opt-in), and be inspired by
DelayedRequeueException::$delay. Ideally the delay should be specifiable in units less than whole seconds (float). This is helpful when dealing with external API's with per-second limitations (i.e Google).
Remaining tasks
None.
User interface changes
None.
API/Behavior changes
After each cron attempts one run per queue, if any queue threw exceptions, then SuspendQueueException cron will wait an amount of time indicated by the exception. Time is limited by queue_config:suspendMaximumWait service parameter. If the seconds limit is not exceeded then the queue will be retried, otherwise it will be skipped.
Data model changes
New container parameter.
Release notes snippet
Cron is now capable of revisiting a queue in the same run if a queue item when an SuspendQueueException is thrown with a sufficiently low suspension duration.
| Comment | File | Size | Author |
|---|---|---|---|
| #27 | 3198868-27-9.5.x.patch | 28.62 KB | acbramley |
| #26 | 3198868-26-9.5.x.patch | 15.56 KB | acbramley |
| #13 | interdiff-3198868-11-13.txt | 783 bytes | acbramley |
| #13 | 3198868-13.patch | 28.49 KB | acbramley |
Issue fork drupal-3198868
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
dpiFeature and tests.
Comment #3
dpiNinja CS fix. 🤺
Comment #5
dpiMore fixes
Comment #6
larowlannit: mixed variable naming styles here
any reason not to do this in one line?
note to self, php7.1+ syntax, so fine for 9.2+
this is a BC change right? previously the first argument was a string, now its a float
should we put the delay as the fourth argument to maintain BC?
Nice work, the execution approach makes sense to me
Comment #7
dpiHey thanks for the feedback @larowlan.
Exception param order fixed in the patch in comment 5, all others addressed in this patch, including the order-of-execution test.
Comment #9
kim.pepperComment #11
acbramley commentedRe-roll of #7
Comment #13
acbramley commentedComment #15
dpiFixed conflicts after #3230541: Queue items only reserved by cron for 1 second and created MR.
Comment #16
dpiComment #19
smustgrave commentedCan MR be updated for 10.1.x please.
Also see there was failure for 9.4.x that may need to be addressed.
Comment #22
dpiBack to review w/ new MR on 10.1.x.
Commit
1428a775contains fixes to tests, which look like may be broken by #3230541: Queue items only reserved by cron for 1 second.Comment #24
smustgrave commentedThe code looks good. Not sure about the commits afters #23? Were the tests not passing before?
Moving to NW for a change record, which I assume is needed for a change to default.setting file.
Comment #25
smustgrave commentedRemoving credit for the users who rebased the issue (including myself)
Comment #26
acbramley commentedReroll against 9.5.x
Comment #27
acbramley commentedMissed the new tests.
Comment #28
acbramley commentedDraft a CR https://www.drupal.org/node/3343288
Feel free to reword however you see fit, I'm not great at this stuff.
Comment #30
acbramley commentedLatest run is green.
Comment #31
smustgrave commentedSeems to be 2 open threads if those could be answered
Comment #32
dpi@catch documentation feedback addressed, switched settings to container parameters per review.
MR needed a bit of attention after #2867001: Dont treat suspending of a queue as erroneous was merged.
Added references to a new change record.
Comment #33
dpiComment #34
smustgrave commentedAll threads appear to be resolved
Change records is added
Think this is good.
Comment #36
catchCommitted 2d0782d and pushed to 10.1.x. Thanks!
Comment #37
dpiThanks @catch 👋
Comment #38
dpiClosing out remaining task. Added optional release note.
Comment #40
megachrizThis unfortunately broke the Feeds module. Running queues using a derivative no longer works after this change.
This is how the definition of a Feeds queue can look like:
Note that the key is different from the 'id'. The id is the base plugin ID and the key is the derivative plugin ID.
Comment #41
megachrizFor the issue mentioned in the comment above I created a new issue: #3348832: Running cron queues that use a derivative is broken