Problem/Motivation

Currently when a queue is suspended with SuspendQueueException in cron, the queue is skipped entirely and the next queue (if any) is processed. The queue is not revisited until the next cron run.

In some cases a worker may be able to determine when the queue can resume processing. It would be helpful for the cron queue runner to revisit the queue when its ready to process again.

Drupal already has \Drupal\Core\Queue\DelayedRequeueException which can currently specify how long to wait, however this is on the individual queue item level. Its not efficient to process all queue items and throw DelayedRequeueException for each.

  • A worker should be able to specify how long in seconds, and less than seconds, when to retry the queue.
  • We should keep the existing behavior where you can resume a queue before it's ready: queues should remain stateless.
  • A suspended queue should be non blocking: continue to process other queues while a queue is suspended.
  • If there are no queues ready to be processed, then sleep until the queue is ready. Except if the queue wont be ready in a timely fashion, see next.
  • A configurable timeout: in most cases we want the cron runner to exit in a timely fashion. If the queue wont be ready for a while, then skip the queue and dont resume until the next cron run.
  • API for suspending a queue for a time should be optional (opt-in), and be inspired by DelayedRequeueException::$delay . Ideally the delay should be specifiable in units less than whole seconds (float). This is helpful when dealing with external API's with per-second limitations (i.e Google).

Remaining tasks

None.

User interface changes

None.

API/Behavior changes

After each cron attempts one run per queue, if any queue threw exceptions, then SuspendQueueException cron will wait an amount of time indicated by the exception. Time is limited by queue_config:suspendMaximumWait service parameter. If the seconds limit is not exceeded then the queue will be retried, otherwise it will be skipped.

Data model changes

New container parameter.

Release notes snippet

Cron is now capable of revisiting a queue in the same run if a queue item when an SuspendQueueException is thrown with a sufficiently low suspension duration.

Issue fork drupal-3198868

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

dpi created an issue. See original summary.

dpi’s picture

Status: Active » Needs review
StatusFileSize
new22.62 KB

Feature and tests.

dpi’s picture

StatusFileSize
new22.62 KB
new456 bytes

Ninja CS fix. 🤺

Status: Needs review » Needs work

The last submitted patch, 3: 3198868-suspend-queue-with-delay-3.patch, failed testing. View results

dpi’s picture

Assigned: dpi » Unassigned
Status: Needs work » Needs review
StatusFileSize
new23.86 KB
new3.66 KB

More fixes

larowlan’s picture

  1. +++ b/core/lib/Drupal/Core/Cron.php
    @@ -166,55 +179,115 @@ class Cron implements CronInterface {
    +    $maxWait = $this->settings->get('queue_suspend_maximum_wait', 30.0);
    +    assert(is_float($maxWait));
    ...
    +      return isset($queueInfo['cron']);
    ...
    +    $queues = array_map(function (array $queue_info) {
    

    nit: mixed variable naming styles here

  2. +++ b/core/lib/Drupal/Core/Cron.php
    @@ -166,55 +179,115 @@ class Cron implements CronInterface {
    +    $queues = array_values($this->queueManager->getDefinitions());
    +    $queues = array_filter($queues, function (array $queueInfo) {
    

    any reason not to do this in one line?

  3. +++ b/core/lib/Drupal/Core/Cron.php
    @@ -166,55 +179,115 @@ class Cron implements CronInterface {
    +        'queue' => $queue,
    +        'worker' => $worker,
    +        'process_from' => $process_from,
    

    note to self, php7.1+ syntax, so fine for 9.2+

  4. +++ b/core/lib/Drupal/Core/Queue/SuspendQueueException.php
    @@ -12,4 +12,54 @@ namespace Drupal\Core\Queue;
    +  public function __construct(?float $delay = NULL, string $message = '', int $code = 0, \Throwable $previous = NULL) {
    +    parent::__construct($message, $code, $previous);
    

    this is a BC change right? previously the first argument was a string, now its a float

    should we put the delay as the fourth argument to maintain BC?

  5. I think we're missing test coverage for the sort-ordering of the different process_from values

Nice work, the execution approach makes sense to me

dpi’s picture

StatusFileSize
new6.63 KB
new28.52 KB

Hey thanks for the feedback @larowlan.

Exception param order fixed in the patch in comment 5, all others addressed in this patch, including the order-of-execution test.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

kim.pepper’s picture

Issue tags: +#pnx-sprint

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

acbramley’s picture

StatusFileSize
new28.49 KB

Re-roll of #7

Status: Needs review » Needs work

The last submitted patch, 11: 3198868-11.patch, failed testing. View results

acbramley’s picture

Status: Needs work » Needs review
StatusFileSize
new28.49 KB
new783 bytes

dpi’s picture

Title: Add delay to queue suspend » 3198868-queue-delay

Fixed conflicts after #3230541: Queue items only reserved by cron for 1 second and created MR.

dpi’s picture

Title: 3198868-queue-delay » Add delay to queue suspend

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

smustgrave’s picture

Status: Needs review » Needs work

Can MR be updated for 10.1.x please.

Also see there was failure for 9.4.x that may need to be addressed.

dpi’s picture

Status: Needs work » Needs review

Back to review w/ new MR on 10.1.x.

Commit 1428a775 contains fixes to tests, which look like may be broken by #3230541: Queue items only reserved by cron for 1 second.

Rajeshreeputra made their first commit to this issue’s fork.

smustgrave’s picture

Status: Needs review » Needs work
Issue tags: +Needs change record

The code looks good. Not sure about the commits afters #23? Were the tests not passing before?

Moving to NW for a change record, which I assume is needed for a change to default.setting file.

smustgrave’s picture

Removing credit for the users who rebased the issue (including myself)

acbramley’s picture

StatusFileSize
new15.56 KB

Reroll against 9.5.x

acbramley’s picture

StatusFileSize
new28.62 KB

Missed the new tests.

acbramley’s picture

Status: Needs work » Needs review
Issue tags: -Needs change record

Draft a CR https://www.drupal.org/node/3343288

Feel free to reword however you see fit, I'm not great at this stuff.

Status: Needs review » Needs work

The last submitted patch, 27: 3198868-27-9.5.x.patch, failed testing. View results

acbramley’s picture

Status: Needs work » Needs review

Latest run is green.

smustgrave’s picture

Status: Needs review » Needs work
Issue tags: +Needs Review Queue Initiative

Seems to be 2 open threads if those could be answered

dpi’s picture

Status: Needs work » Needs review

@catch documentation feedback addressed, switched settings to container parameters per review.

MR needed a bit of attention after #2867001: Dont treat suspending of a queue as erroneous was merged.

Added references to a new change record.

dpi’s picture

Issue summary: View changes
smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

All threads appear to be resolved
Change records is added
Think this is good.

  • catch committed 2d0782d9 on 10.1.x
    Issue #3198868 by dpi, acbramley, larowlan: Add delay to queue suspend
    
catch’s picture

Status: Reviewed & tested by the community » Fixed

Committed 2d0782d and pushed to 10.1.x. Thanks!

dpi’s picture

Thanks @catch 👋

dpi’s picture

Issue summary: View changes

Closing out remaining task. Added optional release note.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

megachriz’s picture

This unfortunately broke the Feeds module. Running queues using a derivative no longer works after this change.

This is how the definition of a Feeds queue can look like:

'feeds_feed_refresh:article' => array (6) [
        'title' => Drupal\Core\StringTranslation\TranslatableMarkup (5) (
            protected 'string' -> string (30) "Feed refresh: @feed_type_label"
            protected 'arguments' -> array (1) [
                '@feed_type_label' => string (7) "Article"
            ]
            protected 'translatedMarkup' -> null
            protected 'options' -> array (0) []
            protected 'stringTranslation' -> null
        )
        'id' => string (18) "feeds_feed_refresh"
        'cron' => array (1) [
            'time' => integer 60
        ]
        'deriver' => string (46) "Drupal\feeds\Plugin\Derivative\FeedQueueWorker"
        'class' => string (43) "Drupal\feeds\Plugin\QueueWorker\FeedRefresh"
        'provider' => string (5) "feeds"
    ]

Note that the key is different from the 'id'. The id is the base plugin ID and the key is the derivative plugin ID.

megachriz’s picture

For the issue mentioned in the comment above I created a new issue: #3348832: Running cron queues that use a derivative is broken