It must be possible to vary delay between job attempts based on schedule.

For example, if a task fails, 1st retry should be done after 10 seconds, 2nd one after 1 minute, 3rd one after 5 minutes, etc.

The patch adds support for specifying "retry_delay_schedule" for example as:

  'retry_delay_schedule' => '5,10,60,300',

(illustrating the scenario described above).

The same time delay could also be used multiple times - for example:

  'retry_delay_schedule' => '5*2,10*4,60*5,300',

would mean that:

  • first 2 retries will be done after 5 second delay each
  • next 4 retries will be done after 10 second delay each
  • next 5 retries will be done after 60 second delay each
  • all the remaining retries will be done after 5 minute delay each

If max attempts value is higher than number of defined retry_delay_schedule delays, the last delay value will be used for all the remaining attempts.

The change is 100% backwards compatible, and using 'retry_delay' => 5 still works as before. retry_delay_schedule takes precedence over retry_delay.

Comments

maciej.zgadzaj created an issue. See original summary.

maciej.zgadzaj’s picture

Status: Active » Needs review
StatusFileSize
new3.41 KB
maciej.zgadzaj’s picture

StatusFileSize
new3.39 KB

Re-rolled the patch for the most recent module version.

zaporylie’s picture

Title: Add support for varying "retry after" time delays » Vary delays between attempts based on schedule
Version: 7.x-1.x-dev » 8.x-1.x-dev
Issue summary: View changes
StatusFileSize
new5.1 KB

I've used #3 and rerolled it to be compatible with 8.x-1.x. I believe having a configurable schedule will be a great addition to this module

We use advanced queue quite extensively with Commerce for export jobs and quite often we find ourselves in the situation where we'd like to increase the interval between attempts. Ex. run the first 3 attempts 2 minutes apart, then increase the time between attempts to 30 minutes for another 3 attempts, then increase it to 4h for the remaining attempts.

The code is missing tests but it should already be working if one wants to test it.

jonathanshaw’s picture

Exponential backoff is a best practice. I wonder if we should implement it by default rather than making it configurable. That keeps the codebase simpler and more maintainable.