Problem/Motivation

Drupal has two ways to handle long-running tasks:

1. Queue API, by default queues are run on cron, often sites run queues via jenkins on different schedules, there are projects like 'waiting queue' which aim to minimise the time before queue items are picked up.

2. Batch API - designed for the browser, although drush can also run batches, completely blocks the user's interaction with the site while it's running. Batches are user-specific so if one gets interrupted, that's pretty much it. Provides feedback, you know when it's finished.

From personal experience, the Queue API is great to work with and expectations are very clear. Batch API is hard to work with and debug.

In the UI, Batch API always blocking isn't necessarily optimal. Bulk operations on entities for example could be put into a queue, and there could be a progress bar in a block somewhere which runs down the queue, but allows the user to click around the rest of the site while that's happening. There are cases though like updates from the UI where things really do need to block and require strict ordering.

Proposed resolution

Add a browser queue runner. Will need a route that takes a queue name, fetches items and runs them (will need to set the current user to anonymous to match drush/cron) then reports back. Then a proof of concept UI widget with some feedback that's activated by a $_SESSION flag. Not sure what the non-js fallback would be, although a 1px gif might be enough.

Remaining tasks

User interface changes

API changes

Data model changes

Cron jobs fall into a couple of main categories:

- things that have to be run periodically and don't care about site events - mainly garbage collection like session and cache.
- batch processing of things - search indexing of new nodes, purging of deleted stuff.

For the latter case, these are increasingly moving to the queue API, although it's not 100% consistent in core.

Issues like #943772: field_delete_field() and others fail for inactive fields, and the one I can't find about indexing nodes for searching immediately, might be helped by a poor mans queue runner.

Drupal 7 has a poor mans cron. Currently the implementation is very basic - 1px gif/xhr requests were causing Drupal to be bootstrapped twice each request, and at one point there was a proposal to do ob_flush() during a shutdown function but this didn't take on, so we ended up just running cron inline instead, which is sucky but I argued for that in the end.

With the queue runner, it'd be more a case of setting $_SESSION['run_queues'] after a form submit, check that on the next page, if it's set, add the 1px gif or whatever to that page, which hits /queue_run with a token. This would only ever be triggered by form submissions so it'd not have the page caching issues of cron runs.

Things it could be useful for:

- field deletion
- mass deletes of other stuff
- operations that trigger menu rebuilds or similar expensive operations, that don't necessarily have to happen inline with the request - just very shortly afterwards.
- indexing nodes in the core search module immediately after posting instead of waiting for cron.

Comments

catch’s picture

Issue tags: +Performance

We don't need to use 1px gifs, between these two we can issue http requests from within the request out to the Drupal site then bail without checking the result, which takes around 1ms.

https://github.com/biznickman/PHP-Async

http://drupal.org/project/httprl

Berdir’s picture

tsphethean’s picture

So I've started having a think about this, but for the async in Guzzle we'll need to add the Async plugin to core (http://guzzlephp.org/guide/plugins.html#async-plugin) - will that be a problem, and is too late in the cycle to get this in before feature freeze?

Was also thinking this queue runner would need to be able to be disabled (in the performance section or elsewhere?) as some use cases may want to prevent anything running "randomly"?

For implementation, I was thinking of extending hook_cron_queue_info() to add a flag for whether the queue specified should be executed by this queue runner. There is hook_queue_info which has been exposed by the contrib Queue UI module (http://drupal.org/project/queue_ui) but I guess getting that into core is pushing it?

Berdir’s picture

I don't think that is a problem at all, the issue linked above in fact already adds that plugin.

tsphethean’s picture

Ok great, so for working on it I'll apply the patch from #1599622: Run poormanscron via a non-blocking HTTP request and we can make that a dependency.

I think I was editing my post as you replied, do you think identifying the queues to be run in queue runner should be configured in hook_cron_queue_info()?

tsphethean’s picture

One further thought on this, why would we only want to spawn these queue runner processes from a form submit? Looking at how the poormanscron will work (and works in D7) we'll be spawning a request to /cron/%process every time we reach the cron processing threshold, is there any reason why we wouldn't want to do the same for queues?

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

andypost’s picture

Version: 8.1.x-dev » 8.2.x-dev
Related issues: +#1599622: Run poormanscron via a non-blocking HTTP request

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

catch’s picture

Title: Add a 'poor mans queue runner' to core » Add a browser-based queue runner
Component: base system » ajax system
Issue summary: View changes
Related issues: +#1797526: Make batch a wrapper around the queue system

Tried updating the issue summary.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.