Add an 'instant' queue runner

We don't need to use 1px gifs, between these two we can issue http requests from within the request out to the Drupal site then bail without checking the result, which takes around 1ms.

https://github.com/biznickman/PHP-Async

http://drupal.org/project/httprl

Tried updating the issue summary.

I think the first thing to do here would be to implement something which just helps to run-off a few queue items in-between cron jobs, for #2878119: Whether queued or not, update the media thumbnail and metadata before beginning the entity save database transaction and similar issues.

So:

- Add an item to the queue that we want processing asap

- Add a session flag with the queue name we want to run off.

- Each request, look for that session flag, and if it's there, render a 1px gif

- The 1px gif links to a URL with the queue name and a token. Configuration/settings sets the number of items to process (default to 5 or 10?) or do it a bit like batch where we do as many as we can within a second or two.

@dawehner a new module sounds good to me.

iirc that only works on fastcgi, see #2851111: fastcgi_finish_request and shutdown functions (e.g. batch) do not play nicely together and similar issues.

Given the queue API can't provide feedback, we can skip the browser altogether now that Fibers is available: #3257726: [meta] Use Fibers for concurrency in core

It can't, and I got overexcited opening this issue before fully reading through the docs. If we were to try to use it for this, it would end up needing to be something like Guzzle's non-blocking HTTP request (which might get refactored on top of a Fiber). Fibers don't allow for simultaneous execution by themselves.

@andypost amp or reactphp might help on the cli, but this requires running a daemon waiting for queue items to come in, which isn't going to be an option on a lot of hosting environments. That's more what https://www.drupal.org/project/advancedqueue and similar are trying to do.

What I am thinking about here is a module we can ship with core, similar to automated_cron, that will run some queues down in the browser. It is not as good as a waiting queue, but it would allow us to do things like add functionality like https://www.drupal.org/project/image_style_warmer to core.

When we originally added automated cron to core, one of the ideas was to add a 1px gif following POST requests. This was rejected at the time because it was not guaranteed to run often enough on a site without much authenticated activity, but I think it would be enough here since we're explicitly hoping to execute queue items created by things like saving an entity form.

After #3295790: Post-response task running (destructable services) are actually blocking; add test coverage and warn for common misconfiguration there's a possible path forwards here. I think we could add it to automated_cron

1. Automated cron adds a decorator for the queue service, this keeps a record of any queue that has an item added during a request.

2. That same decorator implements DestructableInterface and runs some (configurable via container parameter?) number of queue items from the list of queues it has at the end of the request - so could do do 1 item each from five queues or 5 items from one queue or similar.

We cant guarantee that the queue items that were added will be the ones that were created during the request, but some queue items will be processed and anything that doesn't gets picked up by the next cron run.

This would allow us to fix long-standing issues like #504012: Use a queue for node create/update indexing, that issue could continue to mark the node for reindex, but also add a queue item that will reindex it when it runs, then automated cron picks that queue item up and reindexes it at the end of the request.

It would also open up the possibility to bring the functionality of https://www.drupal.org/project/image_style_warmer and similar modules into core - we just create the queue item and if you have automated cron it might handle it, but if also it would work if you have a dedicated permanent queue runner or frequent drush jobs just for running queues too.

@sukr_s when we originally implemented automated cron in core, we looked at making asynchronous http requests to a Drupal route, but eventually gave up on it because some hosting environments do not cleanly support making http requests to themselves. This is why automated_cron uses the terminate event, to run after the response has been sent to the browser but before script termination. The post response logic works a lot more consistently after #3295790: Post-response task running (destructable services) are actually blocking; add test coverage and warn for common misconfiguration.

So I would just directly process the queue items in the terminate event listener without trying to deal with async http requests. This also avoid having to worry about an access-free route for processing the queue items, which we'd probably have to lock down with a token or similar.

- Min number of items to trigger the queue. Setting to zero will disable instant queue processing
- Max number of concurrent queue processing

Why not a maximum number of queue items to process, and if there's one or more queue item added during the request, process them at the end? If the maximum number is 0, nothing ever gets processed. I don't see what a minimum number of items gets us.

With this suggestion, I'm assuming that processing in the event listener will be non-blocking, otherwise async call is better. Will go with your suggestion for now.

Yes it's non-blocking - the terminate event runs after the response is sent to the browser. Prior to Drupal 10.2-ish this only worked on certain server configurations but since the issue linked above it works on nearly all.

Do you mean not to process more than the maximum number of items in the queue at a time. I was thinking of setting the time limit to zero so that all the items in the queue will be processed. Otherwise in each terminate event we will have to check the remaining number of items in the queue and trigger again, which would mean additional db calls in the terminate event.

The queue could potentially have thousands of items in it, that could lead to OOM errors etc if we tried to process everything. The way I thought of this working was that we would track which queue and how many items are added during a request, probably in a class property of the decorator or similar.

Say someone submits a node form, and that adds one queue item to two different queues, A and B, then in the terminate event, we'd process one item from queue A and one item from queue B. Obviously there's no guarantee that these are the same queue items at all, but this feature is mostly intended for lower traffic sites where that's more likely to be the case.

However if submitting the node form added 500 items to the queue (e.g. it adds queue item for every person following an issue or something), we'd only process the configured maximum of items then stop (anything else would have to eventually be picked up by cron, which also processes queue items).

Left some comments/questions on the MR.

Took a detailed look at the MR, I think there is still a lot of complexity here that is probably not needed and noticed some more things:

1. For search_api instant indexing + https://www.drupal.org/project/image_style_warmer they might want to add their items to the instant queue runner for sites that don't necessarily have automated_cron enabled.

2. Adding all queue items to the instant runner in automated cron might be heavy-handed, not all queue items benefit from being processed immediately.

3. I don't think the concurrency protection is necessary, this only impacts requests that actually add items to the queue (and less likely to happen if we don't do #2 by default).

4. The trait feels a bit too high level, I think extracting the code that runs a single queue item would be more re-usable, and allow dropping the semaphore usage.

These are fairly large changes from the current approach so going to start a new MR here.

Not sure we need to do the automatic cron integration here, that could be its own issue - add an extra configuration key for whether to instant-process queue items and ideally we'd find a way to decorate all the queue backends without having to add a subclass for each one.

In its current state, the new MR should allow for search_api modules and potentially core search to add items to the queue runner themselves so they get processed end of request.

No test coverage yet, since this depends on kernel destruct it needs at least a kernel test or maybe functional test, would be easiest with an existing queue (test or real).

Added a kernel test.

Hiding the other branch for now.

Adding the search issues.

I tried to think of any existing core queues that could use this, but couldn't really yet. However search_api and image cache prewarming are very obvious use cases in contrib.

#3316209: Leakage usernames (timing attack) from password reset form is a candidate for using this API and also a security improvement.