Refactor QueueFactory to get queue backend service from QueueWorker plugin instead of Settings

current active development branch is 8.4.x-dev

Are there specific usecases you have in mind which are not possible right now? I think it would be conceptually wrong to have to couple queue plugins to concrete queues. IMHO separating the configuration step from the queue plugin itself is exactly how it was designed to work.

Isn't that the reason why you can have different queue configurations? 'queue_service_' . $name

While I can understand the desire to decouple workers from the backend service. I've also encountered a need to use a custom queue implementation (extending DatabaseQueue). Managing this through settings.php adds some additional integration requirements to other sites using this queue that I don't think should be necessary.

As long as an explicit backend_service is not required I think this would be useful.

It would still be nice to have a issue summary which describes WHY this feature is needed / helpful.

I've updated the IS summary with a concrete example, which for my case is process heavy actions on orders, but that need to be reliable run if an order is about to be processed.

In the meantime, I suggest look at/using https://www.drupal.org/project/advancedqueue
It's the Queue API rewrite that we needed to do for our needs in Commerce.

I can second @malcom_p's use case. I recently ran into the same issue of needing to extend DatabaseQueue for a module I'm currently writing, but being unable use the queue service in the module to make use of it.

Introduced the first version of the refactoring. Reviews are welcome.

Changing target branch to 10.0.x.

I'm gonna remove settings service from arguments in #3299828: Stop storing Settings singleton in object properties

Rerolled against 10.1.x

Moving to NW for the open threads and suggestions made.

Needs review

This issue is being reviewed by the kind folks in Slack, #needs-review-queue-initiative. We are working to keep the size of Needs Review queue [2700+ issues] to around 400 (1 month or less), following Review a patch or merge request as a guide.

Code wise everything looks good
New functions and properties are typehinted
trigger_errors are call where needed

Only issue I see is that the link the trigger_error calls should point a change record that shuld be added to this ticket.

Other then that everything looks good

Good work!

Added CR https://www.drupal.org/node/3338022 , updated links in PR

Thanks! Can you also include in the CR that the QueueFactory now requires QueueWorkerManagerInterface.

Would search for other CR where parameters are added for wording suggestions.

Updated CR.

Think that should be good.

Left some comments on the MR.

So instead of having hardcoded variable in settings.php I propose a new attribute is added to the QueueWorker annotation class backend_service. This would default to the current queue.database service if no value would be provided.

The issue summary mentions an annotation class, but that's not how the merge request is solving the problem. Is there a way to use an annotation? Or should the solution continue down the path of a member variable/constant/function?

The definition is the annotation and the MR updates the annotation.

This would need to update the attribute class now as well.

I'm not really convinced this is a good idea. I see that there are sometimes use cases where you implement a custom queue plugin and want to use a specific storage at the same time and that currently requires an extra line in settings.php, but I'd argue it's at least as common to use a certain backend for a queue plugin that you didn't implement yourself and this now requires a module with custom code, that might need its own set of environment-specific control to only do that on certain environments. Maybe we support both without deprecating the existing approach, not really a reason to not support both? The cache backend is similar in that cache bins can have a default backend that can be overridden.

Settings.php overrides are unsuitable for the modules that try to control the backend of the queue worker.
I like the idea of keeping the current approach to override the backend service.

What if, instead of extending attribute/annotation, we introduce an extension point by dispatching an event right after determining the service name?

Updated CR

I still have reservations about this as I said in #38. IMHO, this should only add a new way to configure this and not deprecate the existing settings key.

I understand that there is a use case that specific queues want to default to a specific backend. But it's also a site/infrastructure level question, even for specific bins to move them to a different backend.

IMHO, this should mirror the logic in \Drupal\Core\Cache\CacheFactory::get:

bin/queue setting > bin/queue specific default > global setting > database fallback.

@berdir thoughts on the latest changes?

I still do not understand the need from the IS:

A concrete example of where this would be useful is for queue items that process orders (e.g. in commerce). [...] The obvious way to do this is to use a queue backend that stores the order ID in a separate field that can be used to query/filter other queue backend methods (e.g. claimItem).

The code to place an item surely uses a specific queue name and does not look it up based on the queue worker, does it? If that's so then decorate the queue service and return your own backend based on the queue name. The responsibility of the worker is to process an item, why does it care where it came from?

At the moment, there is no way to provide a different backend for a specific queue other than setting it in settings.php. I attempted to implement the UI for such a configuration, but ended up with no way to alter those values in the dedicated module implementation. QueueWorkerManager aggregates definitions and also allows definition alteration in info hook implementations. I thought the QueueWorker definition could provide a way to switch the selected backend if a different backend is not set for the queue in settings.php.

At the moment, there is no way to provide a different backend for a specific queue other than setting it in settings.php.

This is true. I recommend using a queue service decorator for the purpose. Is that not adequate?

Yes, it is an option. There is a Queue Order project that provides decoration for QueueWorkerManager for a different reason, so I think decorating QueueFactory should do the backend override.

Most likely will need test coverage too.