Problem/Motivation
DatabaseQueue::claimItem() claims an item with two queries. A SELECT finds the oldest unclaimed item. An UPDATE with an expire = 0 condition then claims it. Concurrent consumers select the same row. Only one UPDATE succeeds. The other consumers retry the loop. With N parallel consumers, N claims cost about N² queries. Claim latency grows with the number of consumers.
Proposed resolution
On PostgreSQL the claim can be a single race-free statement:
UPDATE {queue} SET expire = :expire WHERE item_id = ( SELECT item_id FROM {queue} WHERE name = :name AND expire = 0 ORDER BY created, item_id FOR UPDATE SKIP LOCKED LIMIT 1) RETURNING data, created, item_id;
Concurrent consumers skip rows locked by other consumers and each claim a different item. There is no retry loop. A single consumer saves one query per claimed item.
This needs a way for the pgsql driver to provide its own queue implementation, or per-driver logic in DatabaseQueue.
Steps to reproduce
Run several queue consumers in parallel on one queue. Log the queries. Consumers select the same head item and retry after failed updates.
Remaining tasks
Create a merge request. Add a concurrency test. Review.
API changes
None. QueueInterface is unchanged.
Data model changes
None.
For the committer
The changes to the .gitlab-ci.yml file need to be removed before merging!
Issue fork drupal-3615202
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
daffie commentedComment #4
daffie commentedComment #5
daffie commentedReady for a review.
Comment #6
smustgrave commented1 small change requested and 1 question about a CR.
Comment #7
daffie commentedAll remarks of @smustgrave have been addressed.
Bank to NR.
@smustgrave: Thank you for your review.
Comment #8
smustgrave commentedThanks for humoring the suggestions. Feedback looks to be addressed. No additional feedback.
Comment #9
daffie commentedDisclosure: I have used AI on the PR, the IS and the CR.
Comment #11
longwaveAdded a question about the service override.
Comment #12
daffie commentedUnfortunately using a backend-override service does not work. And after looking into the problem with AI, there are a lot of problems with backend overridable services. When we created the tag "backend_override" the way we used services was kind of basic. Now we are doing a lot of special stuff like adding more tags, module Alter(), lazy services. They all have problems with backend overridable services. I have created #3621344: [META] Fix backend overridable services.. We already have #3021299: Ensure that aliased/used backend overridable are not set to private and #3461330: Lazy services (backed by proxy classes) can't be `backend_overridable` because the proxy service isn't tagged.
Comment #13
longwaveThe problem is that anyone enabling pgsql will have the queue service overridden by a pgsql version - but they might not be using pgsql for their main Drupal database.
I've worked on a project that has more than one database driver enabled because I was migrating content from a SQL Server source to Drupal running in MariaDB, so I imagine this is not totally uncommon.
Comment #14
daffie commentedA different solution to override the default queue.database service, and only when the current database driver is PostgreSQL.
Comment #15
ghost of drupal pastWork related note: this query should work as is with MariaDB since the very recent 13.0 (I haven't tested but it should). Perhaps the installer should try the query on the default connection wrapped with try-catch and write $settings['queue_modern_query'] = TRUE; if it succeeded instead of using it on PostgreSQL only?
Source: https://mariadb.com/docs/server/reference/sql-statements/data-manipulati...
(Same source and target support was added in 10.3.2, SELECT ... SKIP LOCKED is available since 10.6, much longer than the brand new UPDATE...RETURNING support.)