Problem/Motivation
DatabaseQueue::claimItem() claims an item with two queries. A SELECT finds the oldest unclaimed item. An UPDATE with an expire = 0 condition then claims it. Concurrent consumers select the same row. Only one UPDATE succeeds. The other consumers retry the loop. With N parallel consumers, N claims cost about N² queries. Claim latency grows with the number of consumers.
Proposed resolution
On PostgreSQL the claim can be a single race-free statement:
UPDATE {queue} SET expire = :expire WHERE item_id = ( SELECT item_id FROM {queue} WHERE name = :name AND expire = 0 ORDER BY created, item_id FOR UPDATE SKIP LOCKED LIMIT 1) RETURNING data, created, item_id;
Concurrent consumers skip rows locked by other consumers and each claim a different item. There is no retry loop. A single consumer saves one query per claimed item.
This needs a way for the pgsql driver to provide its own queue implementation, or per-driver logic in DatabaseQueue.
Steps to reproduce
Run several queue consumers in parallel on one queue. Log the queries. Consumers select the same head item and retry after failed updates.
Remaining tasks
Create a merge request. Add a concurrency test. Review.
API changes
None. QueueInterface is unchanged.
Data model changes
None.
For the committer
The changes to the .gitlab-ci.yml file need to be removed before merging!
Issue fork drupal-3615202
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
daffie commentedComment #4
daffie commentedComment #5
daffie commentedReady for a review.
Comment #6
smustgrave commented1 small change requested and 1 question about a CR.
Comment #7
daffie commentedAll remarks of @smustgrave have been addressed.
Bank to NR.
@smustgrave: Thank you for your review.
Comment #8
smustgrave commentedThanks for humoring the suggestions. Feedback looks to be addressed. No additional feedback.
Comment #9
daffie commentedDisclosure: I have used AI on the PR, the IS and the CR.