Problem/Motivation

DatabaseQueue::claimItem() claims an item with two queries. A SELECT finds the oldest unclaimed item. An UPDATE with an expire = 0 condition then claims it. Concurrent consumers select the same row. Only one UPDATE succeeds. The other consumers retry the loop. With N parallel consumers, N claims cost about N² queries. Claim latency grows with the number of consumers.

Proposed resolution

On PostgreSQL the claim can be a single race-free statement:

UPDATE {queue} SET expire = :expire
WHERE item_id = (
  SELECT item_id FROM {queue}
  WHERE name = :name AND expire = 0
  ORDER BY created, item_id
  FOR UPDATE SKIP LOCKED
  LIMIT 1)
RETURNING data, created, item_id;

Concurrent consumers skip rows locked by other consumers and each claim a different item. There is no retry loop. A single consumer saves one query per claimed item.

This needs a way for the pgsql driver to provide its own queue implementation, or per-driver logic in DatabaseQueue.

Steps to reproduce

Run several queue consumers in parallel on one queue. Log the queries. Consumers select the same head item and retry after failed updates.

Remaining tasks

Create a merge request. Add a concurrency test. Review.

API changes

None. QueueInterface is unchanged.

Data model changes

None.

For the committer

The changes to the .gitlab-ci.yml file need to be removed before merging!

Issue fork drupal-3615202

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

daffie created an issue. See original summary.

daffie’s picture

Issue summary: View changes

daffie’s picture

Issue summary: View changes
daffie’s picture

Status: Active » Needs review

Ready for a review.

smustgrave’s picture

Status: Needs review » Needs work

1 small change requested and 1 question about a CR.

daffie’s picture

Status: Needs work » Needs review

All remarks of @smustgrave have been addressed.
Bank to NR.

@smustgrave: Thank you for your review.

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Thanks for humoring the suggestions. Feedback looks to be addressed. No additional feedback.

daffie’s picture

Disclosure: I have used AI on the PR, the IS and the CR.