Problem/Motivation

DatabaseQueue::claimItem() claims an item with two queries. A SELECT finds the oldest unclaimed item. An UPDATE with an expire = 0 condition then claims it. Concurrent consumers select the same row. Only one UPDATE succeeds. The other consumers retry the loop. With N parallel consumers, N claims cost about N² queries. Claim latency grows with the number of consumers.

Proposed resolution

On PostgreSQL the claim can be a single race-free statement:

UPDATE {queue} SET expire = :expire
WHERE item_id = (
  SELECT item_id FROM {queue}
  WHERE name = :name AND expire = 0
  ORDER BY created, item_id
  FOR UPDATE SKIP LOCKED
  LIMIT 1)
RETURNING data, created, item_id;

Concurrent consumers skip rows locked by other consumers and each claim a different item. There is no retry loop. A single consumer saves one query per claimed item.

This needs a way for the pgsql driver to provide its own queue implementation, or per-driver logic in DatabaseQueue.

Steps to reproduce

Run several queue consumers in parallel on one queue. Log the queries. Consumers select the same head item and retry after failed updates.

Remaining tasks

Create a merge request. Add a concurrency test. Review.

API changes

None. QueueInterface is unchanged.

Data model changes

None.

For the committer

The changes to the .gitlab-ci.yml file need to be removed before merging!

Issue fork drupal-3615202

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

daffie created an issue. See original summary.

daffie’s picture

Issue summary: View changes

daffie’s picture

Issue summary: View changes
daffie’s picture

Status: Active » Needs review

Ready for a review.

smustgrave’s picture

Status: Needs review » Needs work

1 small change requested and 1 question about a CR.

daffie’s picture

Status: Needs work » Needs review

All remarks of @smustgrave have been addressed.
Bank to NR.

@smustgrave: Thank you for your review.

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Thanks for humoring the suggestions. Feedback looks to be addressed. No additional feedback.

daffie’s picture

Disclosure: I have used AI on the PR, the IS and the CR.

longwave made their first commit to this issue’s fork.

longwave’s picture

Status: Reviewed & tested by the community » Needs review

Added a question about the service override.

daffie’s picture

Unfortunately using a backend-override service does not work. And after looking into the problem with AI, there are a lot of problems with backend overridable services. When we created the tag "backend_override" the way we used services was kind of basic. Now we are doing a lot of special stuff like adding more tags, module Alter(), lazy services. They all have problems with backend overridable services. I have created #3621344: [META] Fix backend overridable services.. We already have #3021299: Ensure that aliased/used backend overridable are not set to private and #3461330: Lazy services (backed by proxy classes) can't be `backend_overridable` because the proxy service isn't tagged.

longwave’s picture

The problem is that anyone enabling pgsql will have the queue service overridden by a pgsql version - but they might not be using pgsql for their main Drupal database.

I've worked on a project that has more than one database driver enabled because I was migrating content from a SQL Server source to Drupal running in MariaDB, so I imagine this is not totally uncommon.

daffie’s picture

Status: Reviewed & tested by the community » Needs review

A different solution to override the default queue.database service, and only when the current database driver is PostgreSQL.

ghost of drupal past’s picture

Work related note: this query should work as is with MariaDB since the very recent 13.0 (I haven't tested but it should). Perhaps the installer should try the query on the default connection wrapped with try-catch and write $settings['queue_modern_query'] = TRUE; if it succeeded instead of using it on PostgreSQL only?

Source: https://mariadb.com/docs/server/reference/sql-statements/data-manipulati...

The RETURNING clause is available from MariaDB 13.0.
...
UPDATE statements may have the same source and target.

(Same source and target support was added in 10.3.2, SELECT ... SKIP LOCKED is available since 10.6, much longer than the brand new UPDATE...RETURNING support.)