Problem/Motivation

PostgreSQL 15 added the native MERGE statement. The Merge query of the Database API does not use it. The generic \Drupal\Core\Database\Query\Merge emulates a merge with a SELECT followed by an INSERT or an UPDATE. That is two round trips per merge, and the two-step approach is open to race conditions between the check and the write.

The minimum requuired version for Drupal 12 is PostgreSQL 18, so the driver can rely on MERGE (PostgreSQL 15) and RETURNING merge_action() (PostgreSQL 17) being available.

Core runs merge queries in three places: \Drupal\Core\KeyValueStore\DatabaseStorage::setIfNotExists() on the key_value table, \Drupal\locale\StringDatabaseStorage::updateLocation() on the locales_location table when translatable strings are saved with new locations, and TaxonomyHooks::buildNodeIndex() on the taxonomy_index table, once per term, when a published node is saved. Contrib modules also use the Merge query for insert-or-update writes, such as counters and log records.

Proposed resolution

Add a driver-specific merge implementation to the PostgreSQL driver:

  • New class \Drupal\pgsql\Driver\Database\pgsql\Merge, returned by the driver's Connection::merge(). It executes the merge as a single native MERGE statement with a constant single-row source relation (USING (SELECT 1)).
  • The statement ends with RETURNING merge_action(). From that value, execute() returns Merge::STATUS_INSERT, Merge::STATUS_UPDATE, or NULL when the row existed and no update was requested.
  • The generic emulation of the parent class is kept for queries that a native MERGE cannot express or would not serve:
    • the condition targets a table other than the merge table, so no ON clause can be built;
    • there are no fields to insert, so no WHEN NOT MATCHED THEN INSERT clause can be generated;
    • values are inserted into serial fields, because the generic insert path synchronizes the sequences afterwards.

Performance

Benchmark with a standalone Database API script: 9 rounds of 5000 merge queries on a table with a varchar primary key and two data columns, median round time. Environment: PostgreSQL 18.4 and PHP 8.5 in separate ddev (Docker) containers on one machine. The native MERGE runs one statement per merge, the generic emulation runs two (SELECT plus INSERT or UPDATE). A smaller run (7 rounds of 1000 merges) gave the same picture: 16% to 20% less time per query.

Workload Generic emulation Native MERGE Change
merge-update (row exists) 173 µs/query 148 µs/query -14%
merge-insert (row is new) 158 µs/query 129 µs/query -18%

These numbers are from a local, low-latency connection between web and database containers. The saving is one network round trip per merge, so the gain grows with the latency between the web server and the database server.

Remaining tasks

  • Code review of the merge request.
  • Decide if the CI change stays in this issue or moves to its own issue.
  • Benchmark on a setup with network latency between the web and database servers.

User interface changes

None.

API changes

None. The new Merge class is internal to the PostgreSQL driver and keeps the return values of \Drupal\Core\Database\Query\Merge::execute().

Data model changes

None.

Release notes snippet

On PostgreSQL, merge queries now execute as a single native MERGE statement instead of a SELECT followed by an INSERT or an UPDATE.

For the committer

The changes to the .gitlab-ci.yml file need to be removed before merging!

Issue fork drupal-3315265

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

andypost created an issue. See original summary.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

andypost’s picture

The MR is used to test new images so will be moved to separate issue

andypost’s picture

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.

daffie changed the visibility of the branch 3315265-improve-support-of to hidden.

daffie changed the visibility of the branch 7.x to hidden.

daffie changed the visibility of the branch 10.2.x to hidden.

daffie changed the visibility of the branch 11.x to hidden.

daffie’s picture

Issue summary: View changes

Updated the IS.

daffie’s picture

Title: Improve support of native MERGE on postgresql 15 » Improve support of native MERGE with RETURNING merge_action()
daffie’s picture

Status: Active » Needs review

Ready for a review.
The CI pipeline failures are not related to the changes in the PR.

smustgrave’s picture

Status: Needs review » Needs work

Small comments on the MR

daffie’s picture

Issue summary: View changes
daffie’s picture

Added a CR and updated the IS for removing the changes to .gtilab-ci.yml.

smustgrave’s picture

Status: Needs work » Reviewed & tested by the community

Nice CR!

Believe all feedback has been addressed here.

daffie’s picture

Added the requested return type hint. Not sure if it needs to go back to NR.

daffie’s picture

Disclosure: I have used AI on the PR, IS and CR.

longwave made their first commit to this issue’s fork.

longwave’s picture

Status: Reviewed & tested by the community » Fixed

Committed and pushed 2c9c31c0448 to main. Thanks!

Also published the change record.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

  • longwave committed 2c9c31c0 on main
    refactor: #3315265 Improve support of native MERGE with RETURNING...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.