Problem/Motivation

Now that #3313355: Allow the database query log to be dispatched as log events has laid down a framework to dispatch events from the Database API, it could be interesting to think about adding transaction-related events too (begin transaction, savepoint, rollback, commit, maybe transaction exceptions).

See #3313355-18: Allow the database query log to be dispatched as log events.

Proposed resolution

  • add 11 transaction-related events:
    • 2 related to transaction initiation:
      Drupal\Core\Database\Event\TransactionBeginEvent
      Drupal\Core\Database\Event\TransactionSavepointEvent
    • 4 related to transaction commit/release/rollback:
      Drupal\Core\Database\Event\TransactionCommitEvent
      Drupal\Core\Database\Event\TransactionReleaseSavepointEvent
      Drupal\Core\Database\Event\TransactionRollbackEvent
      Drupal\Core\Database\Event\TransactionRollbackToSavepointEvent
    • 1 related to transaction auto-commit:
      Drupal\Core\Database\Event\TransactionAutoCommitEvent
      
    • 4 related to failure of transaction commit/release/rollback:
      Drupal\Core\Database\Event\TransactionCommitFailureEvent
      Drupal\Core\Database\Event\TransactionReleaseSavepointFailureEvent
      Drupal\Core\Database\Event\TransactionRollbackFailureEvent
      Drupal\Core\Database\Event\TransactionRollbackToSavepointFailureEvent
  • do not dispatch the events unless they're enabled through the Connection::enableEvents() mechanism already in place for statement-related events
  • add unit and kernel testing
  • BONUS - fix inconsistencies in the TransactionManager behavior and stack identified as part of checking test results, especially for the cases of failure of commit/release/rollback operations (they are edge cases anyway, but now we can inspect the state of the stack and the client transaction state in these cases, so why not)

Remaining tasks

User interface changes

None.

API changes

Data model changes

None.

Release notes snippet

Issue fork drupal-3348590

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

mondrake created an issue. See original summary.

mfb’s picture

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

mondrake’s picture

mondrake’s picture

Assigned: Unassigned » mondrake

On this

mondrake’s picture

Assigned: mondrake » Unassigned
Status: Active » Needs review

Started something, fit nicely with latest improvements in #3364706: Refactor transactions. Looking forward to an early review - this sets the basis, more events are just more meat around this bone.

mondrake’s picture

Title: Add transaction-related events to the Database API » [PP-1] Add transaction-related events to the Database API
Status: Needs review » Postponed
Related issues: +#3384960: Strengthen TransactionManager
daffie’s picture

Title: [PP-1] Add transaction-related events to the Database API » Add transaction-related events to the Database API
Status: Postponed » Needs work
mondrake’s picture

rebased

mondrake’s picture

Status: Needs work » Postponed

We cannot do this now, #3405976: Transaction autocommit during shutdown relies on unreliable object destruction order (xdebug 3.3+ enabled) introduced Database::commitAllOnShutdown() which runs at a time when the event dispatcher has been destroyed already, so events fail. One more reason to move towards explicit commit.

mfb’s picture

eesh, seems not-ideal to wait until most-everything is destroyed to commit transactions, i.e. difficult to respond to various transaction commit failure modes.

mondrake’s picture

mondrake’s picture

Title: Add transaction-related events to the Database API » [PP-1] Add transaction-related events to the Database API
Issue summary: View changes
Parent issue: #3364706: Refactor transactions » #3495728: [meta] Drop implicit autocommit on Transaction destruction
Related issues: -#3495728: [meta] Drop implicit autocommit on Transaction destruction +#3364706: Refactor transactions
mondrake’s picture

Title: [PP-1] Add transaction-related events to the Database API » [PP-2] Add transaction-related events to the Database API
Issue summary: View changes
Related issues: +#3406985: Convert all transactions in core to use explicit ::commitOrRelease()
mondrake’s picture

Title: [PP-2] Add transaction-related events to the Database API » Add transaction-related events to the Database API
Issue summary: View changes
Status: Postponed » Needs review
Issue tags: +Needs tests

As long as we can accept that these events are not dispatched when the event dispatcher service is not available - that is: 1) early in the request before the full container is available, and 2) during shutdown when the service can have already been destructed - this can proceed.

Still needs some tests for commit/release/rollback failure events - not easy to create the context - so I will look into mocking.

But the biggest part here is already reviewable.

mondrake’s picture

Assigned: Unassigned » mondrake
Status: Needs review » Needs work

Working on the unit tests.

mondrake’s picture

Issue summary: View changes
mondrake’s picture

Issue summary: View changes
Status: Needs work » Needs review
Issue tags: -Needs tests
mondrake’s picture

Assigned: mondrake » Unassigned
Issue summary: View changes
mondrake’s picture

Issue summary: View changes
mondrake’s picture

Status: Needs review » Needs work

I think we need to add an additional event for the autocommit on DDL, separate from the regular one, and still emit the regular one in the expected moment even if the autocommit already happened. This will make simpler moving post-transaction callbacks to event subscriptions.

AFK for a while, will come back later.

mondrake’s picture

Issue summary: View changes
Status: Needs work » Needs review
dcam’s picture

Status: Needs review » Needs work

I did my best to review the MR. Unfortunately, I don't know enough about the database system to review it well from a technical perspective. My comments on the MR amount almost entirely to nitpicks on documentation. I'm sorry about that. For what it's worth, I did read through all the code change and it looks good to me.

dcam’s picture

Issue tags: +Needs change record

I nearly forgot: since there are three new methods on TransactionManagerInterface we'll need a change record, won't we?

mondrake’s picture

Status: Needs work » Needs review

Thanks for review @dcam

mondrake’s picture

Yes, a CR will be needed here. Let’s do that once we are close to finalize this issue - I’ve seen too many CR written too early that eventually were not reflecting the avtual changes.

nicxvan’s picture

Just curious why these are not hooks?

mfb’s picture

Dispatching an event is faster than invoking a hook (and uses less memory), so it doesn't make sense for low level code like the database layer to invoke extra hooks, especially in the context of performance analysis.

mondrake’s picture

#30 + I also fancy one day we will be able to have events dispatched well before the container and modules are available (#3492391: Make the event dispatcher available before container full bootstrap), which will enable moving post-transaction callback execution to an event subscriber and deprecate execution on Transaction destruction (issue tbd).

See parent plan issue.

mfb’s picture

And I wouldn't mind having some new events for instrumenting how slow the hook system is 🙃

nicxvan’s picture

Dispatching an event is faster than invoking a hook (and uses less memory), so it doesn't make sense for low level code like the database layer to invoke extra hooks, especially in the context of performance analysis.

Do you have a source for that? I don't see how that is possible.

We just decoupled hooks from events because of the performance problems. #3506930: Separate hooks from events

Event Dispatcher does not scale well, I think a hook here is better served.

mondrake’s picture

Also, we already have events developed for statement execution, so the transaction ones here just extend that framework.

jamyotte changed the visibility of the branch 3348590-add-transaction-related-events to hidden.

mondrake’s picture

@jamyotte is there a reason for hiding the MR?

mondrake changed the visibility of the branch 3348590-add-transaction-related-events to active.

mfb’s picture

Do you have a source for that? I don't see how that is possible.

In my testing, Drupal::moduleHandler()->invokeAll() is slower than Drupal::service('event_dispatcher')->dispatch()

Both the first call, and the second call are slower.

Events are of course heavily used in low-level code, for example, request events/event subscribers. So that seems like the obvious building block we'd use to allow for instrumenting various low-level subsystems. And we already are using as mondrake mentioned.

nicxvan’s picture

Sorry to keep belaboring this:

In my testing, Drupal::moduleHandler()->invokeAll() is slower than Drupal::service('event_dispatcher')->dispatch()

Which hook did you invoke, how many implementations were there, what did that hook do?
Which event did you dispatch, how many events were implemented, and what did those events do?

Again I'm asking because this does not align at all with the numbers we saw. Removing event dispatcher from hooks showed the following improvements courtesy of @berdir:

Time: 15% (5.69 ms → 4.82 ms)
IO: 18% (2.05 ms → 1.69 ms)
CPU: 14% (3.64 ms → 3.13 ms)
peak memory: 23% (2.68 MB → 2.07 MB)
network transfer: 29% (0.914 MB → 652 kB)

If you share how you tested this I can try to replicate your findings.

mondrake’s picture

IMHO #39 could use a separate discussion and issue.

mfb’s picture

I'm not sure how to dynamically register a hook implementation so that we can make a nice little test script, but we can add an event listener and dispatch an event, and then in comparison, invoke a hook that doesn't exist:

$bytes = memory_get_usage();
$nanoseconds = hrtime(TRUE);
Drupal::service('event_dispatcher')->addListener('my event', fn () => print 'EVENT: ' . PHP_EOL);
Drupal::service('event_dispatcher')->dispatch(new stdClass(), 'my event');
echo (hrtime(TRUE) - $nanoseconds) / 1000, ' microseconds', PHP_EOL;
echo memory_get_usage() - $bytes, ' bytes', PHP_EOL;

echo 'HOOK: ', PHP_EOL;
$bytes = memory_get_usage();
$nanoseconds = hrtime(TRUE);
Drupal::moduleHandler()->invokeAll('not actually a hook');
echo (hrtime(TRUE) - $nanoseconds) / 1000, ' microseconds', PHP_EOL;
echo memory_get_usage() - $bytes, ' bytes', PHP_EOL;

mfb’s picture

Looks like moduleHandler()->invokeAll() can make database queries - cache gets - which I can't imagine you'd want to do as part of dispatching events inside the database layer.

mondrake’s picture

Version: 11.x-dev » main
smustgrave’s picture

This is a very large MR, difficult to review I'd imagine (not sure I'm one to do that) hence leaving in review. But wanted to ask if could be broken up?

mondrake’s picture

#44 it's hard for me to think about splitting this now, 3 years after initial commits - I see it has multiple aspects, but the journey to get to it was one that while developing the events I figured out the underlying improvements to transaction handling edge cases, and now I would find difficult to phase them.

mondrake’s picture

Status: Needs review » Needs work
Related issues: +#3584322: Cleanup transactions in try/catch blocks

#3584322: Cleanup transactions in try/catch blocks shows that we need also to work on managing failures of BEGIN TRANSACTION and SAVEPOINT statement executions.

mondrake’s picture

Status: Needs work » Postponed
Related issues: +#3584465: Fix inconsistencies in the TransactionManager behavior

I've opened #3584465: Fix inconsistencies in the TransactionManager behavior to deal with fixes that are IMHO needed independently from here (the issue here helped find the issues, but those can be fixed without events...). That also addresses #44. Hope this helps.

mondrake’s picture

Status: Postponed » Needs review