Problem/Motivation
Found via #3377657: Add database query spans to otel traces.
Drupal uses the flood system for user logins.
The database flood backend lazy-creates the database table (using the same pattern as cache tables) which is good.
However, it only does this on writes, not selects.
For cache this is fine because cache misses usually result in a cache write, however a successful login only ever queries the flood table, it never writes to it. This means the flood table stays un-created on new sites until someone gets their password wrong, meaning schema queries against the flood table to check if it's there or not. On logins with the standard profile, this happens once in Flood::isAllowed() and twice after a successful login.
At some point someone will get their password wrong, or the flood table will be used for something else, but until then it's extra database queries every user login.
Steps to reproduce
Proposed resolution
Create the flood database table in Flood::isAllowed() too.
Remaining tasks
User interface changes
API changes
Data model changes
Release notes snippet
Before:

After:

| Comment | File | Size | Author |
|---|---|---|---|
| #8 | Screenshot from 2023-12-21 17-25-49.png | 427.45 KB | catch |
| #7 | Screenshot from 2023-12-21 17-24-01.png | 445.01 KB | catch |
| #6 | Screenshot from 2023-12-21 17-04-46.png | 337.75 KB | catch |
| #6 | Screenshot from 2023-12-21 17-04-09.png | 299.01 KB | catch |
| #5 | Screenshot from 2023-12-21 16-54-26.png | 1.03 MB | catch |
Issue fork drupal-3410312
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3410312-flood-database-backend
changes, plain diff MR !5921
Comments
Comment #3
catchComment #4
catchUnless something very flukey is happening, which I am not ruling out yet, this reduces a core MR test run to 7m48s from ~10m.
Comment #5
catchConfirmed that this is removing three queries against information_schema when running performance tests.
I haven't yet confirmed that we're running this query every time we log in a user during tests - if the installer queries flood at all, then it will, if not, it might only affect second or third logins (which the performance tests do).
Comment #6
catchModified StandardPerformanceTest::testLogin() to remove the cache warming.
This shows multiple information_schema queries on the first login in 11.x:

vs. with the MR, a single information_schema query, then a create table, then 'normal' flood queries after that:

The CREATE TABLE takes 46ms on my local machine, but a possible explanation for the faster test runs is that because gitlab is on a ramdisk, table creation is cheaper than running all those extra database queries. Or we're getting lucky with the runners today and it's not a full minute's improvement, but it shows that the performance tests are correctly diagnosing a bug!
Even though the first flood query in a test appears to be the first login, this is still potentially saving test time, because after a successful login, we also DELETE any flood entries for that user (one for the uid, one for the uid + IP if that's also configured), whether or not there are any - so every login until the flood table is created actually results in multiple queries to information_schema. I'm counting three, which will go down to one (plus the table creation).
Comment #7
catchBefore image had the wrong queries expanded, so updating the issue summary.
Comment #8
catchBetter 'before' screenshot too.
Comment #9
catchComment #10
catchComment #11
fjgarlin commentedI've reviewed the code and checked all the previous comments with performance information. It makes total sense to create the table the first time we query it and don't find it, rather than catching exceptions all the time and not creating the table.
We can see performance gains in the run too, which is a nice side effect too.
Marking RTBC.
Comment #12
mondrakeGreat find!
Personally, I find on-the-fly table creation a rather bad approach. If we had the table created at installation, we wouldn’t have to care about this. On-the-fly table creation breaks transactions for DBs that do not support transactional DDL - not just the transaction where ensureTable is called, also any concurrent transaction from other sessions. But I suppose this train is lost now :(
Comment #13
catch@mondrake we've got a few more options now - for example services that need to create tables could subscribe to a module installed event and create the storage then. The main reason for on-the-fly table creation originally was to avoid creating database tables from hook_schema() for services that were using alternative storage - like multiple cache tables when a site was using redis or memcache. Worth a new issue to discuss.
Comment #14
mondrakeYeah… I was also thinking events would help in #3397622: Adding GIN and GIST indexes to PostgreSQL databases, to allow database drivers to decorate db objects specifications with db specific features. I will open a new issue to discuss.
Comment #15
catchNot sure whether it's enabled by this issue or test runs have been slowly getting faster, but new tests we can mark with @group #slow which are overhanging the end of the test runs #3410396: Add @group slow to ForumTest, HelpTopicSearchTest, ModulesListFormWebTest.
Comment #18
larowlanCommitted to 11.x and backported to 10.2.x
Nice one!
Comment #20
catchThis narrowed the query range too much, opened #3410450: Fix random performance test failures. Ironically although I found this looking at performance test query spans, the assertions only count successful queries - if a query triggers an exception, the 'after' event doesn't get fired, so we don't see a strong decrease in the number of queries in the test coverage itself even though less are happening in reality. It might be possible to check if the start and end events match and log some kind of error if they don't to cover that case too.
Comment #21
mondrakeWe may want to add another statement execution event on top of the start/end ones, e.g.
StatementExecutionFailureEvent, so that exception handling may dispatch them.Comment #22
mondrakeFiled #3410476: Add event for statement execution failure.
Comment #23
mondrakeAlso filed #3410480: [META] Use events in Database Schema operations re #13/#14
Comment #25
catch