Problem/Motivation

Here's an example from \Drupal\Core\Batch\BatchStorage::ensureTableExists()

  protected function ensureTableExists() {
    try {
      $database_schema = $this->connection->schema();
      if (!$database_schema->tableExists(static::TABLE_NAME)) {
        $schema_definition = $this->schemaDefinition();
        $database_schema->createTable(static::TABLE_NAME, $schema_definition);
        return TRUE;
      }
    }
    // If another process has already created the batch table, attempting to
    // recreate it will throw an exception. In this case just catch the
    // exception and do nothing.
    catch (DatabaseException $e) {
      return TRUE;
    }
    return FALSE;
  }

The only way we can return FALSE is if the table exists! Which is completely wrong according to the docs and how things are supposed to work. We return false if $database_schema->tableExists(static::TABLE_NAME) returns TRUE!

This causes problems if a table is being creating by an ajax request and there several hitting the server at the same time. We get races and this code then fails to protect us.

Steps to reproduce

Proposed resolution

Change the method to do this:

  protected function ensureTableExists() {
    try {
      $database_schema = $this->connection->schema();
      $schema_definition = $this->schemaDefinition();
      $database_schema->createTable(static::TABLE_NAME, $schema_definition);
    }
    // If another process has already created the batch table, attempting to
    // recreate it will throw an exception. In this case just catch the
    // exception and do nothing.
    catch (DatabaseException $e) {
    }
    catch (\Exception $e) {
      return FALSE;
    }
    return TRUE;
  }

Table existence is already checked in $database_schema->createTable() and if that throws an exception... it is caught and handled. The caller is rethrowing the initial exception in \Drupal\Core\Batch\BatchStorage::create().

Remaining tasks

Fix:

  • \Drupal\Core\Batch\BatchStorage::ensureTableExists()
  • \Drupal\Core\Cache\DatabaseCacheTagsChecksum::ensureTableExists()
  • \Drupal\Core\Flood\DatabaseBackend::ensureTableExists()
  • \Drupal\Core\Lock\DatabaseLockBackend::ensureTableExists()
  • \Drupal\Core\Queue\DatabaseQueue::ensureTableExists()
  • \Drupal\Core\Config\DatabaseStorage::ensureTableExists()
  • \Drupal\Core\KeyValueStore\DatabaseStorage::ensureTableExists()
  • \Drupal\Core\Routing\MatcherDumper::ensureTableExists()
  • \Drupal\Core\Menu\MenuTreeStorage::ensureTableExists()

User interface changes

API changes

The method Drupal\Core\Cache\DatabaseCacheTagsChecksum::catchException() is deprecated.

Data model changes

Release notes snippet

CommentFileSizeAuthor
#23 3240601-23.patch17.67 KBalexpott
#23 20-23-interdiff.txt3.78 KBalexpott
#20 3240601-20.patch15.84 KBalexpott
#20 15-20-interdiff.txt502 bytesalexpott
#18 3240601-18.patch18.64 KBdaffie
#17 3240601-17.patch18.64 KBdaffie
#17 interdiff-3240601-15-17.txt4.48 KBdaffie
#15 3240601-15.patch16.05 KBalexpott
#15 3240601-15.test-only.patch3.01 KBalexpott
#15 13-15-interdiff.txt3.15 KBalexpott
#13 3240601-13.patch15.73 KBalexpott
#13 3240601-13.test-only.patch2.89 KBalexpott
#13 7-13-interdiff.txt3.82 KBalexpott
#12 3240601-7.patch11.96 KBalexpott
#10 3240601+3240813-9.2.x.patch12.3 KBalexpott
#8 3240601-9.2.x-8.patch11.95 KBdaniel.bosen
#7 3240601-7.patch11.96 KBalexpott
#7 5-7-interdiff.txt2.64 KBalexpott
#5 3240601-5.patch11.62 KBalexpott
#5 4-5-interdiff.txt461 bytesalexpott
#4 3240601-4.patch11.56 KBalexpott
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

alexpott created an issue. See original summary.

alexpott’s picture

@dbosen found this issue while working on broken javascript tests in Thunder.

alexpott’s picture

Status: Active » Needs review
FileSize
11.56 KB

Here's the fix. It might be nice to put this into a trait and somehow notify modules that have copy&pasted this code.

alexpott’s picture

catch’s picture

Very nice find.

  1. +++ b/core/lib/Drupal/Core/KeyValueStore/DatabaseStorage.php
    @@ -255,17 +255,17 @@ public function deleteAll() {
         }
         // If the table already exists, then attempting to recreate it will throw an
         // exception. In this case just catch the exception and do nothing.
    +    // @todo Should this be DatabaseException like the others?
         catch (SchemaObjectExistsException $e) {
    -      return TRUE;
         }
    

    I think it should, although you could also argue that all the others should use SchemaObjectExistsException? But either way we should use one consistently.

  2. +++ b/core/lib/Drupal/Core/Menu/MenuTreeStorage.php
    @@ -247,6 +247,7 @@ protected function safeExecuteSelect(SelectInterface $query) {
           // Some other failure that we can not recover from.
    +      // @todo this is actually unreachable code - should we fix this.
           throw $e;
         }
    

    Probably just in a follow-up.

It might be nice to put this into a trait and somehow notify modules that have copy&pasted this code.

There are two issues trying to do something like this already:

#2721271: Simplify exception handling for lazy-load pattern and #2371709: Move the on-demand-table creation into the database API.

alexpott’s picture

Thanks for the review @catch.

Re #6.1 I think we should go for DatabaseException. There's a chance of a race in \Drupal\Core\Database\Schema::createTable() between the table existence check and executing the query. If the query fails we won;t change the exception to a SchemaObjectExistsException - it'll be a normal DatabaseException.

Re #6.2 Well the changes we make here make it completely impossible whereas before it was only unlikely. So I think we need to address it here. I've moved the throws from ::ensureTableExists() up to the caller to maintain the API of ::ensureTableExists always returning a TRUE or FALSE regardless of errors and giving the caller a chance to emit the original exception and at the same time maintaining the current exception classes.

I think we should leave the trait / further API stuff to #2371709: Move the on-demand-table creation into the database API

daniel.bosen’s picture

To be able to use this in our tests where we initially ran into the problem, I backported it to 9.2.x

alexpott’s picture

Fixing issue credit... @dbosen is not @daniel.bosen

alexpott’s picture

Here's a patch that combines this with #3240813: \Drupal\Core\KeyValueStore\DatabaseStorage::catchException() fails due to races for testing on 9.2.x with both patches... so Thunder can see if the combined patch fixes the issues.

daffie’s picture

  1. The impression that I get when I look at the patch is that we are doing the same thing or almost the same thing about 9 times in core. Maybe we should move the code to a trait (https://en.wikipedia.org/wiki/Don%27t_repeat_yourself).
  2. I am missing testing for this change/fix.

@alexpott: Good find!

alexpott’s picture

Issue tags: -Needs tests
FileSize
11.96 KB

@daffie see earlier comments re trait / common way of doing it... i.e. #6 / #7.

Re testing - this will be exceptionally hard as this is caused by race. So it'll have to be forks and all sorts... I'll have a look to see what is possible. IN the meantime will remove the needs tests since I'm not sure that it is possible.

Re-upping the 9.3.x patch as that should be the last

alexpott’s picture

Okay I can write a test that proves the problem on SQLite. For the other DB drivers the way we or PDO manages the connections makes it impossible :( I've tried quite a few different ways to keep the db connections out of the forks but it keeps on breaking.

Also since the test is on the key value database I need to include the fix fro the other race condition - ie. #3240813: \Drupal\Core\KeyValueStore\DatabaseStorage::catchException() fails due to races. I propose we merge the issues since have a test is more valuable than the pure application of issue scoping.

alexpott’s picture

+++ b/core/tests/Drupal/KernelTests/Core/KeyValueStore/DatabaseStorageTest.php
@@ -12,6 +13,14 @@
+  /**
+   * {@inheritdoc}
+   */
+  protected function setUp(): void {
+    parent::setUp();
+    $this->factory = 'keyvalue.database';
+  }

FWIW this test is not actually testing the database storage in HEAD :D ... it's testing the memory storage because it is missing this!

alexpott’s picture

The last submitted patch, 15: 3240601-15.test-only.patch, failed testing. View results

daffie’s picture

@alexpott: thank for for adding the Drupal\KernelTests\Core\KeyValueStore\DatabaseStorageTest::testConcurrent() test. It was not the test I was looking for, as I did not know that we could do what your test is testing. Really great!

I have added the testing that I was looking for and I have removed a couple of small changes that to me are not necessary. More testing for the other changes is still necessary to me.

daffie’s picture

FileSize
18.64 KB

Fixed the style code violations.

alexpott’s picture

@daffie I don't think we should be adding the test that #17 adds. Using reflection to test protected method is not really the best. It ties our tests to internals that are not supposed to be exposed. It's not necessary. It would pass against EHAD and you've re-broken DatabaseStorageTest :( The changes to add the setUp are vital so it testing what it actually claims to.

alexpott’s picture

Removing the test from #17 and fixing the change to public for $connection - I did that to work out how to actually test this.

alexpott’s picture

The memory storage does not need to be tested twice - it has it's own test in \Drupal\KernelTests\Core\KeyValueStore\MemoryStorageTest

daffie’s picture

Status: Needs review » Needs work
  1. +++ b/core/lib/Drupal/Core/Flood/DatabaseBackend.php
    @@ -150,19 +150,19 @@ public function garbageCollection() {
    +      return TRUE;
    

    Why do we have this line here. As far as I can see it can be removed.

  2. +++ b/core/lib/Drupal/Core/Menu/MenuTreeStorage.php
    @@ -1163,21 +1163,17 @@ protected function treeDataRecursive(array &$links, array $parents, $depth) {
    -      throw new PluginException($e->getMessage(), 0, $e);
    +      return FALSE;
    

    If we make this change the docblock also need an update.

  3. +++ b/core/lib/Drupal/Core/Menu/MenuTreeStorage.php
    @@ -247,7 +247,7 @@ protected function safeExecuteSelect(SelectInterface $query) {
    -      throw $e;
    +      throw new PluginException($e->getMessage(), 0, $e);
    

    For this change the methods docblock needs to be updated.

  4. +++ b/core/lib/Drupal/Core/Routing/MatcherDumper.php
    @@ -100,7 +100,9 @@ public function dump(array $options = []) {
    -        $this->ensureTableExists();
    +        if (!$this->ensureTableExists()) {
    +          throw $e;
    +        }
    

    Can we do without this change? If not, then the methods docblock needs to be updated.

  5. +++ b/core/lib/Drupal/Core/Cache/DatabaseCacheTagsChecksum.php
    @@ -47,7 +47,7 @@ protected function doInvalidateTags(array $tags) {
    -        $this->catchException($e);
    +        throw $e;
    
    @@ -63,7 +63,7 @@ protected function getTagInvalidationCounts(array $tags) {
    -        $this->catchException($e);
    +        throw $e;
    

    We are updating these methods. Can we then also update the docblock for the thrown exceptions.

  6. +++ b/core/lib/Drupal/Core/Cache/DatabaseCacheTagsChecksum.php
    @@ -133,6 +130,7 @@ public function schemaDefinition() {
    +    @trigger_error('@todo', E_USER_DEPRECATED);
    

    The @todo needs to be fixed.

  7. We are updating the method ensureTableExists() in 9 classes. We have only addeed testing for one class. Is that enough or do we need to add testing to all 9 classes? Can we add a dataprovider to the added test?
alexpott’s picture

Status: Needs work » Needs review
FileSize
3.78 KB
17.67 KB

Thanks for the review @daffie

Re #22

  1. Nice spot. Removed.
  2. Nice spot. Removed incorrect docs
  3. I don't think so. We already have docs for this.
  4. We need this. Added the missing docs.
  5. I'm not wild about this - putting @throws on protected abstract methods feels pretty inconsequential and the reality of these methods throwing an exception is not changed by this patch. Yes they throw directly rather than from \Drupal\Core\Cache\DatabaseCacheTagsChecksum::catchException() but I think we should only document @throws when exceptions are used for control and passing information or on public interfaces. But that's a discussion for another issue.
  6. Fixed - created https://www.drupal.org/node/3243014
  7. I feel that we have copious test coverage of this code. It's very very low level and if any of the ::ensureTableExists() is actually broken our test suite would fail very very hard. I think that it is great to have coverage of the fact that the current approach is broken when multiple threads call this code and that the approach fixes it but doing this for every instance is unnecessary. Especially as we have plans to make this code generic.
daffie’s picture

Issue summary: View changes
Status: Needs review » Reviewed & tested by the community

All code changes look good to me.
All the 9 methods ensureTableExists() are the same. In #2371709: Move the on-demand-table creation into the database API will these methods be replaced by something else.
The class for the database storage of keyvalue values has testing added for asynchronous table creation. According to @alexpott is this enough.
For the added deprecation of Drupal\Core\Cache\DatabaseCacheTagsChecksum::catchException() which is a protected method is a CR created. I have updated the IS for it.
For me it is RTBC.

  • catch committed e0b4b7e on 9.3.x
    Issue #3240601 by alexpott, daffie, daniel.bosen: The standard logic we...
catch’s picture

Status: Reviewed & tested by the community » Fixed

Looks good to me now. I think the single point of test coverage is OK given we're going to remove all the copypasta in the follow-up, and this code runs implicitly thousands of times in the test suite overall.

Committed e0b4b7e and pushed to 9.3.x. Thanks!

It would be possible to backport this to 9.2.x, but given the new deprecation leaving this in 9.3.x

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.