TestSiteApplicationTest::testInstallWithNonExistingFile() fails when another test creates database tables during the test run

This happens when a unit test that touches the database runs at the same time as tests in\Drupal\Tests\Scripts\TestSiteApplicationTest::testInstallWithNonExistingFile

You can cause the error by running ./vendor/bin/phpunit -v core/tests/Drupal/Tests/Scripts/TestSiteApplicationTest.php --filter testInstallScript in one terminal and then in another terminal you can do ./vendor/bin/phpunit -v core/tests/Drupal/Tests/Scripts/TestSiteApplicationTest.php --filter testInstallWithNonExistingFile repeatedly till it fails for this reason.

This used to happen on tests for security issues because they ran all tests in one go whereas core tests use core/drupalci.yml which splits them up and makes this not happen because this test's methods are run sequentially and there are no other unit tests that create database tables.

After discussing this with mixologic it's because of #3038596: Mechanism that adds custom drupalci.yml configurations does not detect configs in new branches. and I've fixed this before - see #3122328: TestSiteApplicationTest::testInstallWithFileWithNoClass() fails on 9.1.x

Let's leave this one open so we can find it and also fix it once and for all. The correct fix is to do something in CI about #3038596: Mechanism that adds custom drupalci.yml configurations does not detect configs in new branches.

@mixologic has inserted a row into the CI db to fix the problem for 9.3.x

Thanks for fixing this, it looks like the failing test passes now:

19:08:01 Drupal\Tests\Scripts\TestSiteApplicationTest 7 passes

From https://dispatcher.drupalci.org/job/drupal8_core_regression_tests/38131/...

According to #4 this will be fixed by #3038596: Mechanism that adds custom drupalci.yml configurations does not detect configs in new branches., which was committed 5 days ago.

What, if anything is left to do here?

This is one of the fortnightly triage issues for bug smash.

I think given that #3038596: Mechanism that adds custom drupalci.yml configurations does not detect configs in new branches. this is no longer critical. However I think we could still decide to remove the table counting from \Drupal\Tests\Scripts\TestSiteApplicationTest::testInstallWithNonExistingFile() as it is counting tables on the main connection of the runner so that's always going to be fragile. Maybe the correct thing to do here is to test something else from the install process other than table creation. Looking at \Drupal\TestSite\Commands\TestSiteInstallCommand::execute() and when \Drupal\TestSite\Commands\TestSiteInstallCommand::getSetupClass() perhaps the best we can do it test for the exception text (as we are already doing) and be done.

Here's the suggested fix

Just IS updates

However I think we could still decide to remove the table counting from \Drupal\Tests\Scripts\TestSiteApplicationTest::testInstallWithNonExistingFile() as it is counting tables on the main connection of the runner so that's always going to be fragile.

That seems like a reasonable thing to do, the removal of this part reduces fragility which is a great thing to do.