[meta] DrupalCI pgsql Random exceptions, fails, testbot issues

Thanks @isntall for opening this issue. I can confirm the random exceptions and they are all about AlreadyInstalledException(). I think the higher the DCI_Concurrency, the higher the exceptions. Added missing tags...

Thanks @isntall for the additional test with lower concurrency! It's way better than with a concurrency of 27, I just don't get why and where exactly this happens. Spend already some hours tracing this down with no luck, guess we need @amateescu here once we get the SQLite randoms done :)

BTW: Not sure if this a known issue, but if there are exceptions during a bot run on drupalCI the result still has a status of 'SUCCESS'!? Please see this run as an example.

Maybe that's caused previous steps in installer where few Database:: static methods are called so makes some internal structures dirty...

Yup, very strange thinks seem to happen as we also see random fails in the Installer test group itself. Just stumbled upon #2459671: [PP-1] Add detail to AlreadyInstalledException and think this can help us a bit to identify the main cause. I have no time today to run a bot test with #2459671: [PP-1] Add detail to AlreadyInstalledException applied, but will try to do tomorrow...

I tryed this to run locally (ubuntu pg 9.4)

got some failures when running 3 concurrent tests in separate terms

d8t --class 'Drupal\aggregator\Tests\AggregatorCronTest'
*d8t alias for - run-tests.sh --sqlite /dev/shm/d8.db --url http://d8.l/ --concurrency 3

then after installing simpletest module have no failures

the failed test reports absence of different tables, key_value and like that:

Exception Uncaught e Connection.php     609 Drupal\Core\Database\Connection->ha
    Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[42P01]: Undefined
    table: 7 ERROR:  relation "simpletest486318aggregator_feed" does
    not exist
    LINE 1: UPDATE simpletest486318aggregator_feed SET queued='143751622...
                   ^: UPDATE simpletest486318aggregator_feed SET
    queued=:db_update_placeholder_0
    WHERE  (fid = :db_condition_placeholder_0) ; Array
    (
    )
     in Drupal\aggregator\Tests\AggregatorCronTest->testCron() (line 37 of
    /home/andypost/www/core8/core/modules/aggregator/src/Tests/AggregatorCronTest.php).
    Drupal\Core\Database\Connection->handleQueryException(Object, Object, Array,
    Array)
    Drupal\Core\Database\Connection->query(Object, Array, Array)
    Drupal\Core\Database\Driver\pgsql\Connection->query(Object, Array, Array)
    Drupal\Core\Database\Driver\pgsql\Update->execute()
    Drupal\aggregator\Tests\AggregatorCronTest->testCron()

@catch suggested we might be able to work around this and #2508567: DrupalCI SQLite random failures by at least temporarily using single concurrency for the SQLite and Postgres builds, which could be acceptable since these test suites mostly only run nightly. Is that an option to get more reliable results from DrupalCI until any other issues are resolved?

Tagging.

I finally got DCI on my home linux box running docker natively instead of via VM. Running with concurrency 27 I run into similar random exceptions for pgsql-9.1-php-5.5 and mysql-5.5-php-5.5. The exceptions are of the same class: PDOException due to SQLite database locking (related: #1713332: The SQLite database driver fails to drop simpletest tables).

It would be curious to look at the sqlite artifacts for all test builds and run the query in sqlite3 SELECT test_class, message FROM simpletest WHERE status = 'exception'; setting, and compare the results. For easy of use, set .output <FILE>.

Also pgsql-9.1-php-5.5 is running at concurrency 28 rather than 27 as the rest of the config sets are.

Here's a patch that increases volatility of sqlite test-runner database and may improve scalability for concurrent test runners.

Builds started, I don't see any random exceptions in sqlite yet, but I do see them still on the postgresql test build.

Didn't seem to improve sqlite performance either on drupalci. :(

Next option seems to be to try to reduce IO use by PostgreSQL so that SQLite does not lock: #2548439: Turn autovacuum off on pgsql containers.

This is no longer about a few exceptions, but about a general catastrophe. We can't even finish a testbot run because docker stops reporting back at some point, which means we cannot even get meaningful results to see what is wrong.

There are probably at least 4 or 5 issues at the moment if not more, including

1. The initial issue - fix random exceptions. I *think* this *was* caused by SQLite and PostgreSQL fighting over I/O, but since we have bigger issues these might have slipped off the radar.
2. Either one of the following caused the Upgrade path to break horribly and columns just are not correct causing serialization errors: #2544954: SqlContentEntityStorageSchema does not detect column-level schema changes in non-base-fields or #2532476: Menu links should use a TranslationWrapper to encapsulate safe translatable strings from YAML files (#2548725: Fix regression to menu serialization in upgrade path that causes thousands of errors in PostgreSQL)
3. My local test bot now runs out of shared memory, which may be related to #2542776: Add an Upsert class but it is hard to tell running testbot with high concurrency, which by the way, is the only way to test in order to try to replicate what is going on with the official testbot.
4. Test bots, local or official, do not finish and docker times out. This does not appear to be isolated to a single test, but I will keep trying to find it. Some times the testbot can get through 100 test cases and some times not even over 20 test cases.
5. There is a fail in FieldFieldTest again, but I can no longer tell because of the above test bot issues.

Finally the problem is going to keep getting worse and not better.

We cannot stop the bleeding if we keep shooting it.

Great, my last comment got lost on preview, so again in a shorter version.

The last successfully build (with the randoms) was on 10. Aug. 2015 build #124. As of build #125 bot did no longer complete a full run. We need the drupalCI guys here to figure out what is going on. New image deployed and if so which branch in use?
Looking at Jenkins I see that the pgsql-9.1 image is pulled and later on also pgsql-9.4!?

#17.2 The serialization exceptions are all about:
unserialize(): Error at offset 61 of 64 bytesunserialize('O:48:"Drupal\Core\StringTranslation\TranslationWrapper":3:{s:9:"')
So it seems this is a one fix for all ;)

#17.3 Didn't see/noticed this locally but exactly what you described in #17.4
#17.5 Fix is under way #2542124: Drupal\views\Tests\Handler\FieldFieldTest fails on Postgres :)

drumm pointed out that #2336627: Deadlock on cache_config (DatabaseBackend::setMultiple()) was also committed that day and could be the culprit.

Going to try reverting that patch, as well as #2549745: Make DatabaseBackend::doSetMultiple protected, and see what happens.

For posterity, history, and spam: there was a follow-up to that issue that might have caused the revert patch to not apply: #2549745: Make DatabaseBackend::doSetMultiple protected I looked at bright shiny red fail message and IRC instead of comment text.

@webchick, I did a similar revert patch in #2126447-71: Patch testing and the pgsql bot was back to 13k passes, so we know that the new Upsert query used by #2336627: Deadlock on cache_config (DatabaseBackend::setMultiple()) is the problem.

We're trying to fix it in #2549323: Improve the Upsert query implementation for PostgreSQL :)

Thank you!

That issue was committed, but I see 106 failures in the last test run?

That commit was to make the test bot complete a test run again.

Most of the failures are because, somehow, the update path tests won't actually run because the menu gets rebuilt before the update that rebuilds the menu. But only on PostgreSQL - #2548725: Fix regression to menu serialization in upgrade path that causes thousands of errors in PostgreSQL.

Changing to task.

If those test failures are non-random and just a core bug (which I've just bumped to critical at least for now) - does that mean this issue is fixed?

https://twitter.com/alexpott/status/639425771291967488