This issue is intended as a tracking issue only to list all of the known intermittent test failures affecting Drupal core. Some issues are due to bugs in Drupal core; others may be due to issues with DrupalCI infrastructure.

Only open issues should be listed in the summary.

Drupal 8

Consistent failures

Intermittent failures

  1. #2806697: Random fail for AlreadyInstalledException
  2. #2828143: Stop NodeTypeTranslationTest and LocaleConfigTranslationImportTest from failing if l.d.o becomes unavailable
  3. #2796757: Failing test with stable version string
  4. #2825845: DST-related test failures in FilterDateTimeTest
  5. #2843024: Random fail in LanguageUILanguageNegotiationTest
  6. #2906317: Random fail due to problems with database
  7. #2926309: Random fail due to APCu not being able to allocate memory
  8. #2924753: Random fail in ExposedFormUITest
  9. #2920590: (canard?) Big Bos of Random Fails is ChainedFastBackend::markAsOutdated()
  10. #2934997: Intermittent failure in MediaUiJavascriptTest
  11. #2936427: Intermittent failure in MediaSourceImageTest

Investigation and hardening for past random failures

Drupal 7

If your patch has an unexpected test failure

  1. Always click on the test results link to see what tests actually failed.
  2. Check the test results on https://www.drupal.org/node/3060/qa. If the same test appears to be failing on the "tested on commit" environments, HEAD may be broken.
  3. Most test failures are actually caused by bugs in patches. Even if a failure appears to be random, consider that the patch may be introducing the bug.
  4. If the test result says "CI error" or "CI aborted", that generally indicates an issue with the infrastructure and not with the patch. Wait a bit, then requeue the tests for the patch.
  5. If the test says "Fatal error" with no other information, click "View results on dispatcher", then "Console Output", then "Full log", and search on the page for "segmentation fault". #2859704: Intermittent segfaults on DrupalCI (some "did not complete due to a fatal error" with no additional info) is a known issue that may cause the test runner to report a fatal in different tests.
  6. If the test failure appears to be one of the above known failures, there is no need to add additional reports of it (unless the issue specifically requests so). Usually, once a fail is identified, Drupal.org maintainers can locate other, similar fails in server logs if needed. However, you should document on your issue which specific known issue caused the failure.
  7. If there is an unexpected test failure that is not covered by one of the above issues, file a new critical issue and add it to this issue's summary.

Comments

xjm created an issue. See original summary.

xjm’s picture

Priority: Critical » Major
Issue summary: View changes
Issue tags: +Triaged D8 major

@effulgentsia, @Cottser, @alexpott, @catch, and I agreed to track this issue as a major rather than a critical. The individual children will be at least major, but a few are not critical.

arlinsandbulte’s picture

Just a thought or maybe a feature request:
Random fails can be really hard to nail down because they are difficult to reproduce. Is there a way (or could one be created) to save ALL input variables (randomly generated strings & values, current time, etc.) and re-use all those input variables on a test re-run?

catch’s picture

Title: [meta] Known intermittent test failures » [meta] Known intermittent/random test failures

Just seen this one:

https://www.drupal.org/pift-ci-job/535086

Standard.Drupal\Tests\standard\Functional\StandardTest

rupal\Tests\standard\Functional\StandardTe
exception: [Other] Line 0 of sites/default/files/simpletest/phpunit-1715.xml:
PHPunit Test failed to complete; Error: PHPUnit 4.8.27 by Sebastian Bergmann and contributors.

E

Time: 1.26 minutes, Memory: 2.75Mb

There was 1 error:

1) Drupal\Tests\standard\Functional\StandardTest::testStandard
GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 30001 milliseconds with 0 bytes received (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

Haven't opened an issue yet, but doesn't look like it's covered by the issues here afaict.

xjm’s picture

@arlinsandbulte, run-tests.sh already supports that in a way -- you can set up a local test run to repeat hundreds of times, die on a fail, and preserve the environment. (For DrupalCI, the amount of storage needed to save every DB from every test run would be astronomical, but you can click through the results link to the full console output at least for tests going back for months.)

Wim Leers’s picture

mikeryan’s picture

Another seemingly random test failure (two known instances): #2831603: Improve assertion for testEntityReferenceAutocompleteWidget

mikeryan’s picture

Issue summary: View changes
vaplas’s picture

timmillwood’s picture

It looks as though 8.3.x HEAD is failing due to https://www.drupal.org/pift-ci-job/557206, this is \Drupal\views\Tests\Update\ArgumentPlaceholderUpdatePathTest. I can't replicate the issue locally.

Wim Leers’s picture

#15: to the best of my knowledge, #2828559-54: UpdatePathTestBase tests randomly failing is where this is being fixed.

wmostrey’s picture

Would it be a good idea to send out a tweet when new random test failures pop up, and when they are resolved? For example by @drupal_org? Just so we know when we can retest a patch.

Mixologic’s picture

Testbot update:

So we've got the new testbot code in production now, and there were many hiccups with the deployment, which we've ironed out to prevent in future releases. One thing that seems to have been exacerbated are these core dumps on apache. I think this may have had something to do with the way that the AMI created its ramfs filespace so that all the tests run from memory and do not use disk IO at all.

The new deployment came with a new ami, and some ramfs settings were modified in the process, which I *think* may be whats causing apache to core dump.

So, next steps:

I have rebuilt the ami again with the latest drupalci code, and modified things to run on tmpfs instead of ramfs (much more stable and should act more like a normal filesystem). This ami went into production today. Additionally this AMI enabled core dump patterns on the host, which are required to get core dumps inside of docker.

When I get back from vacation (Jan 2nd), my next steps are to work on the container build process, and once thats ironed out, the first changes I plan on making to the containers is to get them to produce core dumps, and make sure that we have gdb installed on them. Ideally if a core dump is detected, I can then hook gdb up to them and at least probe them for some more detail and information.

alexpott’s picture

For most of the end of December the testbots have been very unstable. I asked @Mixologic about this on IRC and he said this was down to:

  • first we deployed the whole new drupalci codebase, upgraded jenkins that drupalci uses, and created a new AMI that feeds the testbots (because drupalci needed php7 now)
  • some of the deployments went awry because there were a couple of paths that changed, and jenkins was really, really sensitive to any signal > 0
  • then, when we noticed that there seemed to be an inordinate amount of apache crashes, I made more changes to the AMI to support core dumps, which introduced another path switcheroo.

However we now have 2 new failure modes and #2828559: UpdatePathTestBase tests randomly failing appears more common. The two new failure modes are:

mpdonadio’s picture

Issue summary: View changes

Cleaned up IS list, and added a potentially new one.

alexpott’s picture

Issue summary: View changes
Mixologic’s picture

#2754217: Random Test Failure with "failed to open stream" for temporary://.htaccess is definitely related to #2843082: Random fails due to test installs trying to create /tmp/.htaccess, and I would bet that #2843024: Random fail in LanguageUILanguageNegotiationTest is also related given that the Language manager wasnt getting the tempnam from the FileSystemInterface Object.

alexpott’s picture

Issue summary: View changes

@Mixologic you're right - I've closed my new random issue.

Mixologic’s picture

Issue summary: View changes

Re-triaging some old issues.

Mixologic’s picture

Issue summary: View changes
xjm’s picture

Title: [meta] Known intermittent/random test failures » [meta] Known intermittent, random, and environment-specific test failures
Issue summary: View changes

Adding #2843358: Postgres fail in Drupal\Tests\path\Kernel\Migrate\d6\MigrateUrlAliasTest and issues like it to the scope, so that we have them on this central list.

xjm’s picture

alexpott’s picture

Issue summary: View changes
alexpott’s picture

mpdonadio’s picture

Is https://www.drupal.org/pift-ci-job/577748 the same problem that was addressed in #2830485: \Drupal\Tests\outside_in\FunctionalJavascript\OutsideInBlockFormTest fails randomly or a new one? Think I have seen this a few times, but they got retriggered by the time I was able to dig into them.

vaplas’s picture

Is https://www.drupal.org/pift-ci-job/577893 the new random test failure? Perhaps it is one of old problems of PageCacheTest::testPageCacheAnonymous403404().

Mixologic’s picture

pwolanin’s picture

I had an issue flopping recently between the random OutsideInBlockFormTest fail and a 404 page cache test related fail

see e.g. https://www.drupal.org/pift-ci-job/581480

alexpott’s picture

Issue summary: View changes

@pwolanin can you create issues rather than just commenting here. The OutsideInBlockFormTest is likely to be a JavascriptTestBase issue with timing. The other needs an issue - #2847268: Random fail in Drupal\page_cache\Tests\PageCacheTest:: testPageCacheAnonymous403404()

tacituseu’s picture

Random segfaults in Drupal\node\Tests\Views\FrontPageTest.testCacheTagsWithCachePluginTime, not recorded in error log, instead console output contains:

Segmentation fault (core dumped)
FATAL Drupal\node\Tests\Views\FrontPageTest: test runner returned a non-zero error code (139).

https://www.drupal.org/pift-ci-job/584125
https://www.drupal.org/pift-ci-job/585098
https://www.drupal.org/pift-ci-job/585131
https://www.drupal.org/pift-ci-job/585157

Edit: changed links to d.o.

xjm’s picture

Issue summary: View changes

https://www.drupal.org/pift-ci-job/585223 appears to be failing on dev versions of PHP 7; needs issue.

tacituseu’s picture

Issue summary: View changes

Adding issue for #35

mpdonadio’s picture

Issue summary: View changes

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

xjm’s picture

Issue summary: View changes

https://www.drupal.org/pift-ci-job/592806 always fails on PHP 7.0.x-dev and 7.1.x-dev. Needs issue.

xjm’s picture

Issue summary: View changes

Er, actually, MANY tests fail on PHP dev versions. AjaxTest is just the first one.

xjm’s picture

tacituseu’s picture

tacituseu’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

Thanks @tacituseu.

#2850639: Doctrine incompatibility with PHP 7.2.x: StaticReflectionClass::newInstance($args) should be compatible with ReflectionClass::newInstance(...$args) is reverted upstream in PHP 7.0.x and 7.1.x, so tests are no longer failing because of it. (It is still a critical on its own because it will still affect 7.2.x, but no longer part of what we need to track here.)

wturrell’s picture

This one @mixologic helped me on IRC earlier (thanks!) and reran just my class, which seems to have deleted the details from https://www.drupal.org/pift-ci-job/601122 - however I found the original job: https://dispatcher.drupalci.org/job/drupal_patches/2628/

exception: [Notice] Line 61 of
core/lib/Drupal/Core/Database/Driver/mysql/Schema.php:
Undefined index:
databaseDrupal\Core\Database\Driver\mysql\Schema->getPrefixInfo('%', 1)
(Line: 76)
Drupal\Core\Database\Driver\mysql\Schema->buildTableNameCondition('%',
'LIKE') (Line: 186)
Drupal\Core\Database\Schema->findTables('%') (Line: 1218)
Drupal\simpletest\TestBase->restoreEnvironment() (Line: 976)
Drupal\simpletest\TestBase->run(Array) (Line: 792)
simpletest_script_run_one_test('17', 'Drupal\action\Tests\ActionListTest')
(Line: 66)

Mixologic’s picture

^ that was the original fail, and a rerun showed https://dispatcher.drupalci.org/job/drupal_patches/2637/ all passes.

xjm’s picture

Issue summary: View changes

#2853905: Figure out why there were segfaults in every test run on PHP 7.1.x-dev has shown up the past three days now; adding to summary.

tacituseu’s picture

#51: NULL pointer dereference in garbage collector triggered in Symfony\Component\DependencyInjection\Compiler\ServiceReferenceGraph, possibly related to #2603152: Fix PHP 7 testbot failures.

gambry’s picture

Issue summary: View changes

Drupal\KernelTests\Core\Entity\ContentEntityChangedTest::testChanged
When editing a non-translatable field the updated changed time is equal across all translations.
Failed asserting that 1487987261 matches expected 1487987260.

https://www.drupal.org/pift-ci-job/608035

Noticed on #2627512: Datetime Views plugins don't support timezones #181.

Created #2857843: Random fail in Drupal\KernelTests\Core\Entity\ContentEntityChangedTest::testChanged .

xjm’s picture

I consolidated the segfault issues into #2859704: Intermittent segfaults on DrupalCI (some "did not complete due to a fatal error" with no additional info); they are probably the same. This should cover #51 and #52.

xjm’s picture

vaplas’s picture

testFeedUpdateFields
fail: [Other] Line 49 of core/modules/aggregator/src/Tests/FeedAdminDisplayTest.php:
The non scheduled feed has not been updated. It does not show "x x ago" as last update.

Created #2861067: Random fail in Drupal\aggregator\Tests\FeedAdminDisplayTest::testFeedUpdateFields

xjm’s picture

xjm’s picture

Issue summary: View changes

Adding a note to the instructions in the summary on how to check for a segfault, since the apparent "failure" moves from test to test as new commits are made, we keep getting duplicate bug reports when it moves to a new test, and it is not obvious how to determine whether or not a segfault caused the test failure.

xjm’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes
Lendude’s picture

@xjm thanks for that segfault info, had us baffled in #2863268: Convert web tests to browser tests for tracker module. The actual string to look for seems to be 'Segmentation fault (core dumped)'.

xjm’s picture

Issue summary: View changes

@Lendude, yep, thanks, that.

mpdonadio’s picture

Did we open or solve the 30sec curl issue? Saw it happen again last night, https://www.drupal.org/pift-ci-job/637415 I thought we did, but I am not seeing an issue for it.

mpdonadio’s picture

xjm’s picture

Issue summary: View changes
Lendude’s picture

Hadn't seen this one before https://www.drupal.org/pift-ci-job/646538

Opened #2868880: Random fails in ContextualFilterTest::testAddContextualFilterUI, looks like another javascript test timing issue, like some previously fixed ones.

xjm’s picture

Issue summary: View changes
Lendude’s picture

alexpott’s picture

tacituseu’s picture

There is a lot of Requeued after CI error lately, problems with checking out docker images, no space left messages, started around "May 8, 2017 5:07 PM" (see "Build History" at https://dispatcher.drupalci.org/job/php5.6_mysql5.5/ or https://dispatcher.drupalci.org/job/php7_postgres9.1/, but other runners too).

First failures:
- php5.6_mysql5.5: #2696
- php7_postgres9.1: #3801

Examples of error messages:

Cannot connect to the Docker daemon. Is the docker daemon running on this host?

error: copy-fd: write returned No space left on device

Edit:
Accounting for time zone difference I ran the baseline/"cratering" test from #2859704: Intermittent segfaults on DrupalCI (some "did not complete due to a fatal error" with no additional info) at "8 May 2017 at 3:56PM" so it might be that fallout wasn't isolated to one instance.

vaplas’s picture

xjm’s picture

Issue summary: View changes

Thanks @vaplas. I promoted #2891911: Random fail in Drupal\Tests\locale\Functional\LocaleTranslationUiTest::testStringTranslation to critical (we can always start with critical priority for random failures) and added it to the summary.

tacituseu’s picture

Both HEAD and issue testers started having segmentation faults in garbage collector after 'tested on commit' was re-enabled.

tacituseu’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

Updating to remove the fixed issues from the summary.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

vaplas’s picture

vaplas’s picture

Issue summary: View changes

Updating to remove the fixed issues from the summary.

vaplas’s picture

#2906317: Random fail due to problems with database Perhaps a new random fail, or another manifestation of the old one

vaplas’s picture

Issue summary: View changes

Updated list of random issues.

xjm’s picture

vaplas’s picture

Issue summary: View changes

Restore issue about random fail with php 5.6.

vaplas’s picture

If someone looking at the sources of random fails, then it can be interesting: #2920590: (canard?) Big Bos of Random Fails is ChainedFastBackend::markAsOutdated()

vaplas’s picture

tacituseu’s picture

All testbots are failing quite spectacularly at the moment (e.g. https://www.drupal.org/pift-ci-job/824177), could be a result of merging #2857788: Patch the testbot to start both chrome in webdriver mode and phantomjs in non webdriver mode related changes into production (see: https://www.drupal.org/node/2251767/commits)

benjifisher’s picture

Issue summary: View changes
benjifisher’s picture

Issue summary: View changes

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.