[meta] Known intermittent, random, and environment-specific test failures

Just a thought or maybe a feature request:
Random fails can be really hard to nail down because they are difficult to reproduce. Is there a way (or could one be created) to save ALL input variables (randomly generated strings & values, current time, etc.) and re-use all those input variables on a test re-run?

Just seen this one:

https://www.drupal.org/pift-ci-job/535086

Standard.Drupal\Tests\standard\Functional\StandardTest
✗
rupal\Tests\standard\Functional\StandardTe
exception: [Other] Line 0 of sites/default/files/simpletest/phpunit-1715.xml:
PHPunit Test failed to complete; Error: PHPUnit 4.8.27 by Sebastian Bergmann and contributors.

E

Time: 1.26 minutes, Memory: 2.75Mb

There was 1 error:

1) Drupal\Tests\standard\Functional\StandardTest::testStandard
GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 30001 milliseconds with 0 bytes received (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

Haven't opened an issue yet, but doesn't look like it's covered by the issues here afaict.

#2830485: \Drupal\Tests\outside_in\FunctionalJavascript\OutsideInBlockFormTest fails randomly is another one.

Another seemingly random test failure (two known instances): #2831603: Improve JSWebAssert::waitOnAutocomplete

It looks as though 8.3.x HEAD is failing due to https://www.drupal.org/pift-ci-job/557206, this is \Drupal\views\Tests\Update\ArgumentPlaceholderUpdatePathTest. I can't replicate the issue locally.

#15: to the best of my knowledge, #2828559-54: UpdatePathTestBase tests randomly failing is where this is being fixed.

Would it be a good idea to send out a tweet when new random test failures pop up, and when they are resolved? For example by @drupal_org? Just so we know when we can retest a patch.

Testbot update:

So we've got the new testbot code in production now, and there were many hiccups with the deployment, which we've ironed out to prevent in future releases. One thing that seems to have been exacerbated are these core dumps on apache. I think this may have had something to do with the way that the AMI created its ramfs filespace so that all the tests run from memory and do not use disk IO at all.

The new deployment came with a new ami, and some ramfs settings were modified in the process, which I *think* may be whats causing apache to core dump.

So, next steps:

I have rebuilt the ami again with the latest drupalci code, and modified things to run on tmpfs instead of ramfs (much more stable and should act more like a normal filesystem). This ami went into production today. Additionally this AMI enabled core dump patterns on the host, which are required to get core dumps inside of docker.

When I get back from vacation (Jan 2nd), my next steps are to work on the container build process, and once thats ironed out, the first changes I plan on making to the containers is to get them to produce core dumps, and make sure that we have gdb installed on them. Ideally if a core dump is detected, I can then hook gdb up to them and at least probe them for some more detail and information.

For most of the end of December the testbots have been very unstable. I asked @Mixologic about this on IRC and he said this was down to:

• first we deployed the whole new drupalci codebase, upgraded jenkins that drupalci uses, and created a new AMI that feeds the testbots (because drupalci needed php7 now)
• some of the deployments went awry because there were a couple of paths that changed, and jenkins was really, really sensitive to any signal > 0
• then, when we noticed that there seemed to be an inordinate amount of apache crashes, I made more changes to the AMI to support core dumps, which introduced another path switcheroo.

However we now have 2 new failure modes and #2828559: UpdatePathTestBase tests randomly failing appears more common. The two new failure modes are:

• https://www.drupal.org/node/2840972
• https://www.drupal.org/node/2840974

Cleaned up IS list, and added a potentially new one.

#2754217: Random Test Failure with "failed to open stream" for temporary://.htaccess is definitely related to #2843082: Random fails due to test installs trying to create /tmp/.htaccess, and I would bet that #2843024: Random fail in LanguageUILanguageNegotiationTest is also related given that the Language manager wasnt getting the tempnam from the FileSystemInterface Object.

@Mixologic you're right - I've closed my new random issue.

Removing #2844509: Random fails in BrowserTestBase tests with Operation timed out after 30001 milliseconds with 0 bytes from the list since we think it is fixed.

Is https://www.drupal.org/pift-ci-job/577748 the same problem that was addressed in #2830485: \Drupal\Tests\outside_in\FunctionalJavascript\OutsideInBlockFormTest fails randomly or a new one? Think I have seen this a few times, but they got retriggered by the time I was able to dig into them.

@vaplas - that may be the case. It appears as though that issue has popped up randomly, recently:

https://www.drupal.org/pift-ci-job/577747
https://www.drupal.org/pift-ci-job/577773
https://www.drupal.org/pift-ci-job/577893
https://www.drupal.org/pift-ci-job/577964
https://www.drupal.org/pift-ci-job/577972

I had an issue flopping recently between the random OutsideInBlockFormTest fail and a 404 page cache test related fail

see e.g. https://www.drupal.org/pift-ci-job/581480

@pwolanin can you create issues rather than just commenting here. The OutsideInBlockFormTest is likely to be a JavascriptTestBase issue with timing. The other needs an issue - #2847268: Random fail in Drupal\page_cache\Tests\PageCacheTest:: testPageCacheAnonymous403404()

Random segfaults in Drupal\node\Tests\Views\FrontPageTest.testCacheTagsWithCachePluginTime, not recorded in error log, instead console output contains:

Segmentation fault (core dumped)
FATAL Drupal\node\Tests\Views\FrontPageTest: test runner returned a non-zero error code (139).

https://www.drupal.org/pift-ci-job/584125
https://www.drupal.org/pift-ci-job/585098
https://www.drupal.org/pift-ci-job/585131
https://www.drupal.org/pift-ci-job/585157

Edit: changed links to d.o.

Adding issue for #35

Added #2848177: More \Drupal\Tests\outside_in\FunctionalJavascript\OutsideInBlockFormTest random fails which is starting to cause RTBC issue bouncing.

Adding #2851840: Random failures in ExposedFormTest, looks like recent commit,
Edit: was reverted already in #1863020: View's build fails when an unrelated form on the same page has validation errors.

This one @mixologic helped me on IRC earlier (thanks!) and reran just my class, which seems to have deleted the details from https://www.drupal.org/pift-ci-job/601122 - however I found the original job: https://dispatcher.drupalci.org/job/drupal_patches/2628/

exception: [Notice] Line 61 of
core/lib/Drupal/Core/Database/Driver/mysql/Schema.php:
Undefined index:
databaseDrupal\Core\Database\Driver\mysql\Schema->getPrefixInfo('%', 1)
(Line: 76)
Drupal\Core\Database\Driver\mysql\Schema->buildTableNameCondition('%',
'LIKE') (Line: 186)
Drupal\Core\Database\Schema->findTables('%') (Line: 1218)
Drupal\simpletest\TestBase->restoreEnvironment() (Line: 976)
Drupal\simpletest\TestBase->run(Array) (Line: 792)
simpletest_script_run_one_test('17', 'Drupal\action\Tests\ActionListTest')
(Line: 66)

^ that was the original fail, and a rerun showed https://dispatcher.drupalci.org/job/drupal_patches/2637/ all passes.

Looks like we may have a new one on php5.6:

https://dispatcher.drupalci.org/job/php5.6_mysql5.5/2134/artifact/jenkin...

Both:
https://dispatcher.drupalci.org/job/php5.6_mysql5.5/2134/

And
https://dispatcher.drupalci.org/job/php5.6_mysql5.5/2140/

Are having segfaults in what look to be the same location in the code:
https://dispatcher.drupalci.org/job/php5.6_mysql5.5/2134/artifact/jenkin...

https://dispatcher.drupalci.org/job/php5.6_mysql5.5/2140/artifact/jenkin...

#51: NULL pointer dereference in garbage collector triggered in Symfony\Component\DependencyInjection\Compiler\ServiceReferenceGraph, possibly related to #2603152: Fix PHP 7 testbot failures.

Drupal\KernelTests\Core\Entity\ContentEntityChangedTest::testChanged
When editing a non-translatable field the updated changed time is equal across all translations.
Failed asserting that 1487987261 matches expected 1487987260.

https://www.drupal.org/pift-ci-job/608035

Noticed on #2627512: Datetime Views plugins don't support timezones #181.

Created #2857843: Random fail in Drupal\KernelTests\Core\Entity\ContentEntityChangedTest::testChanged .

@xjm thanks for that segfault info, had us baffled in #2863268: Convert web tests to browser tests for tracker module. The actual string to look for seems to be 'Segmentation fault (core dumped)'.

Did we open or solve the 30sec curl issue? Saw it happen again last night, https://www.drupal.org/pift-ci-job/637415 I thought we did, but I am not seeing an issue for it.

Figured #64 out, #2866056: ResourceTestBase should not have a timeout

Hadn't seen this one before https://www.drupal.org/pift-ci-job/646538

Opened #2868880: Random fails in ContextualFilterTest::testAddContextualFilterUI, looks like another javascript test timing issue, like some previously fixed ones.

New failures, old culprit: OutsideInBlockFormTest
https://www.drupal.org/pift-ci-job/649976

opened #2870146: Even more random fails in \Drupal\Tests\outside_in\FunctionalJavascript\OutsideInBlockFormTest

Added #2876281: \Drupal\node\Tests\Views\FrontPageTest::testCacheTagsWithCachePluginTime fails randomly

There is a lot of Requeued after CI error lately, problems with checking out docker images, no space left messages, started around "May 8, 2017 5:07 PM" (see "Build History" at https://dispatcher.drupalci.org/job/php5.6_mysql5.5/ or https://dispatcher.drupalci.org/job/php7_postgres9.1/, but other runners too).

First failures:
- php5.6_mysql5.5: #2696
- php7_postgres9.1: #3801

Examples of error messages:

Cannot connect to the Docker daemon. Is the docker daemon running on this host?

error: copy-fd: write returned No space left on device

Edit:
Accounting for time zone difference I ran the baseline/"cratering" test from #2859704: Intermittent segfaults on DrupalCI (some "did not complete due to a fatal error" with no additional info) at "8 May 2017 at 3:56PM" so it might be that fallout wasn't isolated to one instance.

Both HEAD and issue testers started having segmentation faults in garbage collector after 'tested on commit' was re-enabled.

All testbots are failing quite spectacularly at the moment (e.g. https://www.drupal.org/pift-ci-job/824177), could be a result of merging #2857788: Patch the testbot to start both chrome in webdriver mode and phantomjs in non webdriver mode related changes into production (see: https://www.drupal.org/node/2251767/commits)

Looks like theres a new one now with the introduction of the quick start scripts.

#2975644: Random Failure in Drupal\Tests\Core\Command\QuickStartTest

Adding #2980750: [random test failure] Random failure in CommentStatisticsTest.

Adding #2982755: Random failure in SchemaTest::testSchemaChangePrimaryKey with order of composite primary key.

Adding #3045612: Random test failure in MediaStandardProfileTest::testMediaSources.

Adding #3041318: Various random fails due to mis-triggered Mink deprecation error.

Adding to issues summary #3032275: Create a fault-tolerant method for interacting with links and fields in Javascript tests and #2946294: Fix race conditions in OffCanvasTestBase.

Adding to issues summary:
#2892440: Provide helper test method to wait for an element to be removed from the page
#2901792: Disable all animations in Javascript testing
#3055648: Frequent random fail in \Drupal\Tests\media_library\FunctionalJavascript\MediaLibraryTest

Adding to issues summary:
#2856047: Avoid random failures in JavascriptTestBase when testing functionality in a dialog

Removing closed issues.

Adding to issue summary:
#3055982: Remove resizing window in BlockFormMessagesTest::testValidationMessage
#3055983: Occasional locks on SQLite
#3056848: Investigate LayoutBuilderDisableInteractionsTest random fails further

Removing closed and outdated issues from the issue summary.

Hi. We introduced a new intermittent failure a few weeks ago, which broke HEAD yesterday, fixed it and have it at RTBC.
#3067768: Random test fail in ManageGitIgnoreTest

@alexpott was following it, but he said he'd be out afk for a month or so, so it'd be great to have any other core committer interested in less random failures give it either the thumbs up or down.

#3067768: Random test fail in ManageGitIgnoreTest was commited.

Adding to issue summary:
#3070853: Race condition during composer cache clear in functional composer scaffolding plugin tests

Removing closed issues from the issue summary.

Adding #3086001: CreateSampleEntityTest::testSampleValueContentEntity can randomly fail.

#3086001: CreateSampleEntityTest::testSampleValueContentEntity can randomly fail was commited.

Removing closed issues.

Adding a new child issue. #3099427: [random test failure] FieldLayoutTest::testEntityView()

Adding #3101130: ConfigEntityQueryTest::testCaseSensitivity can randomly fail.

Added #3103492: Random fail in WidgetUploadTest

#3101130: ConfigEntityQueryTest::testCaseSensitivity can randomly fail was commited.

Added #3143812: Random test failure Table 'sequences' already exists..

Added #2571183: Deprecate random() usage in tests to avoid random test failures to the issue summary.

Replace #3103492: Random fail in WidgetUploadTest with #3066447: [random test failure] Random failures building media library form after uploading image (WidgetUploadTest). It's a duplicate.

Added #3165263: Allow known flaky tests to be automatically repeated and #3037436: [random test failure] Make QuickEditIntegrationTest more robust and fail proof to the issue summary.

Added #3191559: [random test failure] Random test fail in EntityReferenceWidgetTest to the Intermittent list, as it came up today. I don't see an open issue about it currently.

Added #2921123: Adjust Rectangle class to calculate rotated image dimensions according to libgd 2.2.2+ to the consistent failures section.

Added #3192260: [random test failure] Random fail in media_library CKEditorIntegrationTest to intermittent. 9.2 branch test failed on that today.

#3192231: UnroutedUrlTest is failing on dev versions of PHP seems to be new-ish.

This just failed on 9.1.x branch testing, first time I've seen this one personally.

Drupal\Tests\layout_builder\FunctionalJavascript\LayoutBuilderDisableInteractionsTest::testFormsLinksDisabled
Failed asserting that 'unknown error: session deleted because of page crash\n
from unknown error: cannot determine loading status\n
from tab crashed\n
  (Session info: headless chrome=74.0.3729.157)\n
  (Driver info: chromedriver=2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb),platform=Linux 4.9.0-0.bpo.6-amd64 x86_64)' contains "is not clickable at point".

Looks just like a "random" Chrome crash that we can't do much about.

Session info: headless chrome=74.0.3729.157

Slightly surprised that we are on a nearly 2 year old version of Chrome that is long unsupported, we probably should be testing on modern versions - it is unlikely, but still possible, that a recent change in Chrome breaks somehow (e.g. the off canvas CSS issue, which wouldn't be caught here, but imagine something similar in JavaScript).

Just saw #130 again in https://www.drupal.org/pift-ci-job/2031538, opened #3208791: [random test failure] Random fail in LayoutBuilderDisableInteractionsTest

Added #3210432: [random test failure] LayoutBuilderQuickEditTest::testQuickEditIgnoresDuplicateFields() to intermittent. 9.2 test failed on that today.

Removed closed issues from the Issue Summary.

Added #3214565: [random test failure] Random fail in BuildTestTest::testPortMany to intermittent. 9.3 test failed on that today.

Removed fixed #3214565: [random test failure] Random fail in BuildTestTest::testPortMany from IS.

It looks as though a lot of tests are failing because of #3255749: Composer v2.2 prompts to authorize plugins. It may not be intermittent, but I figure people will look here when their tests start to fail.

Testing Drupal\Tests\Composer\Plugin\Scaffold\Functional\ComposerHookTest
E.                                                                  2 / 2 (100%)

Time: 00:02.074, Memory: 4.00 MB

There was 1 error:

1) Drupal\Tests\Composer\Plugin\Scaffold\Functional\ComposerHookTest::testComposerHooks
RuntimeException: Exit code: 2

I have had two instances of failures of Drupal\Tests\layout_builder\FunctionalJavascript\AjaxBlockTest one is https://www.drupal.org/pift-ci-job/2306318

possibly related Drupal\Tests\layout_builder\FunctionalJavascript\ContentPreviewToggleTest
https://www.drupal.org/pift-ci-job/2306278

On a clean install I'm getting mostly failures locally with occasional passes. The failure doesn't seem to match the browser output so not sure what's causing the failure and inconsistent results are making bisect impossible.

I am also seeing some unexpected failures from layout_builder's functional JavaScript tests: https://www.drupal.org/pift-ci-job/2311799 (AjaxBlockTest and LayoutBuilderTest). Both tests pass locally, at least the first time I try.

The same two tests fail for Drupal core in one of the daily tests: https://www.drupal.org/pift-ci-job/2311855

I notice that Comment #141 was posted a day after #3239123: Refactor (if feasible) uses of the jQuery text function to use vanillaJS was fixed.

https://www.drupal.org/pift-ci-job/2315189 also failed for AjaxBlockTest, ContentPreviewToggleTest, and LayoutBuilderTest. The history shows an alarming uptick in failures on the 9.4.x branch:

I didn't click through all of them, but I have seen a lot more layout_builder random fails than usual.

#3262505: [random test failure] Random error in layout_builder FunctionalJavascript tests now exists (added to the summary), but do we need a more holistic plan for layout_builder tests or something?

Thanks,
-Derek

A new one for me:

https://www.drupal.org/pift-ci-job/2316870

Entity.Drupal\KernelTests\Core\Entity\EntityAutocompleteTest
✗	
Drupal\KernelTests\Core\Entity\EntityAutocompleteTest
fail: [Other] Line 0 of sites/default/files/simpletest/phpunit-319.xml:
PHPUnit Test failed to complete; Error: PHPUnit 9.5.11 by Sebastian Bergmann and contributors.

Testing Drupal\KernelTests\Core\Entity\EntityAutocompleteTest
..                                                                  2 / 2 (100%)

Time: 00:02.829, Memory: 4.00 MB

OK (2 tests, 14 assertions)

Unsilenced deprecation notices (2)

  2x: mb_strtolower(): Passing null to parameter #1 ($string) of type string is deprecated
    2x in EntityAutocompleteTest::testSelectionSettingsHandling from Drupal\KernelTests\Core\Entity

Seen here: #3263391: Remove deprecated code from book.module.

Seems like this isn't too big a deal. Fuzzing triggering a pretty difficult to trigger edge case. #3264050: Fuzzed tag values to EntityAutocompleteController::handleAutocomplete can cause deprecation warning

I faced the same failure as reported in #145 and filled a task for it: #3269085: [random test failure] Random test fail in EntityAutocompleteTest.

I've seen this twice in the past week, Drupal\Tests\quickedit\FunctionalJavascript\QuickEditFileTest::testRemove 1 at https://www.drupal.org/pift-ci-job/2348501

@quietone I saw also #153 in https://www.drupal.org/pift-ci-job/2353983

Multiple Quickedit.Drupal\Tests\quickedit\FunctionalJavascript\SettingsTrayIntegrationTest - WebDriver\Exception\UnknownError: element click intercepted: Element ... is not clickable at point in a row. I had to re-run the test four times, before it finally managed to complete.

Also a few Ckeditor5.Drupal\Tests\ckeditor5\FunctionalJavascript\MediaTest - Drupal\Tests\ckeditor5\FunctionalJavascript\MediaTest::testViewMode with data set "with alignment".

Someone (accidentally?) triggered all ten tests combinations: Seven failed while only three completed, see #3278144: Remove quotes from olivero.info.yml for more.

In the past week Migration tests have failed at least 5 times on fail on Drupal\Tests\quickedit\FunctionalJavascript\QuickEditFileTest::testRemove

https://www.drupal.org/pift-ci-job/2372832

Same for here https://www.drupal.org/pift-ci-job/2370991 .
We got QuickEditFileTest::testRemove and

Drupal\Tests\quickedit\FunctionalJavascript\SettingsTrayIntegrationTest::testQuickEditLinks
WebDriver\Exception\UnknownError: element click intercepted: Element ... is not clickable at point (504, 188). Other element would receive the click: ...

failures.

Quickedit [...] is not clickable at point might have been fixed via #3278215: (Not so) Random test failures SettingsTrayIntegrationTest.

Added #3309907: [random test failure] BookTest::testGetTableOfContents

Removed fixed issues from IS

Removed #3309907: [random test failure] BookTest::testGetTableOfContents and #3311476: Skip DownloadTest on SQLite from IS as they are fixed.

Starting to see some random FunctionalJavascript ones popup. When they fail they sent back over a dozen tickets at once.

https://www.drupal.org/pift-ci-job/2628256

Testing Drupal\Tests\media\FunctionalJavascript\MediaSourceFileTest
E                                                                   1 / 1 (100%)

Time: 00:09.719, Memory: 4.00 MB

There was 1 error:

1) Drupal\Tests\media\FunctionalJavascript\MediaSourceFileTest::testMediaFileSource
Behat\Mink\Exception\ElementHtmlException: The text "text--plain.png" was not found in the attribute "src" of the element matching css "img".

https://www.drupal.org/pift-ci-job/2628180

Testing Drupal\Tests\layout_builder\FunctionalJavascript\ContextualLinksTest
F                                                                   1 / 1 (100%)

Time: 00:18.127, Memory: 4.00 MB

There was 1 failure:

1) Drupal\Tests\layout_builder\FunctionalJavascript\ContextualLinksTest::testContextualLinks
Failed asserting that a NULL is not empty.

/var/www/html/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:122
/var/www/html/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:55
/var/www/html/core/modules/layout_builder/tests/src/FunctionalJavascript/ContextualLinksTest.php:111
/var/www/html/core/modules/layout_builder/tests/src/FunctionalJavascript/ContextualLinksTest.php:80
/var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

Here's a new one #3350973: [random failure] Curl error thrown for http in JSWebAssertTest discovered in https://www.drupal.org/pift-ci-job/2629301

Same as #166 on this issue #3206643: Project messaging channel in core (as experimental) and this job: https://www.drupal.org/pift-ci-job/2629786

Remove fixed issue and add a child to the IS that was not in the list.

There's new InstallerExistingConfigMultilingualTest after upgrade to Symfony 6.3 beta fires very often

Adding #3361121: [random test failure] InstallerExistingConfig[SyncDirectory]MultilingualTest::testConfigSync

Adding #2944421: Double exposed filter field when trying to use "q" as identifier.

And now adding :)

#2944421: Double exposed filter field when trying to use "q" as identifier is an issue that *encountered* a random fail, not an issue *describing* one, removing it from the IS.

(That issue was hitting InstallerExistingConfigSyncDirectoryMultilingualTest and InstallerExistingConfigMultilingualTest, which were recently addressed in #3362306: Skip InstallerExistingConfig[SyncDirectory]MultilingualTest::testConfigSync)

Oops, sorry @tim.plunkett, and thanks for fixing it.

I've been getting these lately,
https://www.drupal.org/pift-ci-job/2734245

1) Drupal\Tests\system\Functional\System\CronRunTest::testAutomatedCron
Failed asserting that 1691374878 is less than 1691374878.

I have not made a child issue.

Got the error above again so I made the issue and added to the IS.

Removed fixed issues from IS.

I saw a failure in core/modules/media/tests/src/FunctionalJavascript/MediaSourceOEmbedVideoTest.php that went away when I re-ran the (GitLab) pipeline. It looks different from the failure reported in #166 on this issue.

https://git.drupalcode.org/issue/drupal-3386762/-/jobs/245517#L405

    Drupal\Tests\media\FunctionalJavascript\MediaSourceOEmbedVideoTest::testMediaOEmbedVideoSource
    WebDriver\Exception\JavaScriptError: javascript error: Cannot read
    properties of null (reading 'getAttribute')
      (Session info: headless chrome=106.0.5249.103)
      (Driver info: chromedriver=106.0.5249.61
    (511755355844955cd3e264779baf0dd38212a4d0-refs/branch-heads/5249@{#569}),platform=Linux
    5.4.241-150.347.amzn2.x86_64 x86_64)
    
    /builds/issue/drupal-3386762/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:198
    /builds/issue/drupal-3386762/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:166
    /builds/issue/drupal-3386762/vendor/instaclick/php-webdriver/lib/WebDriver/Session.php:436
    /builds/issue/drupal-3386762/vendor/behat/mink-selenium2-driver/src/Selenium2Driver.php:973
    /builds/issue/drupal-3386762/vendor/behat/mink/src/Session.php:343
    /builds/issue/drupal-3386762/core/modules/media/tests/src/FunctionalJavascript/MediaSourceOEmbedVideoTest.php:147
    /builds/issue/drupal-3386762/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

Updated latest failed date for #3057767: [random test failure] DialogTest::testDialog() in the IS>

At least some of the 'new' failures on gitlab seem like they might be a regression of https://git.drupalcode.org/project/drupalci_testbot/-/commit/b5e9ad15bcb...

I opened #3317520: Random failure in Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testEditModeEnableDisable but it's a duplicate of #3207125: [random test failure] SettingsTrayBlockFormTest:: testEditModeEnableDisable() which we hadn't seen on DrupalCI since the ipv6 issue was fixed.

New one popped up today #3401720: [random test fail] TimestampFormatterWithTimeDiffViewsTest::testTimestampFormatterWithTimeDiff

https://git.drupalcode.org/issue/drupal-3395404/-/pipelines/49731/test_r...

#191: i've been seeing that one for at least a few days, and I'm pretty sure I've been seeing it since Mon Nov 6.

I also got the one in #188 today. A retest fixed it, I did notice a few dead URL's in that test so not sure if that might be related or not. Create an issue for that as well: #3401764: Replace CollegeHumor URLs and logic in core test cases

Is it just me or have the random failures with night watch and JavaScript gone up with gitlab? And I love gitlab and hope that’s not the case but something I’ve noticed the last 2 weeks

It's definitely gone up, we're seeing frequent failures come back that were previously fixed on DrupalCI via preventing IPV6 requests, but that doesn't appear to be the problem this time, or if it is it's hard to verify, see #3397749: gitlab pod DNS settings for background.

However I've revived #3316274: Stabilize FunctionalJavascript testing AJAX: add ::assertExpectedAjaxRequest() which should help/fix essentially all AJAX tests once it's committed or at least help us to fix any remaining ones.

After that the next worst one, and not fixed by the AJAX assertion changes is #3317520: Random failure in Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testEditModeEnableDisable which has at least one method skipped but might need more until we're able to fix the underlying thing for that too.

Re #194 that can often be the case when your CI environment, or site in general, is more performant. That's because things are faster and tests that previously passed because they took long enough to "wait" for things themselves no longer do so assertions can start failing.

None of the DNS changes we did for the settings ("single-request" and "ndots") made a difference, so we might consider reverting them.

We recently increased the resources requested from some jobs to the pods, as that change seems to have an impact.

I've been doing some additional debugging here #3403162: [investigation] get metrics of request to see which settings need to be changed for random failures, where I'm unskipping most of the skipped tests, and so far I'm only getting one or two repeated offenders, but most of the others no longer fail, or at least I couldn't reproduce.

I'm adding additional artifacts (screenshots and html-shot of the failing step) that might help with debugging, but all that is not finished yet. I just wanted to give an update here.

Added #3414321: Fail in mysql\ConnectionUnitTest::testOpenQueryPrefetchClose

Drupal\FunctionalJavascriptTests\Ajax\AjaxTest failed for me today, even after #3396536: [random test failure] Re-enable AjaxTest::testAjaxFocus()

ref https://git.drupalcode.org/issue/drupal-3421482/-/jobs/927005

    1) Drupal\FunctionalJavascriptTests\Ajax\AjaxTest::testAjaxFocus
    Failed asserting that two strings are equal.
    --- Expected
    +++ Actual
    @@ @@
    -'edit-textfield-3'
    +'edit-textfield-2'
    
    /builds/issue/drupal-3421482/vendor/phpunit/phpunit/src/Framework/Constraint/Equality/IsEqual.php:94
    /builds/issue/drupal-3421482/core/tests/Drupal/FunctionalJavascriptTests/Ajax/AjaxTest.php:335
    /builds/issue/drupal-3421482/vendor/phpunit/phpunit/src/Framework/TestResult.php:728