Come together with the global Drupal community in Rotterdam, 28 Sept – 1 Oct 2026. Sessions, contribution, connection, and Early Bird savings until 8 June.Problem/Motivation
https://git.drupalcode.org/project/drupal/-/jobs/164471
Drupal\Tests\ckeditor5\FunctionalJavascript\MediaLinkabilityTest::testLinkability with data set "restricted" (false)
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'true'
+'false'
/builds/project/drupal/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:122
/builds/project/drupal/vendor/phpunit/phpunit/src/Framework/Constraint/IsIdentical.php:79
/builds/project/drupal/core/modules/ckeditor5/tests/src/FunctionalJavascript/MediaLinkabilityTest.php:195
/builds/project/drupal/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
This only fails on gitlab CI (see issue comments), so repurposing this issue to enable running the same test multiple times on gitlabci, in a similar way to how we do it on DrupalCI.
Steps to reproduce
Proposed resolution
Remaining tasks
User interface changes
API changes
Data model changes
Release notes snippet
| Comment | File | Size | Author |
|---|---|---|---|
| #3 | 500x_MediaLinkabilityTest.patch | 2.01 KB | spokje |
Issue fork drupal-3393475
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
changes
Comments
Comment #2
wim leersSounds like a race condition that can be fixed using a targeted "wait".
Comment #3
spokjeComment #4
spokjeDoesn't seem to fail (frequently) on Drupal CI, no idea and/or documentation how to run a 500x-MR on GitLab, so not much in it for me here.
Comment #7
catchRe-titling for the changed scope.
Comment #8
catchComment #9
catchI have one small question on the help text, but could go either way on that.
This is considerably more ergonomic than 'find a patch that runs the same test 1,000 times from another issue, edit the classname to the one you want, then upload it to Drupal CI', or 'remember how to modify drupalci.yml so it will run the same test 1,000 times', and given we can't add custom help text to the actual gitlab UI is pretty self-documenting with the error message.
The parallel runs are less useful when you have to manually add the variables, but it will allow 'proof of non-failure' to be done more efficiently - i.e. when we want to run the test thousands of times instead of a couple hundred. And we can tweak this as we go along to support more things.
Comment #10
catchSuccessfully used this for #3395989: [random test failure] ComponentsTaggedReleaseTest sometimes fails.
Comment #11
catchMade one commit here - to remove parallel because the gitlab UI was showing the entire pipeline as still in progress, and I think this might be due to parallel manual jobs. Leaving RTBC because it's just removing an added line and very trivial.
One more idea here, although it could happen in a follow-up, would be to add an extra variable for
REPEAT, then in the pipeline default that to 100 but use it to override the number of runs, so someone can put 1000 for a unit test if they want/need to.Comment #14
catchGoing to go ahead and commit this to unblock more random rest failure sleuthing. Committed/pushed to 11.x and cherry-picked to 10.2.x, thanks!