Problem/Motivation
See https://git.drupalcode.org/issue/drupal-2840283/-/jobs/11289519
CKEditor5Markup (Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5Markup)
✔ Attribute encoding
✔ Filter html allowed global attributes
✔ Comments
✘ Styles and scripts
┐
├ Behat\Mink\Exception\ExpectationException: The string "const example = 'Consider this string:
Steps to reproduce
Run repeatedly (assuming ddev and selenium setup locally):
ddev exec vendor/bin/phpunit -c core \
core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5MarkupTest.php \
--filter testStylesAndScriptsLocal results
Two consecutive runs produced different results:
- First run failed at
CKEditor5MarkupTest.php:346, reporting that the first<script>test case was absent. It stopped after 9 assertions. - The immediate second run passed with 19 assertions. GitLab failed on a different script test case, suggesting the markup itself is not the determining factor.
Likely race condition
Each iteration saves the node and immediately inspects the response:
$page->pressButton('Save');
$assert_session->responseContains($expected_content);There is no wait for post-save navigation/rendering to complete. The assertion can therefore inspect the node form or an incomplete response.
Proposed resolution
Wait for an element or text unique to the saved node page before asserting its HTML, for example:
$page->pressButton('Save');
$this->assertNotEmpty(
$assert_session->waitForText("Style and script test - $test_case_name")
);
$assert_session->responseContains($expected_content);
Similar waits already exist elsewhere in CKEditor5MarkupTest for save-related random failures associated with #3520036: [random test failure] CKEditor5MarkupTest.
Remaining tasks
User interface changes
Introduced terminology
API changes
Data model changes
Release notes snippet
Issue fork drupal-3614445
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3614445-random-test-failure-testStylesAndScripts
changes, plain diff MR !16514
Comments
Comment #2
joelpittetComment #3
joelpittetRan
testStylesAndScriptsindependently 20 times:ddev exec vendor/bin/phpunit -c core core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5MarkupTest.php --filter testStylesAndScripts14 passed
6 failed
Failure rate: ~30%
Failed runs: 1, 7, 9, 10, 11, 14
Cases: style x3, script-like tag x2, script x1
Not a lot of test runs but still but this problem is better than less frequent failures (easier to check if solutions help it work).
Comment #5
smustgrave commentedNice find! I got this on another MR and thought it was me was about to debug.
Comment #6
joelpittetThanks @smustgrave, I did the same test 100x (about 45 min) and no failures, should be good to go. 🙌
Comment #9
catchCommitted/pushed to 11.x, thanks!