Problem/Motivation

See https://git.drupalcode.org/issue/drupal-2840283/-/jobs/11289519

     CKEditor5Markup (Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5Markup)
     ✔ Attribute encoding
     ✔ Filter html allowed global attributes
     ✔ Comments
     ✘ Styles and scripts
       ┐
       ├ Behat\Mink\Exception\ExpectationException: The string "const example = 'Consider this string:
    

Steps to reproduce

Run repeatedly (assuming ddev and selenium setup locally):

ddev exec vendor/bin/phpunit -c core \
  core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5MarkupTest.php \
  --filter testStylesAndScripts

Local results
Two consecutive runs produced different results:

  • First run failed at CKEditor5MarkupTest.php:346, reporting that the first <script> test case was absent. It stopped after 9 assertions.
  • The immediate second run passed with 19 assertions. GitLab failed on a different script test case, suggesting the markup itself is not the determining factor.

Likely race condition
Each iteration saves the node and immediately inspects the response:

$page->pressButton('Save');
$assert_session->responseContains($expected_content);

There is no wait for post-save navigation/rendering to complete. The assertion can therefore inspect the node form or an incomplete response.

Proposed resolution

Wait for an element or text unique to the saved node page before asserting its HTML, for example:

$page->pressButton('Save');
$this->assertNotEmpty(
  $assert_session->waitForText("Style and script test - $test_case_name")
);
$assert_session->responseContains($expected_content);

Similar waits already exist elsewhere in CKEditor5MarkupTest for save-related random failures associated with #3520036: [random test failure] CKEditor5MarkupTest.

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Issue fork drupal-3614445

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

joelpittet created an issue. See original summary.

joelpittet’s picture

Issue summary: View changes
joelpittet’s picture

Status: Active » Needs review

Ran testStylesAndScripts independently 20 times:
ddev exec vendor/bin/phpunit -c core core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5MarkupTest.php --filter testStylesAndScripts

14 passed
6 failed
Failure rate: ~30%
Failed runs: 1, 7, 9, 10, 11, 14
Cases: style x3, script-like tag x2, script x1

Not a lot of test runs but still but this problem is better than less frequent failures (easier to check if solutions help it work).

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Nice find! I got this on another MR and thought it was me was about to debug.

joelpittet’s picture

Thanks @smustgrave, I did the same test 100x (about 45 min) and no failures, should be good to go. 🙌

  • catch committed f95ded29 on 11.x
    fix: #3614445 [random test failure] CKEditor5MarkupTest::...

  • catch committed 9e4409d6 on main
    fix: #3614445 [random test failure] CKEditor5MarkupTest::...
catch’s picture

Version: main » 11.x-dev
Status: Reviewed & tested by the community » Fixed

Committed/pushed to 11.x, thanks!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.