[random test failure] CKEditor5Test::testEmphasis and ::testExistingContent

There a neat gitlab trick to run x number of times?

I'm testing the change by adding this assertion:

$this->assertNotEmpty($assert_session->waitForText('page My test content has been created.'));

I'm trying to run the tests locally before submitting the change, using this command in the terminal:

for round in {1..10}; do
  echo "=== Run $round ==="

  if ddev exec ./vendor/bin/phpunit -c core core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php --filter testEmphasis
  then
    echo "✅ PASSED on run $round"
  else
    echo "❌ FAILED on run $round"
    break
  fi
done

Hi @longwave

The tests with these assertions work because they wait for the message confirming that the content was created or updated successfully. Do you think it would be worth changing the tests to use this type of check?

diff --git a/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
index c35bf4d6076..9308a973d93 100644
--- a/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
+++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
@@ -52,6 +52,7 @@ public function testExistingContent(): void {
     $page->fillField('title[0][value]', 'My test content');
     $page->fillField('body[0][value]', '<p>This is test content</p>');
     $page->pressButton('Save');
+    $this->assertNotEmpty($assert_session->waitForText('page My test content has been created.'));
     $assert_session->responseNotContains('<p>This is test content</p>');
     $assert_session->responseContains('&lt;p&gt;This is test content&lt;/p&gt;');
 
@@ -67,6 +68,7 @@ public function testExistingContent(): void {
     $this->assertNotEmpty($assert_session->waitForElement('css', '.ck-editor'));
     $this->assertNotEmpty($assert_session->waitForText('This is test content'));
     $page->pressButton('Save');
+    $this->assertNotEmpty($assert_session->waitForText('page My test content has been updated.'));
 
     // Assert that the HTML is rendered correctly.
     $assert_session->responseContains('<p>This is test content</p>');
@@ -540,6 +542,7 @@ public function testEmphasis(): void {
     $page->fillField('title[0][value]', 'My test content');
     $page->fillField('body[0][value]', '<p>This is a <em>test!</em></p>');
     $page->pressButton('Save');
+    $this->assertNotEmpty($assert_session->waitForText('page My test content has been created.'));
 
     $this->addNewTextFormat();
 
@@ -551,6 +554,7 @@ public function testEmphasis(): void {
     $this->assertNotEmpty($assert_session->waitForElement('css', '.ck-editor'));
     $this->assertTrue($assert_session->waitForText('This is a test!'));
     $page->pressButton('Save');
+    $this->assertNotEmpty($assert_session->waitForText('page My test content has been updated.'));
 
     $assert_session->responseContains('<p>This is a <em>test!</em></p>');
   }

Hi @longwave.
I’ve added my changes, but I’m not sure where I should run the tests. I don’t think I have permission to run them, right?

This looks good. Eventually the selenium/chromium bug might get fixed, but even if it does the assertions won't do any harm.

Committed/pushed to main and 11.x, thanks!

Spotted this https://git.drupalcode.org/issue/drupal-3613882/-/jobs/11964276

Fail          8.943s testEmphasis                                                                    
Pass          6.997s testEmphasisArbitraryHtml                                                       
Failure              *** Process execution output ***                                                
    PHPUnit 12.5.17 by Sebastian Bergmann and contributors.
    
    Runtime:       PHP 8.5.10
    Configuration: /builds/core/phpunit.xml.dist
    
    F.                                                                  2 / 2 (100%)
    
    HTML output was generated.
    http://localhost/subdirectory/sites/simpletest/browser_output/Drupal_Tests_ckeditor5_FunctionalJavascript_EmphasisTest-1-76223404.html
    http://localhost/subdirectory/sites/simpletest/browser_output/Drupal_Tests_ckeditor5_FunctionalJavascript_EmphasisTest-2-76223404.html
    http://localhost/subdirectory/sites/simpletest/browser_output/Drupal_Tests_ckeditor5_FunctionalJavascript_EmphasisTest-3-55609641.html
    http://localhost/subdirectory/sites/simpletest/browser_output/Drupal_Tests_ckeditor5_FunctionalJavascript_EmphasisTest-4-55609641.html
    
    
    Time: 00:16.811, Memory: 16.00 MB
    
    Emphasis (Drupal\Tests\ckeditor5\FunctionalJavascript\Emphasis)
     ✘ Emphasis
       ┐
       ├ Behat\Mink\Exception\ExpectationException: The string "This is a test!" was not found anywhere in the HTML response of the current page.
       │
       │ /builds/vendor/behat/mink/src/WebAssert.php:888
       │ /builds/vendor/behat/mink/src/WebAssert.php:363
       │ /builds/core/tests/Drupal/Tests/WebAssert.php:569
       │ /builds/core/modules/ckeditor5/tests/src/FunctionalJavascript/EmphasisTest.php:142
       ┴
     ✔ Emphasis arbitrary html
    
    FAILURES!
    Tests: 2, Assertions: 21, Failures: 1.

Should I open a new one sonsidering it's a different test method?
testEmphasisArbitraryHtml

Seems reasonable cloned this here #3620770: [random test failure] EmphasisTest::testEmphasis