Problem/Motivation

\Drupal\Tests\ckeditor5\FunctionalJavascript\MediaLibraryTest::testButton was skipped in #3268070: Temporarily skip even more failing tests because it's failing randomly on CI: https://www.drupal.org/pift-ci-job/2333668.

Proposed resolution

Robustify and restore the test case.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

Comments

lauriii created an issue. See original summary.

lauriii’s picture

Status: Active » Needs review
StatusFileSize
new3.08 KB
new1.35 KB
wim leers’s picture

Status: Needs review » Reviewed & tested by the community
+++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/MediaLibraryTest.php
@@ -194,7 +193,8 @@ public function testButton() {
-    $assert_session->waitForElement('css', '.js-media-library-item')->click();
+    $this->assertNotEmpty($assert_session->waitForId('drupal-modal'));
+    $assert_session->elementExists('css', '.js-media-library-item')->click();

Ohhhh — so it used to fail mostly because the modal dialog didn't ever appear, and hence the other selector surely cannot exist either?

But … how is that different than just using the original selector? 🤔

Ah … we could've kept the original selector too. The key thing is that we first check that the element exists and only then click() it.

@lauriii chose to just first assert to wait for the modal, because that is conceptually easier to understand: it documents in the test what we are waiting for. 👍

lauriii’s picture

My theory is that it's a problem with the dialog resizing and positioning dynamically. Basically right now we're waiting for the element to exist, and it exists before the dialog has had a chance to resize and position, and therefore the click fails. I don't think what I have there is 100% solid, but I think it should be solid enough to not fail within realistic constraints.

wim leers’s picture

Ohhhh, I hadn't even thought about repositioning potentially causing clicks to fail! 🤯

wim leers’s picture

To clarify: definitely RTBC now! 🚀

This undoes one of the skips in #3268070: Temporarily skip even more failing tests.

xjm’s picture

Assigned: Unassigned » xjm

We need to run this many more times, and compare it to the baseline each time. I will post patches that do that.

xjm’s picture

xjm’s picture

Assigned: xjm » Unassigned

So the baseline isn't failing under current conditions. We should retest these when there are more jobs (around when there are commits pushed or RTBC retesting is happening), or come up with a 100% fail patch that proves a race condition that is fixed by the fix.

lauriii’s picture

StatusFileSize
new3.35 KB
new4.48 KB
new1.83 KB

This should be a 100% solid solution which would pass also against the changes that make the 100% failing patch fail but requires changes outside of that specific failure. I think this is a better approach - #2 was basically just trying to remain consistent which probably shouldn't be a priority over fixing the random fail in the most robust possible way.

The last submitted patch, 10: 3268368-10-50x-baseline.patch, failed testing. View results

The last submitted patch, 10: 3268368-10-50x.patch, failed testing. View results

lauriii’s picture

StatusFileSize
new5.63 KB
new2.97 KB

Baseline is same as in #10.

xjm’s picture

Unfortunately the fails above are:

There was 1 error:

1) Drupal\Tests\ckeditor5\FunctionalJavascript\MediaLibraryTest::testAllowedMediaTypes
Behat\Mink\Exception\ElementTextException: The text "Fear is the mind-killer" was not found in the text of the element matching css ".media-library-item__name".

/var/www/html/vendor/behat/mink/src/WebAssert.php:821
/var/www/html/vendor/behat/mink/src/WebAssert.php:467
/var/www/html/core/tests/Drupal/Tests/WebAssert.php:1005
/var/www/html/core/modules/ckeditor5/tests/src/FunctionalJavascript/MediaLibraryTest.php:252
/var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:726

--

There was 1 failure:

1) Drupal\Tests\ckeditor5\FunctionalJavascript\MediaLibraryTest::testButton
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'Show Image media (selected)'
+''

/var/www/html/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:121
/var/www/html/vendor/phpunit/phpunit/src/Framework/Constraint/IsIdentical.php:90
/var/www/html/core/modules/ckeditor5/tests/src/FunctionalJavascript/MediaLibraryTest.php:164
/var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:726

....Whereas the fail we expect from the random fail is:

Testing Drupal\Tests\ckeditor5\FunctionalJavascript\MediaLibraryTest
  F..                                                                 3 / 3
(100%)

  Time: 00:39.943, Memory: 4.00 MB

  There was 1 failure:

  1)Drupal\Tests\ckeditor5\FunctionalJavascript\MediaLibraryTest::testButton
  Failed asserting that a NULL is not empty.


/var/www/html/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:121

/var/www/html/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:55

/var/www/html/core/modules/ckeditor5/tests/src/FunctionalJavascript/MediaLibraryTest.php:198
  /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:726

So, I'll try queuing more of #8 in a few hours.

lauriii’s picture

The fails for the 100% failing patch won't be the same because there are other cases in the tests that are prone to the same race condition. That's why I proposed first the pragmatic solution which would have just made all of the cases where dialog is being opened consistent, since the failures were happening in this one case where we were waiting for the dialog being opened differently.

I'm fine with the solution in #13, it just needs more changes that are not directly related to the random failure.

  • xjm committed be839ae on 10.0.x
    Issue #3268368 by lauriii, xjm, Wim Leers: Robustify and restore \Drupal...

  • xjm committed 1ce85c7 on 9.4.x
    Issue #3268368 by lauriii, xjm, Wim Leers: Robustify and restore \Drupal...

  • xjm committed de08a3d on 9.3.x
    Issue #3268368 by lauriii, xjm, Wim Leers: Robustify and restore \Drupal...
xjm’s picture

Thanks @lauriii, that makes sense. This is also an identical fix to the one we've used for other race conditions in the past, so hopefully it will harden this test against other failures in the future.

Committed to 10.0.x, and cherry-picked to 9.4.x and 9.3.x. Thanks!

wim leers’s picture

Status: Reviewed & tested by the community » Fixed
xjm’s picture

Forgot to mention that this one doesn't get 9.2.x backport because no CKE5 in 9.2.x. :)

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.