Problem/Motivation

Seeing errors like:

1) Drupal\Tests\ckeditor5\FunctionalJavascript\AdminUiTest::testSettingsOnlyFireAjaxWithCkeditor5
Failed asserting that a NULL is not empty.

/var/www/html/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:121
/var/www/html/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:55
/var/www/html/core/modules/ckeditor5/tests/src/FunctionalJavascript/AdminUiTest.php:97
/var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:726

Looking at the code here

    $page->checkField('filters[media_embed][status]');
    $this->assertNotEmpty($assert_session->waitForElement('css', '.ajax-progress-throbber'));
    $assert_session->assertWaitOnAjaxRequest();
    $assert_session->responseContains('Media types selectable in the Media Library');
    $assert_session->assertWaitOnAjaxRequest();
    $number_ajax_instances_after = $this->getSession()->evaluateScript('Drupal.ajax.instances.length');

That's three waits and just looks wrong.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

Comments

alexpott created an issue. See original summary.

wim leers’s picture

Assigned: Unassigned » wim leers

🕵️

catch’s picture

Possibility that #3315227: Drupal\Tests\views\FunctionalJavascript\Plugin\views\Handler\FilterTest is failing a lot at the moment also helps with this test, we might want to commit that first and see what the effect is.

alexpott’s picture

wim leers’s picture

Status: Active » Needs review
StatusFileSize
new4.43 KB

Did anything change recently that would cause this to fail now, after it’s not been failing for about a year? (This was introduced in #3231364: Add CKEditor 5 module to Drupal core and was never changed.)

I agree that this looks suspicious: the comment is saying that it's verifying there is NO AJAX rebuild of the form, but then it proceeds to wait for it anyway 🤔

The code you've cited is from line 57, not 97, which is where the fail originated. But perhaps there were multiple failures? According to https://www.drupal.org/pift-ci-job/2497704, it really was just line 97?

I've discovered that many of the waits in the code you cited are basically leftovers to be able to reproduce a problem; they're not actually necessary, plus some of the checks literally are waiting for AJAX responses that are pointless! But I have not

Removing all the unnecessary waits cuts down the test time from 1 minute to 22 seconds on my machine.

Next up to investigate: I suspect that they/we thought $assert_session->responseContains('Media types selectable in the Media Library'); would check the AJAX response … but it does not: it checks the page rendered in the browser, using the DOM… to be continued.

alexpott’s picture

If you change --testgroups in drupalci.yml to testgroups: '--repeat 50 --class "\Drupal\Tests\ckeditor5\FunctionalJavascript\AdminUiTest"' then we can really zero on the random fail.

alexpott’s picture

Status: Needs review » Needs work

$this->assertNotEmpty($assert_session->waitForElement('css', '.ajax-progress-throbber')); this assert does seem dangerous as the ajax will come and go on the page.

I don't think we should be asserting for the absence of ajax occurring ever... I think we should remove all the $assert_session->assertNoAjaxRequestsTriggered(); and ensure that the page as content the we expect after doing whatever we've done via the browser.

wim leers’s picture

StatusFileSize
new504 bytes
new4.75 KB
wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new4.91 KB
new7.7 KB

#7 Yep, I reached the same conclusion about the throbber. But I disagree with your second statement. Verifying that no AJAX request was triggered is literally the purpose of that test!

I think you're going to like this 🤓

wim leers’s picture

StatusFileSize
new1.47 KB
new6.91 KB

Forgot to include this clean-up in #9, but this is functionally equivalent.

The last submitted patch, 8: 3315319-8.patch, failed testing. View results

wim leers’s picture

Assigned: wim leers » Unassigned
StatusFileSize
new3.49 KB
new9.97 KB

Okay, both #9 and #10 passed, which means 100 test runs without a failure. I think that means this is a net stabilizer.

Next:

Next up to investigate: I suspect that they/we thought $assert_session->responseContains('Media types selectable in the Media Library'); would check the AJAX response … but it does not: it checks the page rendered in the browser, using the DOM… to be continued.

I'm 95% certain now that this is the case that these were intended to check that the AJAX response contained that string.

But none of these make sense. Because it is indeed asserting the state of the page, not of the AJAX response. And that string is already present before the AJAX response too, because it is simply shown/hidden by Drupal.behaviors.filterStatus 😅

So, we can simplify those too. Doing that here. Expanding to run all CKE5 tests again, but only once.

alexpott’s picture

  1. +++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/AdminUiTest.php
    @@ -94,11 +78,55 @@ public function testSettingsOnlyFireAjaxWithCkeditor5() {
    +    // Wait for a very short time to allow page state to update after clicking.
    +    usleep(5000);
    +
    +    // Half a second should suffice for any of the test's DOM interactions to
    +    // have triggered an AJAX request, if any.
    +    $condition = <<<JS
    +      (function() {
    +        function isAjaxing(instance) {
    +          return instance && instance.ajaxing === true;
    +        }
    +        return (
    +          // Assert no AJAX request is running (via jQuery or Drupal) and no
    +          // animation is running.
    +          (typeof jQuery === 'undefined' || (jQuery.active === 0 && jQuery(':animated').length === 0)) &&
    +          (typeof Drupal === 'undefined' || typeof Drupal.ajax === 'undefined' || !Drupal.ajax.instances.some(isAjaxing))
    +        );
    +      }())
    +JS;
    +    if (!$session->wait(500, $condition)) {
    +      throw new \LogicException('An AJAX request was still being processed, this suggests a assertWaitOnAjaxRequest() call is missing.');
    +    }
    

    Why is this not all an $this->assertWaitOnAjaxRequest()?

    This looks like lots of duplication.

wim leers’s picture

StatusFileSize
new1.84 KB
new9.38 KB

Hah, in an earlier iteration it was rather different, I had not noticed I ended up with the same exact thing 😅

You're totally right.

Deduplicated 👍

alexpott’s picture

Can you adjust the test groups of the three tests altered here and then run that group with --repeat 50?

Just to make sure we're not introducing any randoms.

This looks really good. Nice sleuthing and adding robustness.

spokje’s picture

StatusFileSize
new10.51 KB

Can you adjust the test groups of the three tests altered here and then run that group with --repeat 50?

spokje’s picture

StatusFileSize
new10.69 KB
spokje’s picture

StatusFileSize
new10.68 KB

Status: Needs review » Needs work

The last submitted patch, 18: 50x_patched_test_classes.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review
StatusFileSize
new11.02 KB

Here's what I meant... we need to put the tests in their own group...

Status: Needs review » Needs work

The last submitted patch, 20: 50x-just-the-changed-tests.patch, failed testing. View results

alexpott’s picture

+++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
@@ -314,10 +314,9 @@ public function testActiveTabsMaintained() {
 
     // Enable media embed to make a second filter config tab visible.
     $this->assertTrue($page->hasUncheckedField('filters[media_embed][status]'));
+    $this->assertNull($assert_session->waitForElementVisible('css', '[data-drupal-selector=edit-filters-media-embed-settings]', 0));
     $page->checkField('filters[media_embed][status]');
-    $assert_session->assertWaitOnAjaxRequest();
-    $assert_session->responseContains('Media types selectable in the Media Library');
-    $assert_session->assertWaitOnAjaxRequest();
+    $this->assertNotNull($assert_session->waitForElementVisible('css', '[data-drupal-selector=edit-filters-media-embed-settings]', 0));

So we have a random fail from line 329 just after these lines.

wim leers’s picture

Assigned: Unassigned » wim leers
wim leers’s picture

Assigned: wim leers » Unassigned
Status: Needs work » Needs review
Related issues: +#3218906: The active "CKEditor 5 plugin settings" vertical tab must not lose focus
StatusFileSize
new12.72 KB
new22.51 KB

This should do the trick. 🤞 I pretty much rewrote \Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5Test::testActiveTabsMaintained() to avoid a huge amount of repetition, and increase legibility.

For context: that test was introduced in #3218906: The active "CKEditor 5 plugin settings" vertical tab must not lose focus, and seems to have contained some erroneous comments since day 1 — which I missed during reviews. This was back in the early days of stabilizing the CKEditor 5 module, so the bar was not quite at the core level yet — the focus was on progress, because nothing was set in stone yet.

I figured it's worth settling this once and forever. So I went deep to ensure I get this right, in particular wrt core/modules/filter/filter.admin.js having rather weird behavior. We're not really testing the behavior of those vertical tabs here; rather we're verifying that CKEditor 5's AJAX updates do not break that behavior. 👍

wim leers’s picture

Here's some commentary of the #24 interdiff to make it easier to understand:

  1. +++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
    @@ -299,6 +299,67 @@ public function languageOfPartsPluginTestHelper($page, $assert_session, $predefi
    +  private function getVerticalTabs(string $container_selector, bool $visible_only = TRUE): array {
    

    New helper, makes it easy to do assertSame([…], $this→getVerticalTabs(…)) calls, which HUGELY improves test legibility and maintainability.

  2. +++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
    @@ -299,6 +299,67 @@ public function languageOfPartsPluginTestHelper($page, $assert_session, $predefi
    +  protected function enableDisabledToolbarItem(string $toolbar_item_id): void {
    

    New helper that also improves legibility.

  3. +++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
    @@ -309,76 +370,134 @@ public function testActiveTabsMaintained() {
    +    // Initial vertical tabs: 3 for filters, 1 for CKE5 plugins.
    +    $this->assertSame([
    +      'Limit allowed HTML tags and correct faulty HTML',
    +      'Convert URLs into links',
    +      'Embed media',
    +    ], $this->getVerticalTabs('#filter-settings-wrapper', FALSE));
    +    $this->assertSame([
    +      'Headings',
    +    ], $this->getVerticalTabs('#plugin-settings-wrapper', FALSE));
    +
    +    // Initial visible vertical tabs: 1 for filters, 1 for CKE5 plugins.
    +    $this->assertSame([
    +      '➡️Limit allowed HTML tags and correct faulty HTML',
    +    ], $this->getVerticalTabs('#filter-settings-wrapper'));
    +    $this->assertSame([
    +      '➡️Headings',
    +    ], $this->getVerticalTabs('#plugin-settings-wrapper'));
    

    Now we start with assessing the initial state, prior to doing ANY interactions.

    Note that the Headings plugin's vertical tab is visible right from the start! This was not the case at the time of #3218906: The active "CKEditor 5 plugin settings" vertical tab must not lose focus, which will come up later.

  4. +++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
    @@ -309,76 +370,134 @@ public function testActiveTabsMaintained() {
    -    // Enable Heading to add a second plugin config form.
    -    $this->assertNotEmpty($assert_session->waitForElement('css', '.ckeditor5-toolbar-button-heading'));
    -    $this->triggerKeyUp('.ckeditor5-toolbar-button-heading', 'ArrowDown');
    

    This is the piece that was just plain wrong/nonsensical. The Heading plugin was already active as described in an earlier bullet.

    The reason this failed silently is because we were just asserting the presence of the "heading" toolbar item, triggering the "arrow down" keyboard button press and … observing that the "Headings" toolbar item was then present. But … it was present since the very beginning… 😅

  5. +++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
    @@ -309,76 +370,134 @@ public function testActiveTabsMaintained() {
    -    // Leave and return to the config form, both sets of tabs should then have
    -    // the first tab active by default.
    +    // Leave and return to the config form, wait for initialized Vertical Tabs.
    

    This is plain wrong, because the Filters plugins settings vertical tabs have … very peculiar behavior (see core/modules/filter/filter.admin.js), and actually the assertions were even in direct contradiction with this comment.

  6. +++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
    @@ -309,76 +370,134 @@ public function testActiveTabsMaintained() {
    -    $this->assertFalse($filter_settings_vertical_tabs[0]->hasClass('is-selected'), "Expected filter tab 1 not selected on initial build");
    -    $this->assertTrue($filter_settings_vertical_tabs[2]->hasClass('is-selected'), "Expected (visible) filter tab 2 selected on initial build");
    

    Here's the direct contradiction: it's the THIRD vertical tab that's selected by default, not the first.

    What complicates this further is that this was just checking ALL filter plugin settings vertical tabs, not just the visible ones.

    All of that complexity/vagueness is now made explicit. 👍

Status: Needs review » Needs work

The last submitted patch, 24: 3315319-24.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new2.66 KB
new22.16 KB
stale element reference: element is not attached to the page document

This actually seems pretty simple to solve … the root cause seems to be that in this particular case, we're dynamically getting the visible vertical tabs … but while we do that, the vertical tabs may be getting overwritten. And … that's true! I just wanted to verify the vertical tabs state before the AJAX request, but after checking a filter's checkbox. In hindsight, it makes sense that this causes a race condition. So let's just do the assertion after the AJAX request has been processed, to avoid this race condition.

Also, since the other functional JS tests have been passing consistently, limiting this to just the one failing test.

wim leers’s picture

StatusFileSize
new531 bytes
new22.16 KB

Green! Let's run it 100 times, not 50. Just to make sure.

wim leers’s picture

Title: Random fail in Drupal\Tests\ckeditor5\FunctionalJavascript\AdminUiTest » Random fails in Drupal\Tests\ckeditor5\FunctionalJavascript\AdminUiTest and Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5Test

The scope grew in #24, where we discovered additional random failures.

wim leers’s picture

StatusFileSize
new2.66 KB
new20.04 KB

59/100 are now green, I think it's time to run the entire test suite now.

Status: Needs review » Needs work

The last submitted patch, 30: 3315319-29.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review

Random fail in SettingsTrayBlockFormTest. Unrelated and out of scope.

lauriii’s picture

  1. Next up to investigate: I suspect that they/we thought $assert_session->responseContains('Media types selectable in the Media Library'); would check the AJAX response … but it does not: it checks the page rendered in the browser, using the DOM… to be continued.

    Not sure if I have been involved in this but I thought this was to guarantee that the page contains "Media types selectable in the Media Library" after the ajax request has completed to make sure that the right ajax request was triggered. Looking at it now, it seems potentially of out of scope to test there so I guess it's fine to remove those assertions.

  2. +++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
    @@ -299,6 +299,67 @@ public function languageOfPartsPluginTestHelper($page, $assert_session, $predefi
    +  private function getVerticalTabs(string $container_selector, bool $visible_only = TRUE): array {
    

    I like what this helper function is doing. 💯 It makes the test cases in ::testActiveTabsMaintained so much easier to follow.

  3. +++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
    @@ -299,6 +299,67 @@ public function languageOfPartsPluginTestHelper($page, $assert_session, $predefi
    +    if (count($vertical_tabs) != 1) {
    

    Nit: probably doesn't matter here but should we use strict comparison?

  4. +++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5Test.php
    @@ -299,6 +299,67 @@ public function languageOfPartsPluginTestHelper($page, $assert_session, $predefi
    +      // When retrieving visible vertical tabs, mark the selected one.
    +      if ($visible_only && $vertical_tab->hasClass('is-selected')) {
    +        $title = "➡️$title";
    +      }
    

    Why are we doing this only when retrieving visible vertical tabs? 🤔

wim leers’s picture

#33:

  1. Worse, it was wrong, because that's not actually what it was testing. See #12 for the explanation.
  2. 🥳
  3. Yeah doesn't matter in this case 🤷‍♀️
  4. Because getting a list of all vertical tabs existing in the DOM at all is a concern independent of which ones are visible. Only visible ones can be selected. That's why I also decided to only have a marker for the active vertical tab when retrieving the visible vertical tabs.
lauriii’s picture

Status: Needs review » Reviewed & tested by the community

#34.1 & 4 👍 makes sense

I think based on that, this is ready for a review from another committer.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Been following this one closely - nice work @Wim Leers and nice review @lauriii

Committed and pushed c0d535a9b5 to 10.1.x and 44fab238e9 to 10.0.x and b038750f0b to 9.5.x and 0ad0f52f7e to 9.4.x. Thanks!

  • alexpott committed c0d535a on 10.1.x
    Issue #3315319 by Wim Leers, Spokje, alexpott, lauriii: Random fails in...

  • alexpott committed 44fab23 on 10.0.x
    Issue #3315319 by Wim Leers, Spokje, alexpott, lauriii: Random fails in...

  • alexpott committed b038750 on 9.5.x
    Issue #3315319 by Wim Leers, Spokje, alexpott, lauriii: Random fails in...

  • alexpott committed 0ad0f52 on 9.4.x
    Issue #3315319 by Wim Leers, Spokje, alexpott, lauriii: Random fails in...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.