Problem/Motivation

Observed in #3315319-9: Random fails in Drupal\Tests\ckeditor5\FunctionalJavascript\AdminUiTest and Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5Test and in #3315490-24: Random fail in Drupal\Tests\layout_builder\FunctionalJavascript\InlineBlockPrivateFilesTest: many tests wait too little (not for enough AJAX responses to arrive) or too much (waiting for an AJAX response that will never come because no AJAX request was triggered, hence waiting for the default timeout of 10 seconds for no reason at all).

Proposed resolution

Add a new ::assertExpectedAjaxRequest() method with a $count parameter to allow explicit testing of the number of AJAX responses any test item should trigger.

Make ::assertWaitOnAjaxRequest() use this new method, but without the assertion.

If either method is called when there are zero ajax requests made, a deprecation error will be fired.

Remaining tasks

User interface changes

None.

API changes

TBD

Data model changes

None.

Release notes snippet

TBD

CommentFileSizeAuthor
#92 3316274-nr-bot.txt15.45 KBneeds-review-queue-bot
#76 3316274-76.patch57.01 KBwim leers
#76 interdiff.txt650 byteswim leers
#74 3316274-74.patch57.02 KBwim leers
#74 interdiff.txt1.5 KBwim leers
#71 3316274-71.patch58.53 KBwim leers
#71 interdiff.txt2.98 KBwim leers
#66 3316274-66.patch56.22 KBwim leers
#66 interdiff.txt2.98 KBwim leers
#65 3316274-64-proof-broken-in-HEAD-proper.patch1.7 KBwim leers
#64 3316274-64-proof-broken-in-HEAD.patch1.7 KBwim leers
#64 3316274-64-FAIL.patch53.45 KBwim leers
#64 interdiff.txt1.22 KBwim leers
#61 3316274-59.patch52.34 KBwim leers
#61 interdiff.txt944 byteswim leers
#58 3316274-58.patch52.34 KBwim leers
#58 interdiff-51-58.txt8.59 KBwim leers
#57 3316274-57.patch52.15 KBwim leers
#57 interdiff-51-57.txt8.4 KBwim leers
#55 interdiff-51-54.txt4.32 KBnod_
#55 core-3316274-54.patch48.69 KBnod_
#52 3316274-51.patch44.08 KBwim leers
#52 interdiff.txt1.08 KBwim leers
#50 3316274-50.patch43.11 KBwim leers
#50 interdiff.txt857 byteswim leers
#49 3316274-49.patch42.38 KBwim leers
#49 interdiff.txt2.76 KBwim leers
#47 3316274-47.patch41.4 KBwim leers
#47 interdiff.txt3.44 KBwim leers
#45 3316274-45.patch38.25 KBwim leers
#45 interdiff-40-45.txt3.3 KBwim leers
#43 3316274-43.patch9.51 KBwim leers
#43 interdiff.txt30.25 KBwim leers
#40 interdiff-37-40.txt780 bytesnod_
#40 core-3316274-40.patch39.76 KBnod_
#37 3316274-37.patch39.76 KBwim leers
#37 interdiff.txt7.62 KBwim leers
#35 3316274-34.patch32.15 KBwim leers
#35 interdiff.txt2.73 KBwim leers
#30 3316274-30.patch30.87 KBwim leers
#30 interdiff.txt550 byteswim leers
#29 3316274-29.patch31.28 KBwim leers
#29 interdiff.txt6.34 KBwim leers
#28 3316274-28.patch28.86 KBwim leers
#28 interdiff.txt4.49 KBwim leers
#27 3316274-27.patch31.02 KBwim leers
#27 interdiff.txt920 byteswim leers
#24 3316274-23.patch30.13 KBwim leers
#24 interdiff.txt9.17 KBwim leers
#22 3316274-22.patch21.59 KBwim leers
#22 interdiff.txt1.56 KBwim leers
#20 3316274-20.patch20.42 KBwim leers
#20 interdiff.txt2.74 KBwim leers
#18 3316274-18.patch19.37 KBwim leers
#18 interdiff.txt823 byteswim leers
#17 3316274-16.patch19.34 KBwim leers
#17 interdiff.txt626 byteswim leers
#15 3316274-15.patch19.4 KBwim leers
#15 interdiff.txt4.23 KBwim leers
#13 3316274-13.patch16.67 KBwim leers
#13 interdiff.txt5.35 KBwim leers
#12 3316274-12.patch12.7 KBwim leers
#12 interdiff.txt778 byteswim leers
#9 3316274-9.patch12.75 KBwim leers
#9 interdiff.txt3.86 KBwim leers
#8 3316274-8.patch12.09 KBwim leers
#8 interdiff.txt6.17 KBwim leers
#4 3316274-4.patch6.27 KBwim leers
#4 interdiff.txt2.01 KBwim leers
#3 3316274-3.patch4.25 KBwim leers

Issue fork drupal-3316274

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

Wim Leers created an issue. See original summary.

wim leers’s picture

Component: base system » phpunit
wim leers’s picture

Status: Active » Needs review
StatusFileSize
new4.25 KB

Let's test this first against the CKE5 module's functional JS tests only.

+++ b/core/tests/Drupal/FunctionalJavascriptTests/JSWebAssert.php
@@ -52,6 +58,38 @@ function isAjaxing(instance) {
+      throw new \LogicException(sprintf('Unnecessary %s call in a test!', __METHOD__));

Let's find out how many of these exceptions are thrown during the tests! 🤓

wim leers’s picture

StatusFileSize
new2.01 KB
new6.27 KB

I know of at least two unnecessary AJAX waits that this will uncover — but curious to see how many more DrupalCI will uncover (I did not run the entire test suite locally).

wim leers’s picture

I'm really hoping this will uncover the root cause of the new random fail in CKEditor5AllowedTagsTest too: https://www.drupal.org/pift-ci-job/2502693 🤞🤞🤞

The last submitted patch, 3: 3316274-3.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 4: 3316274-4.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new6.17 KB
new12.09 KB

This will fix most things.

wim leers’s picture

StatusFileSize
new3.86 KB
new12.75 KB

Most of the remaining failures are due to \Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5TestBase::saveNewTextFormat() doing

    $assert_session->assertWaitOnAjaxRequest();

blindly. Sometimes this already happened previously, explicitly. With this change, we're forced to do it correctly.

The last submitted patch, 8: 3316274-8.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 9: 3316274-9.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new778 bytes
new12.7 KB
+++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/CKEditor5TestBase.php
@@ -72,13 +72,15 @@ public function createNewTextFormat($page, $assert_session, $name = 'ckeditor5')
     $page->selectFieldOption('editor[editor]', $name);
+    if ($name === 'ckeditor5') {
+      $this->assertSame(1, $assert_session->assertWaitOnAjaxRequest());
+    }

This change in #9 was wrong; we need to do this ALWAYS: the Text Editor module loads the selected text editor plugin's configuration UI, if any.

wim leers’s picture

StatusFileSize
new5.35 KB
new16.67 KB

To add to #9 and #12: what we really needed, was to ensure that we do always wait in the one case that we do know needs it: at the end of createNewTextFormat().

As of #12, that now happens … but it caused many duplications to be revealed. Let's remove them.

The last submitted patch, 12: 3316274-12.patch, failed testing. View results

wim leers’s picture

StatusFileSize
new4.23 KB
new19.4 KB
+++ b/core/tests/Drupal/FunctionalJavascriptTests/JSWebAssert.php
@@ -52,6 +58,38 @@ function isAjaxing(instance) {
+    // First invocation on this page: initialization.
+    // @todo Refactor when https://github.com/instaclick/php-webdriver/issues/54 lands.
+    $page_hash = $this->session->getCurrentUrl();
+    if ($current_page_hash !== $page_hash) {

Basically, every ::drupalGet() and ::visit() should cause this hash to be reset, but that's not currently the case. I do see a way of making this semi-reliable for ::drupalGet(), but for \Behat\Mink\Session::visit() we're hard-blocked on upstream support AFAICT, unless we don't mind subclassing external dependencies.

The last submitted patch, 13: 3316274-13.patch, failed testing. View results

wim leers’s picture

StatusFileSize
new626 bytes
new19.34 KB

\Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5AllowedTagsTest::testFullHtml() tests the text format switching security logic, and that executes an XHR request but does not use ?_wrapper_format=drupal_*. So, simplifying the logic slightly.

wim leers’s picture

StatusFileSize
new823 bytes
new19.37 KB

Fix the ::drupalGet() override.

Status: Needs review » Needs work

The last submitted patch, 18: 3316274-18.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new2.74 KB
new20.42 KB

Almost there… now the edge case of asserting zero completed AJAX requests is handled correctly.

Status: Needs review » Needs work

The last submitted patch, 20: 3316274-20.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new1.56 KB
new21.59 KB

This should bring it down further. The last one will be both complex and … majestic 🤓🤯

nod_’s picture

+++ b/core/tests/Drupal/FunctionalJavascriptTests/JSWebAssert.php
@@ -52,6 +73,36 @@ function isAjaxing(instance) {
+    .filter(entry => entry.initiatorType === 'xmlhttprequest')

might need to add 'fetch' there as well, or we can wait until the ajax framework uses fetch to do that.

very clever to use the performance API :D

wim leers’s picture

StatusFileSize
new9.17 KB
new30.13 KB

Alright, that made CKEditor5AllowedTagsTest pass 👍

Now for the last one: AdminUiTest::testSettingsOnlyFireAjaxWithCkeditor5() … I was forced to do something novel and slightly crazy. But it actually enables a next step that will bring sanity. Stay tuned. 🤓

The last submitted patch, 22: 3316274-22.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 24: 3316274-23.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new920 bytes
new31.02 KB

🤞🟢🤞

wim leers’s picture

StatusFileSize
new4.49 KB
new28.86 KB

And now let's clean up JSWebAssert + WebDriverTestBase — thanks to the JS added in #24, we no longer need the brittle in-PHP-test state tracking! 👍

wim leers’s picture

StatusFileSize
new6.34 KB
new31.28 KB
+++ b/core/modules/system/tests/modules/js_testing_log_test/js/js_testing_log.js
@@ -32,4 +32,17 @@
+  // @todo move this to a separate file + module
+  // jQuery.active alone is unable to detect whether an XHR request ever occurred.
+  window.drupalCumulativeXhrCount = 0;
+  $(document)
+    // prettier-ignore
+    .ajaxStart(function() {
+      window.drupalCumulativeXhrCount++;
+      window.drupalActiveXhrCount = window.drupalActiveXhrCount
+        ? window.drupalActiveXhrCount + 1
+        : 1;
+    }).ajaxComplete(function() {
+      window.drupalActiveXhrCount--;
+    });

Let's resolve this @todo, so we can properly evaluate the final result. Introducing core/modules/system/tests/modules/js_testing_ajax_request_test.

wim leers’s picture

Assigned: wim leers » Unassigned
StatusFileSize
new550 bytes
new30.87 KB

As of #29, this is in a clean state and has proven it works. 🥳

The hardened AJAX test infrastructure helped me find 23 ‼️ unnecessary assertWaitOnAjaxRequest() calls in the CKE5 test coverage alone 🤯 I’m hopeful that this will uncover similar test brittleness elsewhere in Drupal core. And I especially hope this will drastically reduce overall FunctionalJavascript.

So … time to run all FunctionalJavascript tests and find out 🤓🙈

The last submitted patch, 29: 3316274-29.patch, failed testing. View results

wim leers’s picture

+++ b/core/tests/Drupal/FunctionalJavascriptTests/JSWebAssert.php
@@ -52,6 +62,28 @@ function isAjaxing(instance) {
+      throw new \LogicException(sprintf('Unnecessary %s call in a test!', __METHOD__));

Obviously this would be a BC break.

Prior to commit, I’d convert the to a trigger_error(…, E_USER_DEPRECATED); — that’d mean contrib is only required to fix this in D11 :)

Status: Needs review » Needs work

The last submitted patch, 30: 3316274-30.patch, failed testing. View results

catch’s picture

I assume we want to backport this to 9.5 - so deprecate in 9.5, will throw an exception in Drupal 11 - which sounds fine to me fwiw and a good plan.

wim leers’s picture

Status: Needs work » Needs review
Issue tags: +Needs tests
StatusFileSize
new2.73 KB
new32.15 KB
+++ b/core/tests/Drupal/FunctionalJavascriptTests/JSWebAssert.php
@@ -50,10 +34,10 @@ public function resetAjaxRequestStateTracking(): void {
-    static $ajax_response_count;

@@ -81,33 +65,25 @@ function isAjaxing(instance) {
-    // Detect unnecessary waits.
-    if ($current_count === $ajax_response_count) {
+    // Detect unnecessary AJAX request waits and inform the test author.
+    if ($drupal_ajax_request_count !== $browser_xhr_request_count) {
       throw new \LogicException(sprintf('Unnecessary %s call in a test!', __METHOD__));
     }

I made a mistake in #28 which causes this detection to no longer work.

This needs tests.

Status: Needs review » Needs work

The last submitted patch, 35: 3316274-34.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new7.62 KB
new39.76 KB

Yay, dozens of problems found: many LogicException: UnnecessaryDrupal\FunctionalJavascriptTests\JSWebAssert::assertWaitOnAjaxRequest call in a test! failures! 🤓

(Which as mentioned in #32 will be converted to deprecation errors later on.)

Let's fix a number of those.

Status: Needs review » Needs work

The last submitted patch, 37: 3316274-37.patch, failed testing. View results

wim leers’s picture

~90% of the remaining failures are in OffCanvasTestBase and SettingsTrayTestBase subclasses, which have historically seen a lot of random JS failures. That they are surfaced, is encouraging; hopefully we can make them more reliable 🤞

nod_’s picture

Status: Needs work » Needs review
StatusFileSize
new39.76 KB
new780 bytes

Issue in CsrfTokenRaceTest is because of the use of ajaxStart which doesn't trigger all the time (by design), it should be ajaxSend which triggers for every ajax request, see https://api.jquery.com/Ajax_Events/

Might fix other tests too

nod_’s picture

Since that changes the check for unnecessary calls, might need to put back all the removed wait on ajax calls to make sure they were necessary or not (we might risk new random failures if we removed a necessary wait).

Status: Needs review » Needs work

The last submitted patch, 40: core-3316274-40.patch, failed testing. View results

wim leers’s picture

Assigned: Unassigned » wim leers
Status: Needs work » Needs review
StatusFileSize
new30.25 KB
new9.51 KB

Woahhhh! So glad you chimed in here! 😬🤩 Agreed with #41 — let's revert all changes in concrete tests to be safe.

Status: Needs review » Needs work

The last submitted patch, 43: 3316274-43.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new3.3 KB
new38.25 KB

Going through every single failure again, and verifying that the changes were necessary…

… and as @nod_ predicted, I found some changes that were not actually necessary:

  1. the one in CKEditor5DialogTest
  2. the one in CKEditor5OffCanvasTest
  3. one of the changes in \Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5AllowedTagsTest::testAllowedTags()
  4. one of the changes in \Drupal\Tests\contextual\FunctionalJavascript\ContextualLinksTest::testContextualLinksClick()

See interdiff-40-45.txt for the fixes that are surfaced by @nod_'s fix! 👍

Status: Needs review » Needs work

The last submitted patch, 45: 3316274-45.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new3.44 KB
new41.4 KB

The remaining ones are tricky; this is the progress I have so far.

Status: Needs review » Needs work

The last submitted patch, 47: 3316274-47.patch, failed testing. View results

wim leers’s picture

Yay, #3316816: Stabilize FunctionalJavascript testing AJAX: make ::setValue() trigger both "input" and "formUpdated" events landed, which means we can vastly simplify

+++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/AdminUiTest.php
@@ -47,25 +46,33 @@ public function testSettingsOnlyFireAjaxWithCkeditor5() {
-    $page->fillField('name', 'trigger validator');
-    $assert_session->waitForText('Machine name');
-    $page->checkField('roles[authenticated]');
+    $this->assertFalse($assert_session->elementExists('css', '#edit-name-machine-name-suffix')->isVisible());
+    $name_field = $page->findField('name');
+    $name_field->setValue('trigger validator');
+    // Trigger AJAX request to generate machine name by interacting with a non-
+    // textual field.
+    $name_field->blur();
+    // Wait for the debounce, which only triggers the 'formUpdated` event up to
+    // once every 0.3 seconds. Don't wait in tests.
+    // @see Drupal.behaviors.formUpdated
+    $this->getSession()->executeScript('jQuery("input[name=name]").trigger("formUpdated");');
+    $this->assertSame(1, $assert_session->assertWaitOnAjaxRequest());
+    $this->assertTrue($assert_session->elementExists('css', '#edit-name-machine-name-suffix')->isVisible());

this 👆 as well as ManageDisplayTest, which also has AJAX waits for machine name-related assertions.

wim leers’s picture

StatusFileSize
new857 bytes
new43.11 KB

It seems many failures in off-canvas-related tests are caused by a single overly wide AJAX wait!

The last submitted patch, 49: 3316274-49.patch, failed testing. View results

wim leers’s picture

StatusFileSize
new1.08 KB
new44.08 KB

And similarly, a single unnecessary AJAX wait was causing all "inline block" test failures (no need to wait for a dialog to close, that's entirely client-side).

The last submitted patch, 50: 3316274-50.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 52: 3316274-51.patch, failed testing. View results

nod_’s picture

Status: Needs work » Needs review
StatusFileSize
new48.69 KB
new4.32 KB

That should take care of the failures in settings tray tests.

Status: Needs review » Needs work

The last submitted patch, 55: core-3316274-54.patch, failed testing. View results

wim leers’s picture

4 failing tests left, each counted twice failure, hence 8.

3 of the 4 are in the "Settings Tray" bucket. Which are somewhat notorious. So I didn't want to change them without understanding the changes. Turns out that since #2785047: In Outside In mode, form validation messages should appear in the off-canvas tray, not the main page, an AJAX redirect is used upon submitting a Settings Tray form through AJAX if there are no validation errors; meaning the page gets reloaded. So technically, the "AJAX waits" are correct, except that chromedriver/mink have already long moved on: they've already followed the redirect, you're already on a new page. Which is why these are correctly detected as invalid waits.

The assertions in HEAD are simply inadequate; what we should be asserting is that the page has been reloaded.

While I was working on this, I see that @nod_ posted #55 with largely the same conclusions, but with a less precise solution. IMHO the approach I took here is safer, because it explicitly waits for the off-canvas dialog to be closed, which really only happens when the redirect is completed. For some reasons, \Drupal\Tests\settings_tray\FunctionalJavascript\ConfigAccessTest::testBlockConfigAccess() was the only one to do that; I just applied it to the other cases too. I verify that a new page is loaded by inspecting window.performance.timeOrigin.

Finally, the

+++ b/core/modules/system/tests/src/FunctionalJavascript/OffCanvasTestBase.php
@@ -123,7 +123,6 @@
   protected function assertElementVisibleAfterWait($selector, $locator, $timeout = 10000) {
-    $this->assertSession()->assertWaitOnAjaxRequest();
     $this->assertNotEmpty($this->assertSession()->waitForElementVisible($selector, $locator, $timeout));
   }

I believe this change is too aggressive/too disruptive, it'll cause changes for the non-Settings Tray tests that use OffCanvasTestBase as a base class.

I believe it's better to convert individual assertElementVisibleAfterWait() calls that don't need to wait for an AJAX request to complete to just call ::waitForElementVisible() directly; that makes them less reliant on this base class too! 👍 That's why the first hunk of #55 is absent from this.

wim leers’s picture

StatusFileSize
new8.59 KB
new52.34 KB

Damn, I meant to post this …

Status: Needs review » Needs work

The last submitted patch, 58: 3316274-58.patch, failed testing. View results

Wim Leers credited bnjmnm.

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new944 bytes
new52.34 KB

Looks like FrameworkTest::testLazyLoadOverriddenCSS() runs so fast on DrupalCI that jQuery has not yet loaded by the time we wait for an AJAX request, and hence it measures it incorrectly. Protect against this.

(Thanks @bnjmnm for the hint on this one!)

But … locally this test never passes — not on the AJAX waiting part, but on the part after that:

    // Verify that the resulting HTML does not load the overridden CSS file.
    // We add a "?" to the assertion, because drupalSettings may include
    // information about the file; we only really care about whether it appears
    // in a LINK or STYLE tag, for which Drupal always adds a query string for
    // cache control.
    $assert->responseNotContains('js.module.css?');

I strongly suspect that this test has been wrong since the day it was introduced; it seems to have made the mistake to think that ::responseContains() and ::responseNotContains() refer to the observed AJAX request, whereas in reality that checks the live page's DOM …

Investigating why…

Status: Needs review » Needs work

The last submitted patch, 61: 3316274-59.patch, failed testing. View results

wim leers’s picture

Turns out the test never passed for me locally simply because I had automatic_updates present on my filesystem and not installed 🙈 #3317229: `automatic_updates_extensions/tests/themes/test_theme` interferes with `core/modules/system/tests/themes/test_theme` caused that. Now looking into the solution…

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new1.22 KB
new53.45 KB
new1.7 KB

Turns out that two choices for testing things in isolation in theme_test actually broke \Drupal\Tests\system\FunctionalJavascript\FrameworkTest::testLazyLoadOverriddenCSS() and this went unnoticed because pretty much no test in Drupal core verifies that an AJAX update triggered by a click on a link or button … actually triggered an AJAX request. This is something I saw in a few of the other updates in above patch updates too: that a test was actually resulting in complete page reloads and that is why tests passed.

wim leers’s picture

+++ b/core/drupalci.yml
@@ -3,9 +3,9 @@
+  testgroups: '--repeat 20 --class "\Drupal\Tests\system\FunctionalJavascript\FrameworkTest'

Wow apparently the missing closing double quote is causing endless retesting on DrupalCI…

wim leers’s picture

Assigned: wim leers » Unassigned
StatusFileSize
new2.98 KB
new56.22 KB

Root causes for the failures in #64:

  1. #2451411: Add libraries-override to themes' *.info.yml started overriding core/jquery to verify that library overrides worked correctly, which means jQuery would no longer load (as @bnjmnm pointed out), which in turns broke core/drupal.ajax, thus breaking FrameworkTest::testLazyLoadOverriddenCSS().
  2. #3301545: Remove popperjs from Drupal 10 started using core/loadjs instead of core/popperjs to test the ability to override individual assets in asset libraries, but this broke core/drupal.ajax, thus breaking FrameworkTest::testLazyLoadOverriddenCSS().

IOW: this test was not at all testing what we thought it did for at least 7.5 years! 😅

This should now be green 🤞

The last submitted patch, 65: 3316274-64-proof-broken-in-HEAD-proper.patch, failed testing. View results

The last submitted patch, 64: 3316274-64-FAIL.patch, failed testing. View results

nod_’s picture

From #57:

The assertions in HEAD are simply inadequate; what we should be asserting is that the page has been reloaded.

Do we though? It's a functional test, so we should only care only about the result not necessarily how it's happening.
I'm not sure that checking that the page changed is making the test more reliable, since we already check that something that displays on the new page does display.

I'm fine with going this way but I wouldn't want to make the tests more brittle by "overchecking" things around what we do want to test explicitly.

In any case, great job making that green :D

wim leers’s picture

It's a functional test, so we should only care only about the result not necessarily how it's happening.

"It depends": if the test is waiting for an AJAX request to complete for a certain thing to happen on the page, and that thing only happens because the AJAX request didn't happen but the page was reloaded instead … then it's necessary. Otherwise we're not at all testing the fact that the AJAX requests triggered page changes without a page reload. Then what's the point of the test? 🙈

I'm fine with going this way but I wouldn't want to make the tests more brittle by "overchecking" things around what we do want to test explicitly.

💯% agreed! I don't want to do this everywhere. Only in cases where I found that tests clearly were aiming to test AJAX functionality but actually weren't. Search for window.performance.timeOrigin — you'll see I only did it in a handful of places :)

wim leers’s picture

Issue tags: -Needs tests +DX (Developer Experience)
StatusFileSize
new2.98 KB
new58.53 KB
  1. Converted from exception to deprecation error
  2. Added test coverage to prove this works as expected
  3. Updated ignored deprecations

P.S.: #3272779: [Symfony 6.1] Replace deprecationListenerTrait with PHPUnitBridge ignoreFile seems to have made deprecation tests a lot trickier — took me 30 mins to figure this out… 🤔

wim leers’s picture

Well that's interesting … #71 had a random failure in ImageTest::testAltTextRequired(), which is A) not using AJAX at all, B) not touched by this issue at all. Created a new issue for it: #3317330: Random failure in ImageTest::testAltTextRequired, and re-testing.

nod_’s picture

Just need to remove the changes to drupalci and that's RTBC for me.

  1. +++ b/core/modules/ckeditor5/tests/src/FunctionalJavascript/AdminUiTest.php
    @@ -78,38 +78,62 @@ public function testSettingsOnlyFireAjaxWithCkeditor5() {
    +  JS
    +        );
    

    nitShould that be on the same line like the code just before? though there was a missing semicolon for a moment here :p

  2. +++ b/core/modules/settings_tray/tests/src/FunctionalJavascript/ConfigAccessTest.php
    @@ -55,10 +55,14 @@ public function testBlockConfigAccess() {
    +    $page_load_hash_1 = $this->getSession()->evaluateScript('window.performance.timeOrigin');
    ...
    +    $page_load_hash_2 = $this->getSession()->evaluateScript('window.performance.timeOrigin');
    +    $this->assertNotSame($page_load_hash_1, $page_load_hash_2);
    

    This makes sense to check for a page change. +1

  3. +++ b/core/modules/views_ui/tests/src/FunctionalJavascript/DisplayTest.php
    @@ -72,7 +72,7 @@ public function testAddDisplay() {
    +    $this->getSession()->wait(1000, "jQuery(':animated').length === 0;");
    

    This one gave me pause since it's a very jQuery thing to test for, and that suppose that jquery is used for the animation (which is the case) if/when we move to CSS animations that will need updating.

    At this point it makes sense here +1.

wim leers’s picture

StatusFileSize
new1.5 KB
new57.02 KB

(Sorry, didn't have time to address the nits ATM.)

Status: Needs review » Needs work

The last submitted patch, 74: 3316274-74.patch, failed testing. View results

wim leers’s picture

Status: Needs work » Needs review
StatusFileSize
new650 bytes
new57.01 KB
1) Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testEditModeEnableDisable
Failed asserting that false is true.

/var/www/html/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:121
/var/www/html/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:55
/var/www/html/core/modules/settings_tray/tests/src/FunctionalJavascript/SettingsTrayTestBase.php:117
/var/www/html/core/modules/settings_tray/tests/src/FunctionalJavascript/SettingsTrayBlockFormTest.php:244
/var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:726

This is happening in \Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayTestBase::assertEditModeEnabled(), which is testing mouse over/hover triggering JS, it's unrelated to AJAX. So fixing this failure is out of scope. Created a new issue: #3317520: [random test failure] Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testEditModeEnableDisable

Fixed #73.1 nit.

wim leers’s picture

FYI: for #72 I opened #3317330: Random failure in ImageTest::testAltTextRequired and in #3317330-6: Random failure in ImageTest::testAltTextRequired fifty test runs all came back green. The only possible conclusion left at that point is that DrupalCI is unreliable?! 😱

andypost’s picture

wim leers’s picture

#78: impossible, because both #3317330: Random failure in ImageTest::testAltTextRequired and #3317520: [random test failure] Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testEditModeEnableDisable do not make any HTTP requests around the code that is failing; it's 100% JS execution on the client without external dependencies.

nod_’s picture

Status: Needs review » Reviewed & tested by the community

All good now, thanks :D

catch’s picture

Status: Reviewed & tested by the community » Needs review

Two questions: one trivial and one not.

+++ b/core/modules/field/tests/src/FunctionalJavascript/EntityReference/EntityReferenceAdminTest.php
@@ -228,8 +228,11 @@ public function testFieldAdminHandler() {
+      if ($checkbox->isChecked()) {
+        $checkbox->uncheck();
+        $assert_session->assertWaitOnAjaxRequest();
+      }

Other places are adding $this->asssertSame(1, $assert_session->waitOnAjaxRequest());. Any reason why not here?

Also not entirely sure about that pattern - most of our assertions don't have a return value. Would there b

+++ b/core/tests/Drupal/FunctionalJavascriptTests/JSWebAssert.php
@@ -28,11 +28,16 @@ class JSWebAssert extends WebAssert {
+   *   When an unnecessary invocation to this method is detected.
    */
-  public function assertWaitOnAjaxRequest($timeout = 10000, $message = 'Unable to complete AJAX request.') {
+  public function assertWaitOnAjaxRequest($timeout = 10000, $message = 'Unable to complete AJAX request.'): int {
     // Wait for a very short time to allow page state to update after clicking.

What if we added an $expected_count parameter here instead of adding the int return?

wim leers’s picture

Status: Needs review » Reviewed & tested by the community

Other places are adding $this->asssertSame(1, $assert_session->waitOnAjaxRequest());. Any reason why not here?

Because I didn't want to presume to know how many checkboxes needed to be unchecked.

The assertion of the count is entirely optional extra hardening, I didn't dig deeper in this particular case because it was a helper.

Also not entirely sure about that pattern - most of our assertions don't have a return value. Would there b

Are you sure about that? Lots of them do, but we often just forget about them … Examples:

  • \Drupal\FunctionalJavascriptTests\JSWebAssert::waitForElement()
  • \Behat\Mink\WebAssert::elementExists()
  • \Drupal\FunctionalJavascriptTests\JSWebAssert::waitForElementVisible()
  • …

What if we added an $expected_count parameter here instead of adding the int return?

Totally possible, but … it'd be more disruptive: any contrib/custom subclass of ::assertWaitOnAjaxRequest() would need to be updated. More importantly still, you'd need to pass in an explicit timeout and message before you could pass that third new parameter … which seems pretty awful and verbose 🙈

catch’s picture

Status: Reviewed & tested by the community » Needs review

@Wim I'm referring to #3131900: Refactor assertions that assign return values to variables and #3129074: Refactor assertions that use return values in conditionals - phpunit doesn't, and we've been slowly updating various legacy SimpleTest assertions not to either.

nod_’s picture

Status: Needs review » Needs work

Patch or MR doesn't apply anymore
The last patch or MR doesn't apply to the target branch, please reroll the code so that it can be reviewed by the automated testbot.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

catch’s picture

Status: Needs work » Needs review
Issue tags: -JavaScript +JavaScript

MR back to green.

Looking at my earlier review point that ended up stalling this:

What if we added an $expected_count parameter here instead of adding the int return?

Totally possible, but … it'd be more disruptive: any contrib/custom subclass of ::assertWaitOnAjaxRequest() would need to be updated. More importantly still, you'd need to pass in an explicit timeout and message before you could pass that third new parameter … which seems pretty awful and verbose

@Wim is correct. The only things I can think of:

1. Adding this as the first parameter to assertWaitOnAjaxRequest(), this would require divining whether the integer is a count or a timeout, but < 25 or < 50 ought to be safe. If it's > that number and no second argument that's also an integer, then issue a deprecation.

2. Add a new method assertAjaxRequestCount($count, $timeout, $message) and just use that where we want to do an assert.

I'm very keen to get this in to (hopefully) stabilise gitlab, but if we add the return now, it's hard to deprecate the method returning anything, unless potentially we do #2 and deprecate the entire old method later, that might be a follow-upable option but could also just do it here.

catch’s picture

Priority: Normal » Critical
Issue tags: +Random test failure
nod_’s picture

Option one just seems terrible, I'd go for option 2 that way we unlock the situation with the return and have a path forward

longwave’s picture

+1 to option 2. Overloading the existing API doesn't feel like a good idea as there is no clean way to do it and the method name doesn't imply that it can also count requests so discoverability of this is hard. Adding a new method explicitly for counting the requests as well makes sense to me.

alexpott’s picture

I agree with @nod_ +1 for option 2 as a way forward.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new15.45 KB

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

nod_’s picture

all good on the js side for me, thanks

catch’s picture

I've added a new method, public function assertExpectedAjaxRequest(?int $count = NULL, $timeout = 10000, $message = 'Unable to complete AJAX request.'): void {

This optionally expects a count, and will run the assertion internally if one is provided, otherwise it's exactly the same as the old method.

https://git.drupalcode.org/project/drupal/-/merge_requests/5367/diffs?co... shows the change. I also updated FrameworkTest.

If this looks OK, I can update all of the calls here which are doing assertSame() on the return value.

We could also entirely deprecate the old method and move everything over - don't particularly mind doing that if it helps this land smoothly, but it'd also be a straightforward follow-up.

catch’s picture

Status: Needs work » Needs review
nod_’s picture

Status: Needs review » Reviewed & tested by the community

All looks good to me

wim leers’s picture

Looks really good. 🤩

Posted one question. But that could be a follow-up too — it's so clear that this is such a huge improvement to functional JS test reliability thanks to:


   * @throws \LogicException
   *   When an unnecessary invocation to this method is detected.

👆 Gone will be the false sense of security of having written a test, when actually that test is incorrect at worst or undeterministic at best!

Note: this will impact contrib, this is a theoretical BC break. Some contrib tests will stop working, because they'll run into this exception. But … that'll actually be a good thing, because each of those will be for a wrong "AJAX wait"! They'll be forced to fix it, and they (and d.o's test infra) will be better for it 👍 Tagging Needs issue summary update and Needs release note because neither the API change is listed nor do we have a release note ready for module maintainers.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

Can we stop this from breaking contrib and custom tests unless they are doing deprecation testing?

I.e. can we only throw the new error if the test is a core test. And if not trigger a deprecation or something. If someone has a large JS test suite that is running fine on their infrastructure causing it to fail and requiring them to remediate immediately feels not great.

Setting to needs work for #97

wim leers’s picture

I.e. can we only throw the new error if the test is a core test. And if not trigger a deprecation or something. If someone has a large JS test suite that is running fine on their infrastructure causing it to fail and requiring them to remediate immediately feels not great.

Yes, using something like \Drupal\Core\Config\Schema\SchemaCheckTrait::isContribViolation():

  /**
   * Whether the current test is for a contrib module.
   *
   * @return bool
   */
  private function isContribViolation(): bool {
    $test_file_name = (new \ReflectionClass($this))->getFileName();
    $root = dirname(__DIR__, 6);
    return !str_starts_with($test_file_name, $root . DIRECTORY_SEPARATOR . 'core');
  }
catch’s picture

I.e. can we only throw the new error if the test is a core test. And if not trigger a deprecation or something.

This is a docs issue - the code triggers a deprecation, the phpdoc appears to have been pre-emptive (or maybe a holdover from before this triggered a deprecation.

catch’s picture

Status: Needs work » Reviewed & tested by the community
wim leers’s picture

I see — my bad! 😅

catch’s picture

Issue summary: View changes
catch’s picture

Title: Stabilize FunctionalJavascript testing AJAX: make ::assertWaitOnAjaxRequest() deterministic » Stabilize FunctionalJavascript testing AJAX: add ::assertExpectedAjaxRequest()

Updated the IS and the CR.

catch’s picture

Issue tags: -Needs release note

Removing the 'needs release note' tag since I think the change record is enough here.

catch’s picture

Above wasn't true for the RuntimeException so I have made that conditional on $count being passed in, we can change that to issue a deprecation when $count isn't passed in later.

longwave’s picture

Status: Reviewed & tested by the community » Needs work

Latest run has a fail that looks relevant:

Drupal\FunctionalJavascriptTests\AjaxWaitTest::testUntrackedXhr
Failed asserting that exception message 'Unable to complete AJAX request.' contains '0 XHR requests through jQuery, but 1 observed in the browser — this requires js_testing_ajax_request_test.js to be updated.'.

Rerunning the pipeline but this is one of the new tests in AjaxWaitTest getting the opposite error message.

catch’s picture

AjaxWaitTest needed updating to use ::assertExpectedAjaxRequest() with a $count parameter, otherwise the error message it's looking for doesn't get triggered (due to tweaks here to make it more bc-friendly and not trigger the new exceptions). Pushed a commit that does that.

catch’s picture

Status: Needs work » Reviewed & tested by the community

Since the fix was just converting the test to use the new API with the stricter behaviour, after relaxing the behaviour for the old API to maintain bc, moving back to RTBC.

  • lauriii committed 2a95e3fd on 11.x
    Issue #3316274 by Wim Leers, catch, nod_, alexpott, longwave, bnjmnm:...

  • lauriii committed a49e4141 on 10.2.x
    Issue #3316274 by Wim Leers, catch, nod_, alexpott, longwave, bnjmnm:...
lauriii’s picture

Version: 11.x-dev » 10.2.x-dev
Status: Reviewed & tested by the community » Fixed

Committed 2a95e3f and pushed to 11.x. Also cherry-picked to 10.2.x. Thanks!

alexpott’s picture

This has caused \Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testBlocks() to fail locally 100% of the time and really really often on gitlabci.

alexpott’s picture

fjgarlin’s picture

I got the same last week too. For example: https://git.drupalcode.org/project/drupal/-/merge_requests/5451/pipelines where the only things that were changed in that MR were "test-only" job code. I tried re-running some of the jobs multiple times and kept getting fails.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

acbramley’s picture

Just FYI for anyone else that finds this issue because their assertWaitOnAjaxRequest() calls started failing, you need to now enable js_testing_ajax_request_test for any JS test that uses this function.

js_testing_ajax_request_test provides test JS code that set window.drupalActiveXhrCount and window.drupalCumulativeXhrCount. Those variables are required to have a successful assertWaitOnAjaxRequest call.

nikolay shapovalov’s picture

Addition to previous comment. You don't need to add js_testing_ajax_request_test if you extends WebDriverTestBase.

Many contrib modules affected by this change:

And in most cases nobody understand the reason of the fail.
Changing from assertWaitOnAjaxRequest() to assertExpectedAjaxRequest() doesn't help at all.
I would love to have instruction how to handle common fail cases, e.g:
RuntimeException: Unable to complete AJAX request.

jsacksick’s picture

I'm confused by this change... Commerce & Commerce shipping tests are affected as well... Even though the failing tests extend the CommerceWebDriverTestBase class (which extends the WebDriverTestBase).
What am I missing? Do we need to not rely on assertWaitOnAjaxRequest() anymore?

bnjmnm’s picture

assertWaitOnAjaxRequest() now fails if during the wait window.drupalCumulativeXhrCount never exceeds 0, so any uses of assertWaitOnAjaxRequest() that weren't actually needed will now fail.

So for tests that were previously passing and now fail on assertWaitOnAjaxRequest(), the first thing to try is removing the call entirely. It was very common to add these calls in places where they were not needed, where they were used for waiting on a JS operation to complete that wasn't necessarily something that used the Drupal AJAX API.

It is possible removing assertWaitOnAjaxRequest() will result a different test failure. Calling assertWaitOnAjaxRequest() when no Ajax calls are present still added a 10000ms wait before proceeding, and tests may have depended on that wait. Ideally, if the assertWaitOnAjaxRequest() removal results in a fail, it should be replaced with a wait for the specific expected change.

One of the reasons for this change was due to every unnecessary assertWaitOnAjaxRequest() adding 10000ms per-call to each test run. The more of these removed, the faster you can expect your tests to run.

There is a deprecation warning that makes this a little more apparent, but if tests are running with --suppress-deprecations it won't be seen. If many contrib projects are running into these failures without seeing that warning, it may be worth opening an issue to have the \RuntimeException message be more detailed than "Unable to complete AJAX request" when it finds no AJAX requests have occurred.

jsacksick’s picture

@bnjmnm: Thanks a lot for this thorough explanation! This is helpful :).

jordan.jamous’s picture

Is there a plan to land this issue in 10.3.x?

acbramley’s picture

@jordan.jamous it is in 10.3.0.