Problem/Motivation
Observed in #3315319-9: Random fails in Drupal\Tests\ckeditor5\FunctionalJavascript\AdminUiTest and Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5Test and in #3315490-24: Random fail in Drupal\Tests\layout_builder\FunctionalJavascript\InlineBlockPrivateFilesTest: many tests wait too little (not for enough AJAX responses to arrive) or too much (waiting for an AJAX response that will never come because no AJAX request was triggered, hence waiting for the default timeout of 10 seconds for no reason at all).
Proposed resolution
Add a new ::assertExpectedAjaxRequest() method with a $count parameter to allow explicit testing of the number of AJAX responses any test item should trigger.
Make ::assertWaitOnAjaxRequest() use this new method, but without the assertion.
If either method is called when there are zero ajax requests made, a deprecation error will be fired.
Remaining tasks
User interface changes
None.
API changes
TBD
Data model changes
None.
Release notes snippet
TBD
| Comment | File | Size | Author |
|---|---|---|---|
| #92 | 3316274-nr-bot.txt | 15.45 KB | needs-review-queue-bot |
| #76 | 3316274-76.patch | 57.01 KB | wim leers |
| #76 | interdiff.txt | 650 bytes | wim leers |
| #74 | 3316274-74.patch | 57.02 KB | wim leers |
| #74 | interdiff.txt | 1.5 KB | wim leers |
Issue fork drupal-3316274
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3316274-stabilize-functionaljavascript-testing
changes, plain diff MR !5367
Comments
Comment #2
wim leersComment #3
wim leersLet's test this first against the CKE5 module's functional JS tests only.
Let's find out how many of these exceptions are thrown during the tests! 🤓
Comment #4
wim leersI know of at least two unnecessary AJAX waits that this will uncover — but curious to see how many more DrupalCI will uncover (I did not run the entire test suite locally).
Comment #5
wim leersI'm really hoping this will uncover the root cause of the new random fail in
CKEditor5AllowedTagsTesttoo: https://www.drupal.org/pift-ci-job/2502693 🤞🤞🤞Comment #8
wim leersThis will fix most things.
Comment #9
wim leersMost of the remaining failures are due to
\Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5TestBase::saveNewTextFormat()doingblindly. Sometimes this already happened previously, explicitly. With this change, we're forced to do it correctly.
Comment #12
wim leersThis change in #9 was wrong; we need to do this ALWAYS: the Text Editor module loads the selected text editor plugin's configuration UI, if any.
Comment #13
wim leersTo add to #9 and #12: what we really needed, was to ensure that we do always wait in the one case that we do know needs it: at the end of
createNewTextFormat().As of #12, that now happens … but it caused many duplications to be revealed. Let's remove them.
Comment #15
wim leersBasically, every
::drupalGet()and::visit()should cause this hash to be reset, but that's not currently the case. I do see a way of making this semi-reliable for::drupalGet(), but for\Behat\Mink\Session::visit()we're hard-blocked on upstream support AFAICT, unless we don't mind subclassing external dependencies.Comment #17
wim leers\Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5AllowedTagsTest::testFullHtml()tests the text format switching security logic, and that executes an XHR request but does not use?_wrapper_format=drupal_*. So, simplifying the logic slightly.Comment #18
wim leersFix the
::drupalGet()override.Comment #20
wim leersAlmost there… now the edge case of asserting zero completed AJAX requests is handled correctly.
Comment #22
wim leersThis should bring it down further. The last one will be both complex and … majestic 🤓🤯
Comment #23
nod_might need to add 'fetch' there as well, or we can wait until the ajax framework uses fetch to do that.
very clever to use the performance API :D
Comment #24
wim leersAlright, that made
CKEditor5AllowedTagsTestpass 👍Now for the last one:
AdminUiTest::testSettingsOnlyFireAjaxWithCkeditor5()… I was forced to do something novel and slightly crazy. But it actually enables a next step that will bring sanity. Stay tuned. 🤓Comment #27
wim leers🤞🟢🤞
Comment #28
wim leersAnd now let's clean up
JSWebAssert+WebDriverTestBase— thanks to the JS added in #24, we no longer need the brittle in-PHP-test state tracking! 👍Comment #29
wim leersLet's resolve this
@todo, so we can properly evaluate the final result. Introducingcore/modules/system/tests/modules/js_testing_ajax_request_test.Comment #30
wim leersAs of #29, this is in a clean state and has proven it works. 🥳
The hardened AJAX test infrastructure helped me find 23 ‼️ unnecessary
assertWaitOnAjaxRequest()calls in the CKE5 test coverage alone 🤯 I’m hopeful that this will uncover similar test brittleness elsewhere in Drupal core. And I especially hope this will drastically reduce overallFunctionalJavascript.So … time to run all
FunctionalJavascripttests and find out 🤓🙈Comment #32
wim leersObviously this would be a BC break.
Prior to commit, I’d convert the to a
trigger_error(…, E_USER_DEPRECATED);— that’d mean contrib is only required to fix this in D11 :)Comment #34
catchI assume we want to backport this to 9.5 - so deprecate in 9.5, will throw an exception in Drupal 11 - which sounds fine to me fwiw and a good plan.
Comment #35
wim leersI made a mistake in #28 which causes this detection to no longer work.
This needs tests.
Comment #37
wim leersYay, dozens of problems found: many
LogicException: UnnecessaryDrupal\FunctionalJavascriptTests\JSWebAssert::assertWaitOnAjaxRequest call in a test!failures! 🤓(Which as mentioned in #32 will be converted to deprecation errors later on.)
Let's fix a number of those.
Comment #39
wim leers~90% of the remaining failures are in
OffCanvasTestBaseandSettingsTrayTestBasesubclasses, which have historically seen a lot of random JS failures. That they are surfaced, is encouraging; hopefully we can make them more reliable 🤞Comment #40
nod_Issue in CsrfTokenRaceTest is because of the use of ajaxStart which doesn't trigger all the time (by design), it should be ajaxSend which triggers for every ajax request, see https://api.jquery.com/Ajax_Events/
Might fix other tests too
Comment #41
nod_Since that changes the check for unnecessary calls, might need to put back all the removed wait on ajax calls to make sure they were necessary or not (we might risk new random failures if we removed a necessary wait).
Comment #43
wim leersWoahhhh! So glad you chimed in here! 😬🤩 Agreed with #41 — let's revert all changes in concrete tests to be safe.
Comment #45
wim leersGoing through every single failure again, and verifying that the changes were necessary…
… and as @nod_ predicted, I found some changes that were not actually necessary:
CKEditor5DialogTestCKEditor5OffCanvasTest\Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5AllowedTagsTest::testAllowedTags()\Drupal\Tests\contextual\FunctionalJavascript\ContextualLinksTest::testContextualLinksClick()See
interdiff-40-45.txtfor the fixes that are surfaced by @nod_'s fix! 👍Comment #47
wim leersThe remaining ones are tricky; this is the progress I have so far.
Comment #49
wim leersYay, #3316816: Stabilize FunctionalJavascript testing AJAX: make ::setValue() trigger both "input" and "formUpdated" events landed, which means we can vastly simplify
this 👆 as well as
ManageDisplayTest, which also has AJAX waits for machine name-related assertions.Comment #50
wim leersIt seems many failures in off-canvas-related tests are caused by a single overly wide AJAX wait!
Comment #52
wim leersAnd similarly, a single unnecessary AJAX wait was causing all "inline block" test failures (no need to wait for a dialog to close, that's entirely client-side).
Comment #55
nod_That should take care of the failures in settings tray tests.
Comment #57
wim leers4 failing tests left, each counted twice failure, hence 8.
3 of the 4 are in the "Settings Tray" bucket. Which are somewhat notorious. So I didn't want to change them without understanding the changes. Turns out that since #2785047: In Outside In mode, form validation messages should appear in the off-canvas tray, not the main page, an AJAX redirect is used upon submitting a Settings Tray form through AJAX if there are no validation errors; meaning the page gets reloaded. So technically, the "AJAX waits" are correct, except that chromedriver/mink have already long moved on: they've already followed the redirect, you're already on a new page. Which is why these are correctly detected as invalid waits.
The assertions in HEAD are simply inadequate; what we should be asserting is that the page has been reloaded.
While I was working on this, I see that @nod_ posted #55 with largely the same conclusions, but with a less precise solution. IMHO the approach I took here is safer, because it explicitly waits for the off-canvas dialog to be closed, which really only happens when the redirect is completed. For some reasons,
\Drupal\Tests\settings_tray\FunctionalJavascript\ConfigAccessTest::testBlockConfigAccess()was the only one to do that; I just applied it to the other cases too. I verify that a new page is loaded by inspectingwindow.performance.timeOrigin.Finally, the
I believe this change is too aggressive/too disruptive, it'll cause changes for the non-Settings Tray tests that use
OffCanvasTestBaseas a base class.I believe it's better to convert individual
assertElementVisibleAfterWait()calls that don't need to wait for an AJAX request to complete to just call::waitForElementVisible()directly; that makes them less reliant on this base class too! 👍 That's why the first hunk of #55 is absent from this.Comment #58
wim leersDamn, I meant to post this …
Comment #61
wim leersLooks like
FrameworkTest::testLazyLoadOverriddenCSS()runs so fast on DrupalCI that jQuery has not yet loaded by the time we wait for an AJAX request, and hence it measures it incorrectly. Protect against this.(Thanks @bnjmnm for the hint on this one!)
But … locally this test never passes — not on the AJAX waiting part, but on the part after that:
I strongly suspect that this test has been wrong since the day it was introduced; it seems to have made the mistake to think that
::responseContains()and::responseNotContains()refer to the observed AJAX request, whereas in reality that checks the live page's DOM …Investigating why…
Comment #63
wim leersTurns out the test never passed for me locally simply because I had
automatic_updatespresent on my filesystem and not installed 🙈 #3317229: `automatic_updates_extensions/tests/themes/test_theme` interferes with `core/modules/system/tests/themes/test_theme` caused that. Now looking into the solution…Comment #64
wim leersTurns out that two choices for testing things in isolation in
theme_testactually broke\Drupal\Tests\system\FunctionalJavascript\FrameworkTest::testLazyLoadOverriddenCSS()and this went unnoticed because pretty much no test in Drupal core verifies that an AJAX update triggered by a click on a link or button … actually triggered an AJAX request. This is something I saw in a few of the other updates in above patch updates too: that a test was actually resulting in complete page reloads and that is why tests passed.Comment #65
wim leersWow apparently the missing closing double quote is causing endless retesting on DrupalCI…
Comment #66
wim leersRoot causes for the failures in #64:
core/jqueryto verify that library overrides worked correctly, which meansjQuerywould no longer load (as @bnjmnm pointed out), which in turns brokecore/drupal.ajax, thus breakingFrameworkTest::testLazyLoadOverriddenCSS().core/loadjsinstead ofcore/popperjsto test the ability to override individual assets in asset libraries, but this brokecore/drupal.ajax, thus breakingFrameworkTest::testLazyLoadOverriddenCSS().IOW: this test was not at all testing what we thought it did for at least 7.5 years! 😅
This should now be green 🤞
Comment #69
nod_From #57:
Do we though? It's a functional test, so we should only care only about the result not necessarily how it's happening.
I'm not sure that checking that the page changed is making the test more reliable, since we already check that something that displays on the new page does display.
I'm fine with going this way but I wouldn't want to make the tests more brittle by "overchecking" things around what we do want to test explicitly.
In any case, great job making that green :D
Comment #70
wim leers"It depends": if the test is waiting for an AJAX request to complete for a certain thing to happen on the page, and that thing only happens because the AJAX request didn't happen but the page was reloaded instead … then it's necessary. Otherwise we're not at all testing the fact that the AJAX requests triggered page changes without a page reload. Then what's the point of the test? 🙈
💯% agreed! I don't want to do this everywhere. Only in cases where I found that tests clearly were aiming to test AJAX functionality but actually weren't. Search for
window.performance.timeOrigin— you'll see I only did it in a handful of places :)Comment #71
wim leersP.S.: #3272779: [Symfony 6.1] Replace deprecationListenerTrait with PHPUnitBridge ignoreFile seems to have made deprecation tests a lot trickier — took me 30 mins to figure this out… 🤔
Comment #72
wim leersWell that's interesting … #71 had a random failure in
ImageTest::testAltTextRequired(), which is A) not using AJAX at all, B) not touched by this issue at all. Created a new issue for it: #3317330: Random failure in ImageTest::testAltTextRequired, and re-testing.Comment #73
nod_Just need to remove the changes to drupalci and that's RTBC for me.
nitShould that be on the same line like the code just before? though there was a missing semicolon for a moment here :p
This makes sense to check for a page change. +1
This one gave me pause since it's a very jQuery thing to test for, and that suppose that jquery is used for the animation (which is the case) if/when we move to CSS animations that will need updating.
At this point it makes sense here +1.
Comment #74
wim leers(Sorry, didn't have time to address the nits ATM.)
Comment #76
wim leersThis is happening in
\Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayTestBase::assertEditModeEnabled(), which is testing mouse over/hover triggering JS, it's unrelated to AJAX. So fixing this failure is out of scope. Created a new issue: #3317520: [random test failure] Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testEditModeEnableDisableFixed #73.1 nit.
Comment #77
wim leersFYI: for #72 I opened #3317330: Random failure in ImageTest::testAltTextRequired and in #3317330-6: Random failure in ImageTest::testAltTextRequired fifty test runs all came back green. The only possible conclusion left at that point is that DrupalCI is unreliable?! 😱
Comment #78
andypostAll unrelated failures are caused by #3316016: DNS lookups on containers can be throttled
Comment #79
wim leers#78: impossible, because both #3317330: Random failure in ImageTest::testAltTextRequired and #3317520: [random test failure] Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testEditModeEnableDisable do not make any HTTP requests around the code that is failing; it's 100% JS execution on the client without external dependencies.
Comment #80
nod_All good now, thanks :D
Comment #81
catchTwo questions: one trivial and one not.
Other places are adding
$this->asssertSame(1, $assert_session->waitOnAjaxRequest());. Any reason why not here?Also not entirely sure about that pattern - most of our assertions don't have a return value. Would there b
What if we added an $expected_count parameter here instead of adding the int return?
Comment #82
wim leersBecause I didn't want to presume to know how many checkboxes needed to be unchecked.
The assertion of the count is entirely optional extra hardening, I didn't dig deeper in this particular case because it was a helper.
Are you sure about that? Lots of them do, but we often just forget about them … Examples:
\Drupal\FunctionalJavascriptTests\JSWebAssert::waitForElement()\Behat\Mink\WebAssert::elementExists()\Drupal\FunctionalJavascriptTests\JSWebAssert::waitForElementVisible()Totally possible, but … it'd be more disruptive: any contrib/custom subclass of
::assertWaitOnAjaxRequest()would need to be updated. More importantly still, you'd need to pass in an explicit timeout and message before you could pass that third new parameter … which seems pretty awful and verbose 🙈Comment #83
catch@Wim I'm referring to #3131900: Refactor assertions that assign return values to variables and #3129074: Refactor assertions that use return values in conditionals - phpunit doesn't, and we've been slowly updating various legacy SimpleTest assertions not to either.
Comment #84
nod_Patch or MR doesn't apply anymore
The last patch or MR doesn't apply to the target branch, please reroll the code so that it can be reviewed by the automated testbot.
Comment #87
catchMR back to green.
Looking at my earlier review point that ended up stalling this:
@Wim is correct. The only things I can think of:
1. Adding this as the first parameter to assertWaitOnAjaxRequest(), this would require divining whether the integer is a count or a timeout, but < 25 or < 50 ought to be safe. If it's > that number and no second argument that's also an integer, then issue a deprecation.
2. Add a new method assertAjaxRequestCount($count, $timeout, $message) and just use that where we want to do an assert.
I'm very keen to get this in to (hopefully) stabilise gitlab, but if we add the return now, it's hard to deprecate the method returning anything, unless potentially we do #2 and deprecate the entire old method later, that might be a follow-upable option but could also just do it here.
Comment #88
catchComment #89
nod_Option one just seems terrible, I'd go for option 2 that way we unlock the situation with the return and have a path forward
Comment #90
longwave+1 to option 2. Overloading the existing API doesn't feel like a good idea as there is no clean way to do it and the method name doesn't imply that it can also count requests so discoverability of this is hard. Adding a new method explicitly for counting the requests as well makes sense to me.
Comment #91
alexpottI agree with @nod_ +1 for option 2 as a way forward.
Comment #92
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".
This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #93
nod_all good on the js side for me, thanks
Comment #94
catchI've added a new method,
public function assertExpectedAjaxRequest(?int $count = NULL, $timeout = 10000, $message = 'Unable to complete AJAX request.'): void {This optionally expects a count, and will run the assertion internally if one is provided, otherwise it's exactly the same as the old method.
https://git.drupalcode.org/project/drupal/-/merge_requests/5367/diffs?co... shows the change. I also updated FrameworkTest.
If this looks OK, I can update all of the calls here which are doing assertSame() on the return value.
We could also entirely deprecate the old method and move everything over - don't particularly mind doing that if it helps this land smoothly, but it'd also be a straightforward follow-up.
Comment #95
catchGreen again.
I opened #3401619: Deprecate ::assertWaitOnAjaxRequest() in favour of ::assertExpectedAjaxRequest().
Still some random test failures from #3317520: [random test failure] Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testEditModeEnableDisable but that is not ajax waits and already has its own issue.
Comment #96
nod_All looks good to me
Comment #97
wim leersLooks really good. 🤩
Posted one question. But that could be a follow-up too — it's so clear that this is such a huge improvement to functional JS test reliability thanks to:
👆 Gone will be the false sense of security of having written a test, when actually that test is incorrect at worst or undeterministic at best!
Note: this will impact contrib, this is a theoretical BC break. Some contrib tests will stop working, because they'll run into this exception. But … that'll actually be a good thing, because each of those will be for a wrong "AJAX wait"! They'll be forced to fix it, and they (and d.o's test infra) will be better for it 👍 Tagging and because neither the API change is listed nor do we have a release note ready for module maintainers.
Comment #98
alexpottCan we stop this from breaking contrib and custom tests unless they are doing deprecation testing?
I.e. can we only throw the new error if the test is a core test. And if not trigger a deprecation or something. If someone has a large JS test suite that is running fine on their infrastructure causing it to fail and requiring them to remediate immediately feels not great.
Setting to needs work for #97
Comment #99
wim leersYes, using something like
\Drupal\Core\Config\Schema\SchemaCheckTrait::isContribViolation():Comment #100
catchThis is a docs issue - the code triggers a deprecation, the phpdoc appears to have been pre-emptive (or maybe a holdover from before this triggered a deprecation.
Comment #101
catchComment #102
wim leersI see — my bad! 😅
Comment #103
catchComment #104
catchUpdated the IS and the CR.
Comment #105
catchRemoving the 'needs release note' tag since I think the change record is enough here.
Comment #106
catchAbove wasn't true for the RuntimeException so I have made that conditional on $count being passed in, we can change that to issue a deprecation when $count isn't passed in later.
Comment #107
longwaveLatest run has a fail that looks relevant:
Rerunning the pipeline but this is one of the new tests in AjaxWaitTest getting the opposite error message.
Comment #108
catchAjaxWaitTest needed updating to use ::assertExpectedAjaxRequest() with a $count parameter, otherwise the error message it's looking for doesn't get triggered (due to tweaks here to make it more bc-friendly and not trigger the new exceptions). Pushed a commit that does that.
Comment #109
catchSince the fix was just converting the test to use the new API with the stricter behaviour, after relaxing the behaviour for the old API to maintain bc, moving back to RTBC.
Comment #113
lauriiiCommitted 2a95e3f and pushed to 11.x. Also cherry-picked to 10.2.x. Thanks!
Comment #114
alexpottThis has caused
\Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testBlocks()to fail locally 100% of the time and really really often on gitlabci.Comment #115
alexpottSee https://git.drupalcode.org/project/drupal/-/pipelines/51990 for example.
Comment #116
fjgarlin commentedI got the same last week too. For example: https://git.drupalcode.org/project/drupal/-/merge_requests/5451/pipelines where the only things that were changed in that MR were "test-only" job code. I tried re-running some of the jobs multiple times and kept getting fails.
Comment #118
acbramley commentedJust FYI for anyone else that finds this issue because their assertWaitOnAjaxRequest() calls started failing, you need to now enable
js_testing_ajax_request_testfor any JS test that uses this function.js_testing_ajax_request_test provides test JS code that set
window.drupalActiveXhrCountandwindow.drupalCumulativeXhrCount. Those variables are required to have a successful assertWaitOnAjaxRequest call.Comment #119
nikolay shapovalov commentedAddition to previous comment. You don't need to add
js_testing_ajax_request_testif you extendsWebDriverTestBase.Many contrib modules affected by this change:
And in most cases nobody understand the reason of the fail.
Changing from
assertWaitOnAjaxRequest()toassertExpectedAjaxRequest()doesn't help at all.I would love to have instruction how to handle common fail cases, e.g:
RuntimeException: Unable to complete AJAX request.Comment #120
jsacksick commentedI'm confused by this change... Commerce & Commerce shipping tests are affected as well... Even though the failing tests extend the
CommerceWebDriverTestBaseclass (which extends theWebDriverTestBase).What am I missing? Do we need to not rely on
assertWaitOnAjaxRequest()anymore?Comment #121
bnjmnmassertWaitOnAjaxRequest()now fails if during the waitwindow.drupalCumulativeXhrCountnever exceeds 0, so any uses ofassertWaitOnAjaxRequest()that weren't actually needed will now fail.So for tests that were previously passing and now fail on
assertWaitOnAjaxRequest(), the first thing to try is removing the call entirely. It was very common to add these calls in places where they were not needed, where they were used for waiting on a JS operation to complete that wasn't necessarily something that used the Drupal AJAX API.It is possible removing
assertWaitOnAjaxRequest()will result a different test failure. CallingassertWaitOnAjaxRequest()when no Ajax calls are present still added a 10000ms wait before proceeding, and tests may have depended on that wait. Ideally, if theassertWaitOnAjaxRequest()removal results in a fail, it should be replaced with a wait for the specific expected change.One of the reasons for this change was due to every unnecessary
assertWaitOnAjaxRequest()adding 10000ms per-call to each test run. The more of these removed, the faster you can expect your tests to run.There is a deprecation warning that makes this a little more apparent, but if tests are running with
--suppress-deprecationsit won't be seen. If many contrib projects are running into these failures without seeing that warning, it may be worth opening an issue to have the\RuntimeExceptionmessage be more detailed than "Unable to complete AJAX request" when it finds no AJAX requests have occurred.Comment #122
jsacksick commented@bnjmnm: Thanks a lot for this thorough explanation! This is helpful :).
Comment #123
jordan.jamous commentedIs there a plan to land this issue in 10.3.x?
Comment #124
acbramley commented@jordan.jamous it is in 10.3.0.