Closed (fixed)
Project:
Drupal core
Version:
10.1.x-dev
Component:
ckeditor5.module
Priority:
Critical
Category:
Bug report
Assigned:
Unassigned
Reporter:
Created:
14 Oct 2022 at 08:30 UTC
Updated:
1 Nov 2022 at 15:34 UTC
Jump to comment: Most recent, Most recent file
Comments
Comment #2
wim leers🕵️
Comment #3
catchPossibility that #3315227: Drupal\Tests\views\FunctionalJavascript\Plugin\views\Handler\FilterTest is failing a lot at the moment also helps with this test, we might want to commit that first and see what the effect is.
Comment #4
alexpottThis one definitely happens with the fix from #3315227: Drupal\Tests\views\FunctionalJavascript\Plugin\views\Handler\FilterTest is failing a lot at the moment applied.
Comment #5
wim leersDid anything change recently that would cause this to fail now, after it’s not been failing for about a year? (This was introduced in #3231364: Add CKEditor 5 module to Drupal core and was never changed.)
I agree that this looks suspicious: the comment is saying that it's verifying there is NO AJAX rebuild of the form, but then it proceeds to wait for it anyway 🤔
The code you've cited is from line 57, not 97, which is where the fail originated. But perhaps there were multiple failures? According to https://www.drupal.org/pift-ci-job/2497704, it really was just line 97?
I've discovered that many of the waits in the code you cited are basically leftovers to be able to reproduce a problem; they're not actually necessary, plus some of the checks literally are waiting for AJAX responses that are pointless! But I have not
Removing all the unnecessary waits cuts down the test time from 1 minute to 22 seconds on my machine.
Next up to investigate: I suspect that they/we thought
$assert_session->responseContains('Media types selectable in the Media Library');would check the AJAX response … but it does not: it checks the page rendered in the browser, using the DOM… to be continued.Comment #6
alexpottIf you change --testgroups in drupalci.yml to
testgroups: '--repeat 50 --class "\Drupal\Tests\ckeditor5\FunctionalJavascript\AdminUiTest"'then we can really zero on the random fail.Comment #7
alexpott$this->assertNotEmpty($assert_session->waitForElement('css', '.ajax-progress-throbber'));this assert does seem dangerous as the ajax will come and go on the page.I don't think we should be asserting for the absence of ajax occurring ever... I think we should remove all the
$assert_session->assertNoAjaxRequestsTriggered();and ensure that the page as content the we expect after doing whatever we've done via the browser.Comment #8
wim leersComment #9
wim leers#7 Yep, I reached the same conclusion about the throbber. But I disagree with your second statement. Verifying that no AJAX request was triggered is literally the purpose of that test!
I think you're going to like this 🤓
Comment #10
wim leersForgot to include this clean-up in #9, but this is functionally equivalent.
Comment #12
wim leersOkay, both #9 and #10 passed, which means 100 test runs without a failure. I think that means this is a net stabilizer.
Next:
I'm 95% certain now that this is the case that these were intended to check that the AJAX response contained that string.
But none of these make sense. Because it is indeed asserting the state of the page, not of the AJAX response. And that string is already present before the AJAX response too, because it is simply shown/hidden by
Drupal.behaviors.filterStatus😅So, we can simplify those too. Doing that here. Expanding to run all CKE5 tests again, but only once.
Comment #13
alexpottWhy is this not all an $this->assertWaitOnAjaxRequest()?
This looks like lots of duplication.
Comment #14
wim leersHah, in an earlier iteration it was rather different, I had not noticed I ended up with the same exact thing 😅
You're totally right.
Deduplicated 👍
Comment #15
alexpottCan you adjust the test groups of the three tests altered here and then run that group with --repeat 50?
Just to make sure we're not introducing any randoms.
This looks really good. Nice sleuthing and adding robustness.
Comment #16
spokjeComment #17
spokjeComment #18
spokjeComment #20
alexpottHere's what I meant... we need to put the tests in their own group...
Comment #22
alexpottSo we have a random fail from line 329 just after these lines.
Comment #23
wim leersComment #24
wim leersThis should do the trick. 🤞 I pretty much rewrote
\Drupal\Tests\ckeditor5\FunctionalJavascript\CKEditor5Test::testActiveTabsMaintained()to avoid a huge amount of repetition, and increase legibility.For context: that test was introduced in #3218906: The active "CKEditor 5 plugin settings" vertical tab must not lose focus, and seems to have contained some erroneous comments since day 1 — which I missed during reviews. This was back in the early days of stabilizing the CKEditor 5 module, so the bar was not quite at the core level yet — the focus was on progress, because nothing was set in stone yet.
I figured it's worth settling this once and forever. So I went deep to ensure I get this right, in particular wrt
core/modules/filter/filter.admin.jshaving rather weird behavior. We're not really testing the behavior of those vertical tabs here; rather we're verifying that CKEditor 5's AJAX updates do not break that behavior. 👍Comment #25
wim leersHere's some commentary of the #24 interdiff to make it easier to understand:
New helper, makes it easy to do
assertSame([…], $this→getVerticalTabs(…))calls, which HUGELY improves test legibility and maintainability.New helper that also improves legibility.
Now we start with assessing the initial state, prior to doing ANY interactions.
Note that the plugin's vertical tab is visible right from the start! This was not the case at the time of #3218906: The active "CKEditor 5 plugin settings" vertical tab must not lose focus, which will come up later.
This is the piece that was just plain wrong/nonsensical. The Heading plugin was already active as described in an earlier bullet.
The reason this failed silently is because we were just asserting the presence of the "heading" toolbar item, triggering the "arrow down" keyboard button press and … observing that the "Headings" toolbar item was then present. But … it was present since the very beginning… 😅
This is plain wrong, because the Filters plugins settings vertical tabs have … very peculiar behavior (see
core/modules/filter/filter.admin.js), and actually the assertions were even in direct contradiction with this comment.Here's the direct contradiction: it's the THIRD vertical tab that's selected by default, not the first.
What complicates this further is that this was just checking ALL filter plugin settings vertical tabs, not just the visible ones.
All of that complexity/vagueness is now made explicit. 👍
Comment #27
wim leersThis actually seems pretty simple to solve … the root cause seems to be that in this particular case, we're dynamically getting the visible vertical tabs … but while we do that, the vertical tabs may be getting overwritten. And … that's true! I just wanted to verify the vertical tabs state before the AJAX request, but after checking a filter's checkbox. In hindsight, it makes sense that this causes a race condition. So let's just do the assertion after the AJAX request has been processed, to avoid this race condition.
Also, since the other functional JS tests have been passing consistently, limiting this to just the one failing test.
Comment #28
wim leersGreen! Let's run it 100 times, not 50. Just to make sure.
Comment #29
wim leersThe scope grew in #24, where we discovered additional random failures.
Comment #30
wim leers59/100 are now green, I think it's time to run the entire test suite now.
Comment #32
wim leersRandom fail in
SettingsTrayBlockFormTest. Unrelated and out of scope.Comment #33
lauriiiNot sure if I have been involved in this but I thought this was to guarantee that the page contains "Media types selectable in the Media Library" after the ajax request has completed to make sure that the right ajax request was triggered. Looking at it now, it seems potentially of out of scope to test there so I guess it's fine to remove those assertions.
I like what this helper function is doing. 💯 It makes the test cases in
::testActiveTabsMaintainedso much easier to follow.Nit: probably doesn't matter here but should we use strict comparison?
Why are we doing this only when retrieving visible vertical tabs? 🤔
Comment #34
wim leers#33:
Comment #35
lauriii#34.1 & 4 👍 makes sense
I think based on that, this is ready for a review from another committer.
Comment #36
alexpottBeen following this one closely - nice work @Wim Leers and nice review @lauriii
Committed and pushed c0d535a9b5 to 10.1.x and 44fab238e9 to 10.0.x and b038750f0b to 9.5.x and 0ad0f52f7e to 9.4.x. Thanks!