Problem/Motivation
In #3267823: \Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest::testCustomBlock(). is failing on latest chromedriver we've disabled \Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest::testCustomBlock(). Also we've seen an increase of "random" test failures on \Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest::testArticleNode() (https://www.drupal.org/pift-ci-job/2333198).
Steps to reproduce
Proposed resolution
Looks like our JavaScript testing is "rather" optimistic. We look for "something" once and be done with it.
Bad thing is, when "the thing" is not there at that time, TestBot is done with us...
It seems to work when when we wrap "looking for something" in the waitFor..() from \Drupal\FunctionalJavascriptTests\JSWebAssert, which does retries until a certain timeout.
Remaining tasks
Patch
Review
Commit
| Comment | File | Size | Author |
|---|---|---|---|
| #15 | QuickEditIntegrationTest_no_article_500x.patch | 9.27 KB | xjm |
| #15 | QuickEditIntegrationTest_no_article_baseline_500x.patch | 5.64 KB | xjm |
Issue fork drupal-3268010
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
spokjeComment #3
spokjeLet's see if a timeout of 15.000ms helps.
Comment #5
spokjeSo, running
\Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest3 x 100 times with a time out of 10.000 and 15.000ms on both D10 and D9.4 gives us:10.000ms:
D9.4: 0 failures
D10.0: 3 failures
15.000ms:
D9.4: 0 failures
D10.0: 4 failures
Which in my mind means:
- Upping the timeout in the
::waitForElementRemoved()in::testArticleNode()doesn't seem to help.-
Don't upgrade to D10.0If a 1/100 test fail is acceptable in
::testArticleNode()I'll leave for Bigger Brains to decide.::testCustomBlock()seems 100% OK after this change.The MR uses the standard 10.000ms time out.
Comment #6
spokjeComment #7
spokjeNote: The above numbers were measured at a Sunday afternoon (at least in my Time Zone) when there were hardly any core tests running.
The failures that show up in the HEAD auto-testing are measured when TestBot is extremely busy: around 5 PHP/DB combos for 2/3 branches (D10/D9.4/D9.3) all at the running at the same time triggered by a Core commit.
Let's see if my numbers change dramatically when it's a bit more busy in about 24hrs or so.
At the very least, I believe we've got the real cause for
::testCustomBlock()fixed.Comment #8
xjmSo what we should do here is:
testCustomBlock()still skipped).testCustomBlock()still skipped).testCustomBlock()only.We should also be doing a higher number of test runs at a time, and always run the baseline at the same time as the patch to make sure there's an improvement.
I'll post a couple patches based on @Spokje's.
Comment #9
xjmHere those are. Let's see if 500x still manages to complete. I left moving the method out of LB so that CI wouldn't try to skip testing the patch for lack of code changes.
Comment #10
xjmBTW,
testArticle()was one of the many JS tests involved in the fail-explosion late last night. It accounted for 6 of 55 random fails that happened over the course of three pushed commits + nightly testing.Comment #11
xjmThose PostgreSQL results are also the database server going away.
Comment #12
xjmSo it's pretty clear that the changes to
testArticle()aren't really helping that test (also, that our SQLite and PostreSQL environments are having serious problems). Now I'll switch it around, skiptestArticle(), and do the same withtestCustomBlock().Comment #13
xjmComment #14
xjmOops, bugged patches. Trying again.
Comment #15
xjmComment #17
xjmOK, now that is compelling. (Ignore the PostgreSQL and SQLite results; those environments appear to be broken ATM.)
I'll queue a second set of MySQL/Maria test runs, but it's pretty clear to me that we should only fix
testCustomBlock()here, remove all changes intestArticleNode(), and open a followup for it. (I am proposing markingtestArticleNode()skipped in #3268070: Temporarily skip even more failing tests; it had a low fail rate by itself earlier but it failed again in HEAD in the past couple hours.)Comment #18
xjmComment #19
xjmComment #20
xjmI filed #3268244: [random test failure] Un-skip and fix QuickEditIntegrationTest::testArticleNode() for the other method.
Comment #21
quietone commentedWow, that appears to be a working solution. thanks, xjm.
I read the MR and it looks fine to me, the comments clearly explain the change, Unfortunately, I don't know enough of javascript and it's testing to RTBC this.
Comment #22
spokjeIf/When this lands into Core, we also have to commit this to the Contrib incarnation of quickedit.
Comment #23
bnjmnmLooking at the fix, I am a little concerned that the approach is basically "if this unpredictable intermittent thing happens, try it a different way". Ideally, this would be a solution that is more reliable in general, vs a fallback for expected unreliability. Could this potentially be accomplished with a helper module that makes the sometimes-missing
.quickedit-toolbar-label > .fieldis visible at all times, or perhaps usingexecuteScriptDrupal.quickedit.(path to the model that needs to be openeed)to directly request the behavior via JS instead of relying on the inconsistent UI behavior?And to qualify the above I wanted to mention it... but I don't feel all that strongly about it. I do think the existing approach is fine. Considering that QuickEdit is leaving core soon anyway + that there are huge benefits in having random failures reduced, I'd be fine RTBCing if there's evidence the suggested approaches aren't feasible or require too much effort for theoretical benefits.
Comment #24
spokjeFixed in #3267258: Remove Quick Edit support from editor.module
Comment #26
quietone commentedI don't see that the followup needs to be made.