Problem/Motivation

In #3267823: \Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest::testCustomBlock(). is failing on latest chromedriver we've disabled \Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest::testCustomBlock(). Also we've seen an increase of "random" test failures on \Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest::testArticleNode() (https://www.drupal.org/pift-ci-job/2333198).

Steps to reproduce

Proposed resolution

Looks like our JavaScript testing is "rather" optimistic. We look for "something" once and be done with it.
Bad thing is, when "the thing" is not there at that time, TestBot is done with us...

It seems to work when when we wrap "looking for something" in the waitFor..() from \Drupal\FunctionalJavascriptTests\JSWebAssert, which does retries until a certain timeout.

Remaining tasks

Patch
Review
Commit

Issue fork drupal-3268010

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

Spokje created an issue. See original summary.

spokje’s picture

StatusFileSize
new11.12 KB
spokje’s picture

Let's see if a timeout of 15.000ms helps.

spokje’s picture

So, running \Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest 3 x 100 times with a time out of 10.000 and 15.000ms on both D10 and D9.4 gives us:

10.000ms:

D9.4: 0 failures
D10.0: 3 failures

15.000ms:

D9.4: 0 failures
D10.0: 4 failures

Which in my mind means:

- Upping the timeout in the ::waitForElementRemoved() in ::testArticleNode() doesn't seem to help.
- Don't upgrade to D10.0

If a 1/100 test fail is acceptable in ::testArticleNode() I'll leave for Bigger Brains to decide.
::testCustomBlock() seems 100% OK after this change.

The MR uses the standard 10.000ms time out.

spokje’s picture

Assigned: spokje » Unassigned
Status: Active » Needs review
spokje’s picture

Note: The above numbers were measured at a Sunday afternoon (at least in my Time Zone) when there were hardly any core tests running.

The failures that show up in the HEAD auto-testing are measured when TestBot is extremely busy: around 5 PHP/DB combos for 2/3 branches (D10/D9.4/D9.3) all at the running at the same time triggered by a Core commit.

Let's see if my numbers change dramatically when it's a bit more busy in about 24hrs or so.

At the very least, I believe we've got the real cause for ::testCustomBlock() fixed.

xjm’s picture

So what we should do here is:

  1. Get the base fail rate of the test (with testCustomBlock() still skipped).
  2. Get the fail rate of the patch (with testCustomBlock() still skipped).
  3. Repeat that for testCustomBlock() only.

We should also be doing a higher number of test runs at a time, and always run the baseline at the same time as the patch to make sure there's an improvement.

I'll post a couple patches based on @Spokje's.

xjm’s picture

Here those are. Let's see if 500x still manages to complete. I left moving the method out of LB so that CI wouldn't try to skip testing the patch for lack of code changes.

xjm’s picture

BTW, testArticle() was one of the many JS tests involved in the fail-explosion late last night. It accounted for 6 of 55 random fails that happened over the course of three pushed commits + nightly testing.

xjm’s picture

Those PostgreSQL results are also the database server going away.

xjm’s picture

So it's pretty clear that the changes to testArticle() aren't really helping that test (also, that our SQLite and PostreSQL environments are having serious problems). Now I'll switch it around, skip testArticle(), and do the same with testCustomBlock().

xjm’s picture

xjm’s picture

Oops, bugged patches. Trying again.

xjm’s picture

xjm’s picture

OK, now that is compelling. (Ignore the PostgreSQL and SQLite results; those environments appear to be broken ATM.)

I'll queue a second set of MySQL/Maria test runs, but it's pretty clear to me that we should only fix testCustomBlock() here, remove all changes in testArticleNode(), and open a followup for it. (I am proposing marking testArticleNode() skipped in #3268070: Temporarily skip even more failing tests; it had a low fail rate by itself earlier but it failed again in HEAD in the past couple hours.)

xjm’s picture

xjm’s picture

Title: Restore/Robustify \Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest » Restore/Robustify \Drupal\Tests\quickedit\FunctionalJavascript\QuickEditIntegrationTest::testCustomBlock()
xjm’s picture

quietone’s picture

Wow, that appears to be a working solution. thanks, xjm.

I read the MR and it looks fine to me, the comments clearly explain the change, Unfortunately, I don't know enough of javascript and it's testing to RTBC this.

spokje’s picture

Issue tags: +Needs followup

If/When this lands into Core, we also have to commit this to the Contrib incarnation of quickedit.

bnjmnm’s picture

Status: Needs review » Needs work

Looking at the fix, I am a little concerned that the approach is basically "if this unpredictable intermittent thing happens, try it a different way". Ideally, this would be a solution that is more reliable in general, vs a fallback for expected unreliability. Could this potentially be accomplished with a helper module that makes the sometimes-missing .quickedit-toolbar-label > .field is visible at all times, or perhaps using executeScriptDrupal.quickedit.(path to the model that needs to be openeed) to directly request the behavior via JS instead of relying on the inconsistent UI behavior?

And to qualify the above I wanted to mention it... but I don't feel all that strongly about it. I do think the existing approach is fine. Considering that QuickEdit is leaving core soon anyway + that there are huge benefits in having random failures reduced, I'd be fine RTBCing if there's evidence the suggested approaches aren't feasible or require too much effort for theoretical benefits.

spokje’s picture

Status: Needs work » Closed (outdated)

quietone’s picture

Issue tags: -Needs followup

I don't see that the followup needs to be made.