Random failure in FormErrorHandlerCKEditorTest

Looks like this test has failed more than 10 times in HEAD over the past month in both branches, starting as far back as Feb. 12 at least:
https://www.drupal.org/pift-ci-job/597238
https://www.drupal.org/pift-ci-job/615344
etc.

Quick patch to see if it works.

Ouch. I made a simple stupid mistake

Cool. It works. Full patch.

+++ b/core/scripts/run-tests.sh
@@ -142,6 +142,14 @@
+  // Do the test 25 times.

The failure rate doesn't seem to be very high so is running this 25 times enough to make us confident we have a fix?

Fix is a little surprising, since the error link is rendered on page load, why would you ever need to wait for it? Am I missing something?

I've seen this random fail quite often lately.

@Lendude,

The failure rate doesn't seem to be very high so is running this 25 times enough to make us confident we have a fix?

this is enough. I triggered more tests.

    $this->submitForm($edit, 'Save and publish');

I guess there's a hidden problem on this line.

I'd suggest to commit #8 first and do further debugging soon.

That's not enough test runs, actually. That's only 325 test runs and this is probably on the order of 1/1000 fail. (Even if it were 1/100, 325 runs is not enough to conclusively demonstrate a fix.)

What we need to do is have a patch that has only the repeated test runs, to try to prove the number of test runs is high enough to catch the random fail when combined with the fix. I know the JS tests take a long time so there's a risk of timing out. These patches are just a starter to see if 100 or 200 test runs will complete. Based on that, we can then queue enough runs to prove it will fail for the patch that completes, and then combine that with @droplet's fix to demonstrate the fix is actually fixing.

200 ran in 30 mins on PHP 5.6, so looks like we can dial that up. If this one completes, then we run it against PHP7/MySQL 5.5 lots of times. (Running against other DBs is not necessary here.)

So the fact that the 400-run test passed demonstrated that 325 runs is not enough to demonstrate a fix.

I've queued the 400-run test-only patch many times. If none of them fail, then there are other factors influencing the fail rate (i.e. it's not just random) and so having multiple runs does not demonstrate coverage by itself anyway.

Okay, there we go. So now we combine that count with @droplet's fix to demonstrate whether it's fixing it.

Hooray, looking good so far. So here's @droplet's full patch again, and RTBC assuming none of the remaining runs in #17 show the fail.

I've checked the latest error log and I can't identify what problem that is.

Page seems loaded but there's a different log:

However, the next line is executed and no errors:

    $session->executeScript("document.getElementById('edit-title-0-value').style.marginBottom = '800px';");

the error message HTML is rendered before title field.

and the next different part is:

// Check that the CKEditor-enabled body field is currently not visible in
    // the viewport.
    $web_assert->assertNotVisibleInViewport('css', $ckeditor_id, 'topLeft', 'CKEditor-enabled body field is not visible.');

I guess that the CKEDITOR is not fully initialized. It's similar to #2843693: Random test failure in CKEditor AjaxCss. But, it can't explain why it will blocking the message checking.

So far, unless there's a hidden bug. The patch provided by me still the way to fix above 2 loading issues.

Thanks All.

Committed and pushed 9a63547 to 8.4.x and 7e39aca to 8.3.x. Thanks!

But, it can't explain why it will blocking the message checking.

I'm happy to see this in, but I'm not happy that we haven't figured out why this is necessary. The need for waiting for a non-dynamic element to appear on page load feels weird, and I'd like to know when we would need to do this in other/future tests so we don't run into this again and again.

Or am I missing something obvious?

@Lendude, best to file a followup issue? I know there was a discussion in Baltimore that I missed.

Moving the parent and fix issues to keep the list clean. Thanks!