Improve the stability of core JS testing and prepare for update of MinkSelenium2Driver

Some more fixes.

Wow just found out that in webdriver we can set an implicit wait for things. This I think is a great idea that will make our tests way more reliable.

#6 is gonna fail lets bump the wait to 1second.

For more info on the setTimeouts thing see https://developer.mozilla.org/en-US/docs/Web/WebDriver/Commands/SetTimeouts

Bump the timeout to 2secs (note the wait timeout is 10secs) and fix a few more colons.

@andypost it's likely that these changes won't be required - just recommended - see https://github.com/minkphp/MinkSelenium2Driver/issues/324#issuecomment-7... and https://github.com/minkphp/MinkSelenium2Driver/pull/325

So let's see if the implicit wait is actually doing anything...

Moving stuff around to see if we can make the quickedit test more reliable. Nearly all of our problems are caused by assertWaitOnAjaxRequest

Ignore the Postgres 9.5 test in #25 D9 does not support that version of Postgres.

Let's get rid of all assertWaitOnAjax... in /QuickEditLoadingTest

Let's repeat all the tests to tease out random fails.

I can reproduce the postgres fails locally and the patch attached seems to fix them.

Interesting new fails not on postgres... let's try to see what is going on.

Here's one more random fail fixed. Postgres sometimes is just a bit slow to load. It fails locally for me everytime without this.

I still can't explain all the fails in non-postgres environments in #29

Let's see...

So #35 is way way more stable (less random fails) then anything previously but there's still at least one random fail in \Drupal\Tests\media_library\FunctionalJavascript\WidgetUploadTest::testWidgetUpload - see https://www.drupal.org/pift-ci-job/1847459 - the html output makes it look like the browser didn't register the click on 'media-3-remove-button' - all I can suspect is that the callback failed for some reason and we're not getting the JS error. So I've added some handling to try to extract JS errors based on https://github.com/minkphp/MinkSelenium2Driver/issues/189

So it seems some of tests get JS errors and they are expected... so we can't always test there a no errors but this does prove that a JS error while AJAXing doesn't automatically cause a fail and so might the source of the remaining random fails...

I've tried everything to find out why \Drupal\Tests\media_library\FunctionalJavascript\WidgetUploadTest::testWidgetUpload() fails sometimes on SQLite. The point of failure seems to have moved from what it outlined in #3066447: [random test failure] Random failures building media library form after uploading image (WidgetUploadTest) but it does seem related. It looks like the button presses sometimes don't not do anything. The two in question are $page->pressButton('media-1-remove-button'); on line 313 and $page->pressButton('media-3-remove-button'); on line 335. I think it is okay to leave this to be fixed in #3066447: [random test failure] Random failures building media library form after uploading image (WidgetUploadTest) - as the recent runs show this is only with SQLite and it very rare.

This issue started about making our tests work with the latest dev version of behat/mink-selenium2-driver but as that changed timeouts it quickly became about the stability of our JS tests. Therefore I'm going to split the scope of this issue and make it solely about JS test stability. I'll open a new issue about JS test conditions and removing the semi-colon.

Here's a patch to do repeat JS testing on HEAD

Here's the fixes from #40 plus repeat testing.

Fixed the issue summary. Opened #3176361: Remove semi-colon from javascript test conditions for the JS conditions.

@andypost what's odd is that the fail is DB specific. I suspect that the fail is due to the concurrency and SQLite. Drupal's use of SQLite does not handle concurrency that well.

Here's the final patch that I think we should consider committing. It:

• doesn't make any changes to run-time code
• waits more appropriately for things
• is more stable on dev-master of behat/mink-selenium2-driver
• saves the html output when there is any error giving more info to go on when debugging

Forgot to upload the patch.

Attaching to the PHP 8.0 meta. The package maintainer will hopefully make a release soon so we need this done.

Making critical as this will block updating because JS testing will become completely unstable.

@Lendude I agree that #3061852: [META] Deprecate assertWaitOnAjaxRequest() and make the JsWebAssert::waitFor*() methods behave like real assertions is desirable but I don't think that this blocks us doing this issue to prep for updating and also contrib can lower the dependency and not support PHP8 until they've fixed their tests. We won't change the constraint.

Also this issue does not wreck havoc itself. This issue will make contrib testing more stable due to improvements in the general test code. It's the update of the driver that might... however I think the general improvements will mitigate that.

@catch - yep it is #3118591: Datetime-related test failures on PostgreSQL 12 - it is only postgres 12. We test with 10 by default.