Nightwatch tests failing >50% of test runs on PHP 7.3 in 9.4.x and 9.5.x, as well as PHP 8.1 on 10.0.x

as a followup we should also somehow kill the test if the setup fails instead of going ahead.

So, a few things from infra perspective -> containers / images shouldnt really have any bearing on underlying hardware/resource contention, *unless* the container itself is somehow hamstrung from accessing the full resources. When the nightwatch tests are running, there is nothing else at all happening on the machine, and thus no contention at all. Nightwatch tests run serially if I remember correctly, so its not an issue of parallelism either.

In this case it could be that the mariadb container isnt properly configured to take advantage of the massive amounts of memory/disk/cpu available. The configurations for those containers were just what I received from a community member, so dont know if there is more performance to be gained there, or if there is some other issue.

The image itself is not new, it was just recently added as another daily test. The image was added in mar, 2020.

So, Im not sure how we can test if mariadb is underperforming/underconfigured.

For reference, the tests are running on a C4 8XL AWS instance (or one of a very comparable one in the pool) which has 32vCPUs/64gigs of RAM. And as @mixologic says, the nightwatch tests run serially, so they shouldn't be constrained by the instance itself.

We think it was maybe @daffie who helped get the MariaDB container set up. Perhaps they would have some ideas here.

MariaDB is saying that they fixed some PHP PDO stuff in version 10.3.27 and we are using version 10.3.22. See: https://mariadb.com/kb/en/mariadb-10327-release-notes/
Maybe update MariaDB to a newer version and see if it fixes the problem.

It's not just MariaDB though... see https://www.drupal.org/pift-ci-job/2388848 - same fail and MySQL.

It's something that has changed on 9.4.x that is not on 9.3.x because it does not seem to be happening at all there - you can see this if you look at the build history on https://www.drupal.org/node/3060/qa - compare https://www.drupal.org/pift-ci-job/2388394 and https://www.drupal.org/pift-ci-job/2388902 - these are both PHP 8.0 and MySQL 5.7 - so exactly the same build environment.

Let's see if we can get Nightwatch to dump the page when it fails to install the module.

Lol - I thought I'd already added the missing semi-colon...

#15 looks promising, but there are two Toolbar tests with different fails in its result set:

• https://dispatcher.drupalci.org/job/drupal_patches/130202/console
• https://dispatcher.drupalci.org/job/drupal_patches/130205/console

We should consider just deploying that fix and then filing followups for the different fails.

#15 didn't give us the info I was looking for... trying to work out why...

Hmmm... I wonder if this is all about the .drupalInstallModule() - when Nightwatch was added we created the ability to pass a PHP script to the installer to do tasks like this. I wonder what happens if we move this there.

Missed the toolbarApiTest.js which does the same thing.

@Wim Leers the problem is all in core/tests/Drupal/Nightwatch/Commands/drupalInstallModule.js - that's proved by #21 which does the module install a different way. But looking at this code. It's not correct. It's waiting for something that is already on the page when it presses the submit button. We can improve this considerably.

I think the attached patch will fix the problem.

@Wim Leers yarn.lock is not currently updated by running a yarn install - so that's not the issue here. I'm not sure about adding --frozen-lockfile yarn respects the lock file in nearly all cases anyway.

What about chromedriver?

I'm pretty sure that the nightwatch tests on DrupalCI are not using the chromedriver in core/node_modules... fwiw. See the DrupalCI logs...

15:46:41 Starting chromedriver container ...
15:46:41 Adding image: drupalci/chromedriver:production
15:46:42 Pulling from drupalci/chromedriver
15:46:42 Digest: sha256:2294a42e164243ae06e617b9c14039613dc69320cc269067389c79c437440fbf
15:46:42 Status: Image is up to date for drupalci/chromedriver:production
15:46:42 
15:46:42 Container chromedriver-jenkins-drupal-patches-130303 created from image drupalci/chromedriver:production with ID 9bad60a6

15:48:40 - Connecting to chromedriver-jenkins-drupal-patches-130303 on port 9515...
15:48:40 
15:48:40 ℹ Connected to chromedriver-jenkins-drupal-patches-130303 on port 9515 (412ms).
15:48:40   Using: chrome (98.0.4758.102) on Linux platform.

This is the chromedriver being used by Nightwatch - the version brought down by yarn is ignored.

+++ b/core/tests/Drupal/Nightwatch/Commands/drupalInstallModule.js
@@ -17,8 +17,12 @@ exports.command = function drupalInstallModule(module, callback) {
-      // Wait for the install message to show up.
-      .waitForElementVisible('.system-modules', 10000);
+      // Wait for the checkbox for the module to be disabled as a sign that the
+      // module has been enabled.
+      .waitForElementVisible(
+        `[name="modules[${module}][enable]"]:disabled`,
+        10000,
+      );

The errors are happening because the .waitForElementVisible('.system-modules', 10000); does not work. On really fast computers that assert is able to complete before the browser has refreshed after clicking on the submit. So we have the opposite of @Wim Leers performance suspicions. If we wait for something that can only be true after the module install has completed that this will work. The reason for the different errors is that sometimes the breakpoint install was being interrupted and sometimes the toolbar was so we'd see slightly different errors after this has occurred. Fun stuff. This will also resolve probable random errors in core/tests/Drupal/Nightwatch/Tests/touchDetectionTest.js as that uses drupalInstallModule() too.

Ah we need to account for the fact that testing modules are hidden. That's not a problem.

And here's just the fix on itself without the changes to drupalci.yml.

ooh I see, very nice find, thanks :)

#29 shows that we still have the problem with #30. The more specific assert narrows it down to drupalInstallModule() in both https://www.drupal.org/pift-ci-job/2389713 and https://www.drupal.org/pift-ci-job/2389714

So options are to do #21 and use a PHP site setup script to do the module enabling and we should deprecate drupalInstallModule() because it is broken. This was introduced in #3255809: Add nightwatch tests for toolbar - so we still have the question as to why 9.3.x does not seem to be as affected.

Let's do two things before we fallback to PHP. Let's make all the data input and clicking and waiting specific to the form.system-modules. I'm pretty sure that is not an issue here but it help make drupalInstallModules more robust. And let's do a little pause after pressing submit. I don;t think there is much harm in that here and if it results in it being fixed that feels acceptable.

+++ b/core/tests/Drupal/Nightwatch/Commands/drupalInstallModule.js
@@ -13,12 +13,25 @@ exports.command = function drupalInstallModule(module, callback) {
+      .click('form.system-modules input[data-drupal-selector="edit-submit"]')

don't remember why I didn't use submit form here: https://nightwatchjs.org/api/submitForm.html

Let's use submitForm() and let's also add some instrumentation - I think I understand why the original effort did not work... you have to tell the wait not to abort on error because you want the callback to fire.

Okay submitForm() got a sea of green - yay... triggering the tests again. If it is another sea of green I think we should proceed here.

So 9.3.x is not failing because for some reason core/modules/toolbar/tests/src/Nightwatch/Tests/toolbarApiTest.js does not use drupalInstallModule on that branch. In fact drupalInstallModule does not exist on 9.3.x so this is all down to how #3255809: Add nightwatch tests for toolbar was committed.

This is incorrect. Still can't explain 9.3.x vs 9.4.x

So one difference between 9.4.x and 9.3.x is that #3272727: Nightwatch's drupalModuleInstall() doesn't handle test modules or modules w underscores in machine name - but that was 7 weeks ago and I don't think this has been happening for 7 weeks.

Anyways #36 is a sea of green after 2 goes so the problem is fixed - yay!

+++ b/core/tests/Drupal/Nightwatch/Commands/drupalInstallModule.js
@@ -13,12 +13,35 @@ exports.command = function drupalInstallModule(module, callback) {
+      .click(`form.system-modules [name="modules[${module}][enable]"]`)
+      .submitForm('form.system-modules')
+      // Ensure module enabling has a chance to start before asserting that it
+      // is finished.
+      // .pause(50)
+      // Wait for the checkbox for the module to be disabled as a sign that the
+      // module has been enabled.
+      .waitForElementPresent(
+        `form.system-modules [name="modules[${module}][enable]"]:disabled`,

So the fix seems to be changing to submitForm() and making the wait something actually specific that has changed due to enabling a module.

Here's a patch the has just the necessary stuff to fix this.

Rtbc+1

Committed/pushed to 10.0.x, cherry-picked to 9.5.x and 9.4.x, thanks!