Update Nightwatch to 3.x

We are having trouble upgrading Nightwatch past 2.4.2 already, see #3323988: Update Nightwatch from 2.4.2 to 2.6.19 - whatever is broken there needs fixing first.

This change will only be committed to 11.x (10.2.x) and will not be backported to 10.1.x. 10.0.x or any version of 9.x.

I was able to get nightwatch 3.0.1 running and have submitted a merge PR. But several tests were failing. I am trying to setup a local environment to try and debug the tests, till now I have not been able to setup a test environment.

The domain setting in cookies does not work the same way when w3c mode is enabled, removing it from the SIMPLETEST_USER_AGENT cookie works locally for me.

Crediting contributors to #3323988: Update Nightwatch from 2.4.2 to 2.6.19

DrupalCI doesn't seem to pick up on the five NightwatchAssertErrors I see in the full test log.

Adding Needs followup for that.
Even whilst the full focus is, rightfully so, on the GitLab move, having a green TestBot whilst there are errors is bad.

00:40:15.270   ✖ NightwatchAssertError
00:40:15.270    Timed out while waiting for element <[href*=edit-editor-settings-plugins-ckeditor5-sourceediting]> to be present for 5000 milliseconds. - expected "visible" but got: "not found" (5082ms)
00:40:15.270 
00:40:15.270     Error location:
00:40:15.270     /var/www/html/core/modules/ckeditor5/tests/src/Nightwatch/Tests/ckEditor5CodeSyntaxTest.js:
00:40:15.270     ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:40:15.270      36 |         .keys(browser.Keys.DOWN) // Hit the down arrow key to move it to the toolbar.
00:40:15.270      37 |         // Wait for new source editing vertical tab to be present before continuing.
00:40:15.270      38 |         .waitForElementVisible( 
00:40:15.270      39 |           '[href*=edit-editor-settings-plugins-ckeditor5-sourceediting]',
00:40:15.270      40 |         )
00:40:15.270     ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:40:15.391   FAILED: 1 assertions failed and  5 passed (8.962s)


00:40:26.978    Timed out while waiting for element <[href*=edit-editor-settings-plugins-ckeditor5-sourceediting]> to be present for 5000 milliseconds. - expected "visible" but got: "not found" (5088ms)
00:40:26.978 
00:40:26.978     Error location:
00:40:26.978     /var/www/html/core/modules/ckeditor5/tests/src/Nightwatch/Tests/ckEditor5EditorHeightTest.js:
00:40:26.978     ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:40:26.978      38 |         .keys(browser.Keys.DOWN) // Hit the down arrow key to move it to the toolbar.
00:40:26.978      39 |         // Wait for new source editing vertical tab to be present before continuing.
00:40:26.978      40 |         .waitForElementVisible( 
00:40:26.978      41 |           '[href*=edit-editor-settings-plugins-ckeditor5-sourceediting]',
00:40:26.978      42 |         )
00:40:26.978     ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:40:26.978 
00:40:27.196 
00:40:27.196   FAILED: 1 assertions failed and  5 passed (9.067s)

00:41:54.180   ✖ NightwatchAssertError
00:41:54.180    Failed [ok]: (The expression evaluated to a falsy value:
00:41:54.180 
00:41:54.180   assertModule[propName].apply(null, this.args)
00:41:54.180 ) - expected "true" but got: "false" (0ms)
00:41:54.180 
00:41:54.180     Error location:
00:41:54.180     /var/www/html/core/tests/Drupal/Nightwatch/Tests/Olivero/oliveroMobileMenuTest.js:
00:41:54.180     –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:41:54.180      117 |         [linkSubMenuId],
00:41:54.180      118 |         (result) => {
00:41:54.180      119 |           browser.assert.ok(result.value); 
00:41:54.180      120 |         },
00:41:54.180      121 |       )
00:41:54.180     –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:41:54.180 
00:41:54.334 
00:41:54.334   FAILED: 1 assertions failed and  4 passed (1.001s)


00:42:50.917   ✖ NightwatchAssertError
00:42:50.917    Timed out while waiting for element <.block-search-wide__wrapper> to not be visible for 5000 milliseconds. - expected "not visible" but got: "visible" (5114ms)
00:42:50.917 
00:42:50.917     Error location:
00:42:50.917     /var/www/html/core/tests/Drupal/Nightwatch/Tests/Olivero/oliveroSearchFormTest.js:
00:42:50.917     –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:42:50.917      81 |       .keys(browser.Keys.TAB)
00:42:50.917      82 |       .pause(50)
00:42:50.917      83 |       .waitForElementNotVisible(searchWideSelector) 
00:42:50.917      84 |       // Release all keys.
00:42:50.917      85 |       .keys(browser.Keys.NULL);
00:42:50.917     –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:42:50.917 
00:42:51.081 
00:42:51.081   FAILED: 1 assertions failed and  3 passed (5.959s)

00:43:03.680   ✖ NightwatchAssertError
00:43:03.680    Testing if element <button.sticky-header-toggle> is visible in 5000ms - expected "is visible" but got: "not visible" (5191ms)
00:43:03.680 
00:43:03.680     Error location:
00:43:03.680     /var/www/html/core/tests/Drupal/Nightwatch/Tests/Olivero/oliveroStickyHeaderToggleTest.js:
00:43:03.680     –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:43:03.680      21 |       .assert.attributeEquals(buttonSelector, 'aria-checked', 'false')
00:43:03.680      22 |       .getLocationInView('footer.site-footer', () => {
00:43:03.680      23 |         browser.assert.visible(buttonSelector); 
00:43:03.680      24 |         browser.assert.not.visible('#site-header__inner');
00:43:03.680      25 |       })
00:43:03.680     –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:43:03.680 
00:43:03.833 
00:43:03.833   FAILED: 1 assertions failed and  2 passed (5.753s)

Also, we can now drop our workaround with the "resolutions"-section in core/package.json, since nightwatch 3.1.0 has a "safe" version of the semver-package.

I don't know if there are any differences between Chrome 106 and Chrome 115 that are relevant to this issue, but FYI that I just now opened #3377509: Update Chrome container to use newer version (115 or higher).

oliveroStickyHeaderToggleTest fails due to getLocationInView() not working now we are in w3c mode: https://github.com/nightwatchjs/nightwatch/issues/3091

We will need to find a workaround for that.

The other issues are due to sending special keys such as Tab or arrow-down - this also no longer seems to work, the driver responds with

{
     value: {
       error: 'unknown command',
       message: 'unknown command: Cannot call non W3C standard command while in W3C mode',
       stacktrace: ''
     }
  }

Whoops, got to a similar solution as @longwave, but was trying to post it in my [ignore]-patch issue.

EDIT: Well, the testrun over there was promising, let's put it into the MR.
Ideally we will be only left with the getLocationInView-replacement.

I think we have another candidate follow-up issue:

core/tests/Drupal/Nightwatch/Tests/claroAutocompleteTest.js passes, but the autocomplete JavaScript used in the test seems to be 404?
artifact

So, per usual, @longwave found the correct way.

Also, per usual, I stumble over the weirdest stuff:

Commit 8ebeb952 has only one error showing in the full CI log, by now we know not to trust the green label).
Yet looking at the result artifacts created by nightwatch, we have 5 failures.

I strongly suspect any Nightwatch-test ending with a

(result) => {
 browser.assert.[something](result-from-a-function)
}

in an execute() will not show up in the log when it fails.

It's seems we're _really_ want to make it hard to see a nightwatch failure :/
Yet Another Follow-Up, me thinks.

Not sure what's happening with the Nightwatch output, can't reproduce this locally - yarn exits with a non-zero return code which should indicate failure, but on DrupalCI it just seems to abort and truncate the log.

So if we dig really deep (https://dispatcher.drupalci.org/job/drupal_patches/195437/artifact/jenki...), we see that there are 2 remaining failing tests.

The fact that we have to dig _really_ deep, since:

1) TestBot is green anyway
2) You have to look at the full log to see any reported fail in the log
3) Which then turns out to not even show all the fails

leaves me wondering if there's any use of a test subsystem that is giving a false sense of security and most probably have been doing so since it introduction without being noticed by anybody since it introduction in 2018.

Add to that the fact we didn't seem to mind too much, when it couldn't be updated when 9.5/10.0 came out, nor when 10.1 was released, is making me feel Nightwatch-testing isn't getting same amount of attention/love the rest of core is getting.

For me, it's a reason to spend my time elsewhere in the queue.
I think it's not worth putting time in upgrading a test system that's "green" anyway, unless you really, really spend time to find a failure that is big enough to show a glimpse of the fail showing up in the top reporting layer.

it needs CR to point changes in tests

Opened #3387927: Failing Nightwatch jobs result in a passed pipeline job but agreed with #23 it's not clear how useful the nightwatch suite is if it's been giving us false negatives for years.

This is also blocking further progress on #3386474: [omnibus] Speed up gitlab ci runs now since after the changes on that issue, nightwatch becomes the joint-slowest test group.

Can we skip the failing tests and open a follow up issue for each (or all) of them?

https://nightwatchjs.org/guide/running-tests/skipping-disabling-tests.html

@mstrelan yeah I think that's reasonable, otherwise any new nightwatch tests we add could also end up blocking this issue too.

It took me a very, very long time to get Nightwatch tests to pass on GitLab CI for a contrib module I maintain.

Turns out the root cause is that the d.o GitLab template uses the Nightwatch test runner in a very good way but sadly broken way: #3389763: Impossible to run only Nightwatch tests in a given directory (f.e. for contrib modules).

leaves me wondering if there's any use of a test subsystem that is giving a false sense of security and most probably have been doing so since it introduction without being noticed by anybody since it introduction in 2018.

I've noticed this too in the past few years, occasionally, specifically while working on Nightwatch tests. I think the very low number of Nightwatch tests in Drupal core is an important factor in how this has slipped through the cracks. Another important factor seems to be the less-than-awesome mapping of DrupalCI "number of tests" (and failures) to actual reports. (Meaning: it's AFAICT not actually limited to Nightwatch tests.)

Hopefully the transition to GitLab CI will make things more precise, standardized and reliable!

I had a go at rebasing this. There were conflicts in the following files and I may have not done it correctly. Nightwatch tests do not complete. They hang at installProfileTest.js. See https://git.drupalcode.org/project/drupal/-/jobs/770136

• core/tests/Drupal/Nightwatch/Tests/Olivero/oliveroSearchFormTest.js
• core/tests/Drupal/Nightwatch/Tests/Olivero/oliveroMobileMenuTest.js
• core/modules/ckeditor5/tests/src/Nightwatch/Tests/ckEditor5CodeSyntaxTest.js
• core/modules/ckeditor5/tests/src/Nightwatch/Tests/ckEditor5EditorHeightTest.js

#3421202: Enable W3C-compliant webdriver testing landed which gives us a newer Selenium/Chrome to talk to and which may solve the issues we were seeing here.

Locally I bumped to Nightwatch 3.7.0 which works OK with the new Selenium setup and latest Chrome, but tests appear to crash and then the Selenium server no longer responds:

$ yarn test:nightwatch tests/Drupal/Nightwatch/Tests/loginTest.js

[Tests/Login Test] Test Suite
──────────────────────────────────────────────────────────
ℹ Connected to selenium on port 4444 (505ms).
  Using: chrome-headless-shell (126.0.6478.114) on LINUX.

  ℹ Loaded url http://ddev-drupal8-web in 446ms

  Running Test login:
───────────────────────────────────────────────────────────────────────────────────────────────────
  ℹ Loaded url http://ddev-drupal8-web/user/reset/1/1721811628/FNZSmygKgxaar4E9AmDAlZ75Q61bU4ew0iwBaov3XUA/login
 in 440ms
  ℹ Loaded url http://ddev-drupal8-web/admin/people/roles/add in 133ms
  ✔ Expected element <.user-role-form .machine-name-value> to be visible in 2000ms (36ms)
  ✔ Testing if element <[data-drupal-messages]> contains text 'Role xt4aw4gms9 has been added.' (24ms)
  ℹ Loaded url http://ddev-drupal8-web/admin/people/permissions in 160ms
  ✔ Element <table.permissions> was visible after 34 milliseconds.
  ✔ Testing if element <[data-drupal-messages]> contains text 'The changes have been saved.' (22ms)
  ℹ Loaded url http://ddev-drupal8-web/user/logout/confirm in 69ms
  ℹ Loaded url http://ddev-drupal8-web/user/reset/1/1721811630/RH7RHeQ6JGfOHJ7RGMIXnuu3wYDMoU3KEqDnBTZrHfI/login
 in 83ms
  ℹ Loaded url http://ddev-drupal8-web/admin/people/create in 135ms
  ✔ User "user" was created successfully (34ms)
  ℹ Loaded url http://ddev-drupal8-web/user/logout/confirm in 47ms
  ℹ Loaded url http://ddev-drupal8-web/user/login in 54ms
  ✔ Passed [equal]: The user "user" was logged in.
  ℹ Loaded url http://ddev-drupal8-web/admin/reports in 76ms
  ✔ Element <body> was visible after 30 milliseconds.
  ✔ Testing if element <h1> contains text 'Reports' (32ms)
$

I was expecting further output here, if I run this on the older version of Nightwatch it ends as follows:

  ✔ Testing if element <h1> contains text 'Reports' (29ms)
  ✔ Ensuring no deprecation errors have been triggered (10ms)

  ✨ PASSED. 9 assertions. (3.879s)
 Wrote HTML report file to: /var/www/html/drupal/core/reports/nightwatch/nightwatch-html-report/index.html

$

If I then run the test again it hangs:

$ yarn test:nightwatch tests/Drupal/Nightwatch/Tests/loginTest.js

[Tests/Login Test] Test Suite
──────────────────────────────────────────────────────────
⠴ Connecting to selenium on port 4444...

Restarting the Selenium container fixes it.

Will push this branch up anyway but not sure what's happened here at all.

api.execute() takes the callback as the third argument instead of the second now, it seems.

Last run was looking promising until it appeared to hang after installProfileTest: https://git.drupalcode.org/project/drupal/-/jobs/2214554

I'm slightly amazed that it was only another call to execute() that needed fixing, but the run is green.

I think we can now also drop the resloutions section from core/package.json.
These were only needed to keep the dependencies of the old version of nightwatch clashing with our other JS-dependencies.

Tests are still green.

For the record, just removed the resolutions-section and did a $ yarn install.

This should have been an 11.0.0 beta blocker, but got missed. We might make an exception to policy and commit this to 11.0.x despite that branch being almost-stable, because as @longwave points out being on a very old version of Nightwatch means being stuck with old Chrome in our tests.

If we do commit it, it should go in the release notes. The CR and release notes should also mention any gotchas that have surfaced from the updates (and linking out to Nightwatch's own release notes for more details also a good strategy).

Not best to review this only moving to NW for the open tags and mainly because MR needs a rebase.

Rebased.

Although there are still release notes and CR needed, all tests are green and some manual poking would be more than welcome IMHO.

Removed the "Needs follow-up"-tag since my woes in #13 seem to be resolved already.

I added a CR but it just points to Drupal's nightwatch docs and nightwatch's 3.x post, not sure what else there is to say tbh. https://www.drupal.org/node/3467273

The release note as it is looks OK except I updated the version from 2.0.0 to 3.0.0 which looked like a typo.

Bumping this to critical, every other core test run fails on nightwatch random test failures, and even if this doesn't help, we need to rule it out.

Leaning on the tests passing that upgrading doesn't break anything.

Not sure a good to see what contrib modules are using nightwatch tests but if being honest knowing how difficult it is to get nightwatch running locally I would bet not many would be using it. So probably safe.

Opened follow up #3467492: [policy, no patch] Replace Nightwatch with Playwright. Feel free to tell me it's a straight-up won't fix.

Let's get this into 11.x so if it somehow makes stability issues worse (doesn't seem possible, but who knows) we'll be able to see them with plenty of notice before the 11.1.0 release.

Amending attribution.