Impossible to run only Nightwatch tests in a given directory (f.e. for contrib modules)

Problem/Motivation

Or more generically: Drupal's nightwatch.conf.js does not allow specifying a subdirectory to run tests from. It only supports two things:

• all tests in the entire Drupal site
• all tests in the entire Drupal site + tests one level higher

IOW: it is impossible to specify a directory inside the Drupal site whose tests to run. And this happens to be the strategy that the d.o GitLab CI template uses to run Nightwatch tests for contrib modules:

    - yarn --cwd $_WEB_ROOT/core test:nightwatch  $CI_PROJECT_DIR/$_WEB_ROOT/modules/custom/$CI_PROJECT_NAME/tests

Quoting core/.env.example:

# The path that Nightwatch searches for assumes the same directory structure as
# when you download Drupal core. If you have Drupal installed into a docroot
# folder, you can use the following folder structure to add integration tests
# for your project, outside of tests specifically for custom modules/themes/profiles.
#
# .
# ├── docroot
# │   ├── core
# ├── tests
# │   ├── Nightwatch
# │   │   ├── Tests
# │   │   │   ├── myTest.js
#
# and then set DRUPAL_NIGHTWATCH_SEARCH_DIRECTORY=../
#
#DRUPAL_NIGHTWATCH_SEARCH_DIRECTORY=

— committed in #2869825: Leverage JS for JS testing (using nightwatch) and never updated since.

Steps to reproduce

On GitLab CI, if you are using the d.o GitLab template:

$ sudo BABEL_DISABLE_CACHE=1 DRUPAL_NIGHTWATCH_SEARCH_DIRECTORY=modules/custom -u www-data yarn --cwd ./core test:nightwatch
yarn run v1.22.19
$ cross-env BABEL_ENV=development node -r dotenv-safe/config -r @babel/register ./node_modules/.bin/nightwatch --config ./tests/Drupal/Nightwatch/nightwatch.conf.js
Browserslist: caniuse-lite is outdated. Please run:
  npx update-browserslist-db@latest
  Why you should do it regularly: https://github.com/browserslist/update-db#readme
   Error: Cannot read test source location: /builds/project/acquia_migrate/docroot/core/modules/custom../CUSTOM_MODULE_NAME/tests/src/Nightwatch/Tests

The only work-around that is possible:

--tag=something

(Typically "something" would be the name of a specific module.)

Which requires your Nightwatch tests to contain:

…
module.exports = {
  '@tags': ['something'],
…

Or on your local machine, run all Nightwatch tests of all installed contrib modules:

cd /path/to/drupal/root/that/contains/index.php
DRUPAL_NIGHTWATCH_SEARCH_DIRECTORY=modules/contrib yarn --cwd ./core test:nightwatch

Proposed resolution

Tweak the logic in nightwatch.conf.js to accept not only supersets (go up the directory tree) to find Nightwatch tests, but also subsets (go inside a directory tree, such as the contrib modules directory, the custom modules directory, or one specific module directory).

Hot fix on macOS:

sed -i '' 's/const key = \`\.\.\/.*/const moduleDirectory = m[1];/' core/tests/Drupal/Nightwatch/nightwatch.conf.js
sed -i '' 's/\${searchDirectory}\${key}/\.\.\/\${searchDirectory}\/\${moduleDirectory}/' core/tests/Drupal/Nightwatch/nightwatch.conf.js

Hotfix on Linux (and GitLab CI — which does not seem to like backticks):

sed -i 's/const key = \`\.\.\//const key = \`/' $CI_PROJECT_DIR/$_WEB_ROOT/core/tests/Drupal/Nightwatch/nightwatch.conf.js
sed -i 's/\${searchDirectory}\${key}/\.\.\/\${searchDirectory}\/\${key}/' $CI_PROJECT_DIR/$_WEB_ROOT/core/tests/Drupal/Nightwatch/nightwatch.conf.js

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet