Migrate from Nightwatch to Playwright (and phpunit Axe) tests

Trial run, the login test passes inside ddev:

$ yarn test:playwright

Running 1 test using 1 worker

  ✓  1 tests/Drupal/Playwright/Tests/loginTest.spec.js:11:3 › Login › Test login @core (6.8s)

  1 passed (7.7s)

To open last HTML report run:

  yarn playwright show-report tests/Drupal/Playwright/reports

This MR was heavily assisted by Claude Code.

Amazing!

$ yarn run test:playwright
Running 1 test using 1 worker
  ✓  1 tests/Drupal/Playwright/Tests/loginTest.spec.js:11:3 › Login › Test login @core (8.1s)
  1 passed (9.7s)

All non-a11y tests converted, 73 passes locally. The test suite takes just under 14 minutes on my machine, hopefully we can parallelise it in some way.

CI is timing out on every test.

Why don't use ES Modules for new test scripts?

According to npm-esm-vs-cjs data, over 10% of NPM packages already adopted ES Module only package in Dec 2026.
Furthermore, no supporting ES Modules packages decrease year by year and it's pace are increased.

I think adopting ES Modules for new test scripts is better than CommonJS.

@tom konda Good idea, thanks - converted in a recent commit.

@longwave
Good works, thanks.

Thinking about this again: while we agreed to swap Nightwatch for Playwright, there don't seem to be many people interested in Playwright tests. If we can use AI for this, should we just use AI to convert the Nightwatch tests to PHPUnit FunctionalJavascript tests instead? Would that make them easier to maintain?

That was my original proposal in the policy issue. The people who write NW/PW tests don't seem to be the same people who are maintaining them.

Tested the Playwright migration, and it works pretty well, thank you!

To speed up, I think we can start with small steps, the first one is just to provide the Playwright's drupal-fixtures.mjs and the working pipeline, having this in Core - users will be able to start writing their own Playwright tests already, instead of Nightwatch.

For me, everything in the MR looks great, so seems we have to resolve only the "CI is timing out on every test" issue, right?

And in addition to introducing core/tests/Drupal/Playwright/drupal-fixtures.mjs - would be great to provide a contrib helper module that provides these functions as a dev dependency.

This will allow switching from Nightwatch to Playwright tests for contrib module developers without waiting for the Drupal 11.3 end-of-life.

Because with the current approach, if users still want to keep test coverage for Drupal below 11.4, they should support both - Nightwatch for Drupal <11.4 and Playwright for Drupal 11.4+.

I think a good candidate for this can be https://www.drupal.org/project/playwright - just put there the drupal-fixtures.mjs and other fixtures, and developers will be able to install the drupal/playwright module as dev dependency to start using it even for old Drupal versions. What do you think about this?

CI is timing out on every test.

Found the source of this issue! The root issue is that Drupal is intentionally installed into the subdirectory http://localhost/subdirectory to check that everything works with subdir installation mode.

Here is a pipeline that passes at least three tests: https://git.drupalcode.org/issue/drupal-3553673/-/jobs/8940595

...
  67 did not run
  3 passed (2.3m)

And to make Playwright navigate by relative URLs, keeping the subdirectory, we should do two things:

1. Create the browser context with / symbol at the end of the directory.
I tested this via a quick fix in the drupal-fixtures.mjs:

-  const context = await browser.newContext({ baseURL: baseUrl });
+  const context = await browser.newContext({ baseURL: `${baseUrl}/` });

2. Use page.goto() with relative urls everywhere:

-  await page.goto('/user/login');
+  await page.goto('./user/login');

Also, using no leading slash like this await page.goto('user/login') should work too.

Maybe we can invent a way to simplify this somehow, to not ask to add a dot on every page.goto() call, but for now - have no idea how to do this.

But we should somehow resolve the ./ URLs issue to make it impossible to make mistakes by passing non-relative URLs, because this will produce tricky errors like we have now - the tests work well on the localhost, but fails on the pipeline, because the pipeline runs the tests with the subdirectory, but localhost - not.

Any ideas on how to resolve these two issues?

Also, found one more repo that tries to provide Playwright fixtures for Drupal: https://gitlab.com/mog33/playwright-for-drupal-tests - but there the described issue is not resolved too, it just tests the root path without the subdirectory. And the same issue is with www.drupal.org/project/playwright

Also, would be great to create a separate container image with pre-installed Playwright and browsers, to prevent re-installing a bunch of packages and downloading browsers on every run.

Downloading browsers can be resolved by caching, but installing system packages can be cached only as a separate container image, so I'd prefer to put the pre-downloaded browsers into that image too.

The image can be done over the current Drupal image: registry.gitlab.com/drupal-infrastructure/drupalci/drupalci-environments/php-8.1-apache:production

Here is a list of packages that should be added for Playwright to work well:

The following NEW packages will be installed:
  at-spi2-common fonts-freefont-ttf fonts-ipafont-gothic fonts-liberation
  fonts-noto-color-emoji fonts-tlwg-loma-otf fonts-unifont fonts-wqy-zenhei
  libasound2-data libasound2t64 libatk-bridge2.0-0t64 libatk1.0-0t64
  libatspi2.0-0t64 libavahi-client3 libavahi-common-data libavahi-common3
  libcairo2 libcups2t64 libdatrie1 libdbus-1-3 libdrm-amdgpu1 libdrm-common
  libdrm-intel1 libdrm2 libelf1t64 libfontenc1 libfribidi0 libgbm1 libgl1
  libgl1-mesa-dri libglib2.0-0t64 libglvnd0 libglx-mesa0 libglx0
  libgraphite2-3 libharfbuzz0b libice6 libllvm20 libnspr4 libnss3
  libpango-1.0-0 libpciaccess0 libpixman-1-0 libsensors-config libsensors5
  libsm6 libthai-data libthai0 libunwind8 libvulkan1 libx11-6 libx11-data
  libx11-xcb1 libxau6 libxaw7 libxcb-dri3-0 libxcb-glx0 libxcb-present0
  libxcb-randr0 libxcb-render0 libxcb-shm0 libxcb-sync1 libxcb-xfixes0 libxcb1
  libxcomposite1 libxdamage1 libxdmcp6 libxext6 libxfixes3 libxfont2 libxi6
  libxkbcommon0 libxkbfile1 libxmu6 libxmuu1 libxpm4 libxrandr2 libxrender1
  libxshmfence1 libxt6t64 libxxf86vm1 mesa-libgallium x11-common x11-xkb-utils
  xauth xfonts-cyrillic xfonts-encodings xfonts-scalable xfonts-utils xkb-data
  xserver-common xvfb
0 upgraded, 92 newly installed, 0 to remove and 12 not upgraded.
Need to get 84.2 MB of archives.
After this operation, 317 MB of additional disk space will be used.

And here are the browsers that downloads on every CI run:

Downloading Chrome Headless Shell 145.0.7632.6 (playwright chromium-headless-shell v1208) from https://cdn.playwright.dev/builds/cft/145.0.7632.6/linux64/chrome-headless-shell-linux64.zip (110.9 MiB)
Chrome Headless Shell 145.0.7632.6 (playwright chromium-headless-shell v1208) downloaded to /root/.cache/ms-playwright/chromium_headless_shell-1208
Downloading FFmpeg (playwright ffmpeg v1011) from https://cdn.playwright.dev/dbazure/download/playwright/builds/ffmpeg/1011/ffmpeg-linux.zip (2.3 MiB)

But this can be done later as a separate task.

And to make Playwright navigate by relative URLs, keeping the subdirectory, we should do two things

Filed an issue on the Playwright to simplify: this https://github.com/microsoft/playwright/issues/39720 - upvote pls ;)

There is still the question in #13 of whether we want to do this at all, because there don't seem to be many people interested in maintaining these tests.

There is still the question in #13 of whether we want to do this at all, because there don't seem to be many people interested in maintaining these tests.

Many contrib projects already use Playwright tests, you can see the usage here: https://search.tresbien.tech/search?q=%40playwright%2Ftest&num=50&ctx=0

At least, the Canvas project has used Playwright for a long time, instead of Nighwatch.

But all of them invent their own kludges to make it work with Drupal, instead of consolidating and reusing the same approach for all projects. So, I believe that Drupal Core should provide this "the best approach" out of the box, and modules will use it in their pipelines.

About converting all the Core tests from Nightwatch to Playwright, we can actually keep it as is with Nightwatch, just start to write new tests in Playwright.

The first and crucial step is to provide Drupal fixtures for Playwright, and a couple of test examples to use as a base for new tests in Core and contrib.

About converting all the Core tests from Nightwatch to Playwright, we can actually keep it as is with Nightwatch, just start to write new tests in Playwright.

The whole point of this is to remove nightwatch, because the tests we currently have are incredibly flakey and no one wants to maintain them.

The whole point of this is to remove nightwatch, because the tests we currently have are incredibly flakey and no one wants to maintain them.

Then, the easiest solution is just to remove Nightwatch tests, and live calmly without any functional tests? 😂
Wait, this is not a good solution, right? 😉

So, we have two options:
1. Keep the Nightwatch tests as is and live in the flaky world.
2. Migrate all tests to Playwright, with the hope that there will be more Playwright fans than Nightwatch fans, who will want to maintain them.

Let's choose the second option, and convert the current Nightwatch tests to Playwright, and make it work well, as the largest part of the job was already done by @longwave.

I updated all the tests to use relative paths in the page.goto(), that should fix the pipeline. Until https://github.com/microsoft/playwright/issues/39720 is implemented, let's live with this approach then.

And, 🥁 - it works! The Playwright pipeline job is green with all tests passed well:
73 passed (15.0m)

Please review the changes and the approach, and then we can clean up all debug code, and prepare the final version to merge.

As a maintainer of one of the contrib projects currently using Playwright (because yes, Nightwatch is flaky as heck, and my use case is not covered by FunctionalJavascript tests, I tried):

But all of them invent their own kludges to make it work with Drupal, instead of consolidating and reusing the same approach for all projects. So, I believe that Drupal Core should provide this "the best approach" out of the box, and modules will use it in their pipelines.

At this time I can't offer to help develop or maintain Playwright tests in core, but would certainly make use of a consolidated approach if/when it gets into core. Edit: For the record, my approach was mostly a copy+paste of the implementation in the Canvas project, but I'm sure it has drifted since then.

I will say, because of the ability to record tests in Playwright, there seems to be potential for folks from a wider set of skills to at least help in putting tests together. This is speculative, those folks may never materialize, but worth recognizing I think.

This looks great! Can we modify the runner to run them 100 times to see if they are more stable than nightwatch?

This looks great! Can we modify the runner to run them 100 times to see if they are more stable than nightwatch?

GreenPeace wouldn't like the carbon footprint of this 😁 but let's try: here is the pipeline https://git.drupalcode.org/issue/drupal-3553673/-/jobs/8961401

Also, would be great to create a separate container image with pre-installed Playwright and browsers, to prevent re-installing a bunch of packages and downloading browsers on every run.

Downloading browsers can be resolved by caching, but installing system packages can be cached only as a separate container image, so I'd prefer to put the pre-downloaded browsers into that image too.

The image can be done over the current Drupal image: registry.gitlab.com/drupal-infrastructure/drupalci/drupalci-environments/php-8.1-apache:production

Seems this works! I published an image, based on the registry.gitlab.com/drupal-infrastructure/drupalci/drupalci-environments/php-8.3-apache:production here: https://gitlab.com/murz-drupal/drupal-playwright/container_registry

And it works!!! Tested on my own project, here is the green pipeline: https://git.drupalcode.org/project/logger/-/jobs/8961204

To test this image, set the path to the image like this in the Playwright job:

.playwright-base:
  image: registry.gitlab.com/murz-drupal/drupal-playwright:main

Re 28 yes, I am very aware of the energy usage, but it's a one time check to validate it's less flaky which will save countless runs over time.

This looks great! Can we modify the runner to run them 100 times to see if they are more stable than nightwatch?

100 times is too much, even 10 times does not fit into 30 min limit of the job. But, 8 times went well: https://git.drupalcode.org/issue/drupal-3553673/-/jobs/8975910

Since the original issue closed - https://github.com/microsoft/playwright/issues/39720
As an option for relative urls we may consider to use base.extend and add wrapper for goto in order to normalize url and use relative path
something like this:

page: async ({ page, baseURL }, use) => {
     const originalGoto = page.goto.bind(page);
     page.goto = (url, options) => originalGoto(normalizeGotoUrl(baseURL, url), options);
     await use(page);
 },

function normalizeGotoUrl(baseURL, url) {
  if (/^[a-z][a-z0-9+.-]*:/i.test(url)) return url;
  if (!baseURL) return url;
  try {
    const base = new URL(baseURL);
    if (base.pathname === '/' || base.pathname === '') return url;
    if (!base.pathname.endsWith('/')) base.pathname += '/';
    if (url.startsWith('/')) url = '.' + url;
    return new URL(url, base.href).href;
  } catch {
    return url;
  }
}

Let me know please if it makes sense to you

@kir lazur, yeah - thank you for sharing! Since https://github.com/microsoft/playwright/issues/39720 is closed, seems we have to invent something on our side.
Your approach with a proxy function looks good to me! Now, we have to decide where to put this function, to make it included by default in all tests.

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.