Problem/Motivation

Nightwatch tests have proven to be too shaky for my liking and may not be a good long-term choice.

Details:

Although Drupal core uses some Nightwatch tests, we don't need to restrict ourselves to the testing tools shipped with core.

Also, Drupal core might adopt Playwright.

Proposed resolution

Replace Nightwatch with Playwright.

Remaining tasks

  • Set up Playwright for testing
  • Port existing tests to Playwright

User interface changes

n/a

API changes

n/a

Data model changes

n/a

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

star-szr created an issue. See original summary.

star-szr’s picture

Issue summary: View changes
star-szr’s picture

I did a quick search today and found that core is looking to go in the direction of Playwright, which based on my recent evaluations for a client project would be very high on my list as well.

Tools to check out:

https://github.com/Lullabot/ddev-playwright
https://github.com/julienloizelet/ddev-playwright

star-szr’s picture

Title: Replace Nightwatch testing with another solution » Replace Nightwatch testing with Playwright
Category: Plan » Task
Issue summary: View changes

At this point the path seems clear to use Playwright, of course things can change but converting to a task for now.

I am hoping to have some time available in the next week or two to continue this work.

star-szr’s picture

Version: 1.0.x-dev » 1.1.x-dev

Taking up work on this again.

star-szr’s picture

Assigned: Unassigned » star-szr

star-szr’s picture

Some notes to share, in hopes that this can be helpful to others setting up/converting Playwright in their projects, and especially those relatively or entirely new to Playwright:

  • I used DDEV and https://github.com/Lullabot/ddev-playwright with kasmvnc to make use of the actual Playwright UIs.
  • From what I can tell, the Lullabot Playwright DDEV add-on assumes a path of test/playwright, but I just followed their setup instructions and didn't make use of those files at any point. Arguably this should be configurable/overrideable, and I might create an issue/MR in the add-on to address this in future.
  • The Playwright docs are great and I found it easy to look up the various types of assertions and such.
  • In particular reading through the first few parts of https://playwright.dev/docs/test-fixtures was very helpful to better understand things.
  • Code generation can be handy to get a starting point, but be prepared to refine the code that comes out of it. Also, even with VS Code, you can't really "append" to an existing test case like I was hoping. I wasn't really able to find anything satisfying here, especially since I don't normally use VS Code, but I may need to experiment with the approach described in https://github.com/microsoft/playwright/issues/28571 to see if this would be viable without VS Code. See also https://github.com/microsoft/playwright/issues/23625.
  • Related to code generation, you may find it useful to run yarn playwright codegen $(drush uli --uri=http://web) (this is assuming DDEV, adjust as needed) to generate initial test code in an environment similar to the test environment. If you use test modules, add to settings.php/settings.local.php the line $settings["extension_discovery_scan_tests"] = TRUE;.
  • I found it to be important that drush is available and installed in the correct location, otherwise you may get errors that are very hard to diagnose. At one point I unintentionally installed drush inside the module rather than in the site's bin, and my test module was failing to install in Drupal 11 but all I could see was that the site hadn't installed.
  • I found await page.waitForTimeout(60_000); useful in conjunction with kasmvnc, at least before I learned about the built-in debugging tools available.
  • Depending on what you are testing, be aware that this setup is not fully isolated, since all the workers use the same site URL. So in my experience config changes can easily bleed in between workers and cause problems. This led me to be add steps to reset config and in general trying to isolate individual test cases. In the end, I'm able to run the tests fully parallel without issue, at least with a single browser.
  • In CI, sometimes the database will fail to connect on site install. This is very similar to the main issue I had with Nightwatch, but Playwright (at least as configured here) will just retry it so it's much less of an issue. I spent a short amount of time seeing if this could be fixed but didn't get too far.

  • star-szr committed 522135b2 on 1.1.x
    [#3509004] test: add DRUPALORG_CI_SERVER_URL
    

  • star-szr committed 5d098393 on 1.1.x
    [#3509004] test: show environment variables in log
    

  • star-szr committed 8c2fc4c5 on 1.1.x
    revert: [#3509004] test: update to latest playwright
    
    This reverts...

  • star-szr committed af13184a on 1.1.x
    [#3509004] test: only test chrome for now
    
    We will add back other...

  • star-szr committed 770d9cf8 on 1.1.x
    [#3509004] test: initial gitlab ci setup
    
    Again, almost fully copying...

  • star-szr committed 18103672 on 1.1.x
    [#3509004] test: port config tests to playwright
    

  • star-szr committed ea208034 on 1.1.x
    [#3509004] test: update to latest playwright
    

  • star-szr committed 7095ef88 on 1.1.x
    [#3509004] test: initial playwright setup
    
    Many thanks to...
star-szr’s picture

Assigned: star-szr » Unassigned
Status: Active » Fixed

Going to call this done.

Now that this issue is closed, please review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, please credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.