Enable W3C-compliant webdriver testing

Can you rerun the javascript tests? Want to make sure they aren't random but even I request access I can't rerun.

Glad I asked!

I've started using/recommending this locally too fwiw - a lot of people (more than 2-3 I could name) have trouble with the drupalci one which they don't run into with the selenium one.

I've split the part about upgrading the tests to https://www.drupal.org/project/drupal/issues/3432099 as we can test this is working using geckodriver

After #3240792: Use lullabot/mink-selenium2-driver and lullabot/php-webdriver for functional browser testing I wasn't able run test locally with selenium/standalone-chrome, which I'd otherwise been using successfully for months, gave up trying to debug it and went back to the drupalci image (which has been working OK). I haven't seen anyone else run into issues since that commit, but presumably people will start to switch over if we switch gitlab, so just making a note here in case other people do indeed run into issues down the line.

Just to chime in re #12 I was seeing exactly the same thing (see https://github.com/ddev/ddev-selenium-standalone-chrome/issues/41). It appears that \WebDriver\WebDriver::session() is checking for the w3c flag one level higher up than where it's being set. I'm not sure if the issue should be fixed in lullabot/php-webdriver or by changing the capabilities we're passing. Locally I've just changed the capabilities, but I've seen plenty of examples on Stack Exchange etc. of the w3c flag being within goog:chromeOptions and I can't find a canonical reference on the matter.

I was seeing the same problem as #12, running test on my local machine with Drupal 11.x-dev.

I fixed it by implementing the workaround posted in #13 - I simply replaced line 15 in config.selenium-standalone-chrome.yaml to this:

- MINK_DRIVER_ARGS_WEBDRIVER=[\"chrome\", {\"browserName\":\"chrome\",\"w3c\":false,\"goog:chromeOptions\":{\"args\":[\"--disable-gpu\",\"--headless\", \"--no-sandbox\", \"--disable-dev-shm-usage\"]}}, \"http://selenium-chrome:4444/wd/hub\"]

Thank you @AndyF!

The change record needs to be written but otherwise the tests are green so marking NR, hopefully someone else familiar with Chrome testing can take a look.

We still need to decide how to deprecate non-W3C mode and what to do about tests in that time, do we run them in both modes or for core do we just switch to W3C mode only?

I think we should also add a manual job to run tests using Firefox.

nice idea, there's selenium/standalone-firefox image

Locally in docker I got few tests running fine using Firefox image with following arguments
MINK_DRIVER_ARGS_WEBDRIVER: '["firefox", {"browserName":"fixrefox","moz:firefoxOptions":{"args":["--headless"]}}, "http://browser:4444"]'

Somehow FIrefox tests are x2 faster then selenium jobs

Thanks! it explains difference (all skipped for ff)

btw performance metrics also could be collected from FF, at least it has tracing out of box and allows to enable dev-tools (even via remote somehow) and get better details... for follow-up

Moreover it makes sense to explore optimizations related to --headless mode in browsers

An interesting thing with the four vs. two javascript runners is that overall time still seems to be constrained by the slowest individual tests (either Drupal\Tests\ckeditor5\FunctionalJavascript\MediaTest or Drupal\Tests\ckeditor5\FunctionalJavascript\SourceEditingTest seem to be the longest running) and overall run duration is also similar to what it is now. I would kind of have expected the bottleneck to shift to the number of http requests away from installing Drupal, although maybe those tests also do loads of http requests too.

I wonder if we're hitting concurrency/cpu limits. However anything like that we can explore in a follow-up - four runners seems fine and there's no overall regression in the pipeline feedback.

If we're just moving Nightwatch entirely to w3c mode then we can drop the separate selenium environment and just turn on the w3c switch in the default environment.

Added some nits/comments to the MR.

Let's update the CR for Nightwatch and also add a followup to investigate Selenium switches, otherwise this is ready to go.

Alright, let's ship this - tests are green, and the test and configuration changes are the minimum required to enable W3C mode. This then unlocks some future improvements including hopefully being able to upgrade Nightwatch to a more recent version.

Committed/pushed to 11.x, thanks!

If this does indeed make functional js tests a lot slower, it may break the performance tests dashboard. I've opened #3463351: Consolidate Umami performance tests which should cover that.

Alongside that, I also opened #3463456: Consolidate ckeditor5's FunctionalJavascript tests for core's other longest running functional js tests.

I think we should open a follow-up to revisit concurrency once those two issues are in, since they may be enough to lower it again, I'll open it postponed now.

I am assuming that we don't want to try to backport this to 10.4.x, but if we do, please re-open with a backport MR.

edit: #3463286: Fix selenium performance/stampede issues in gitlab config and BrowserTestBase covers the concurrency follow-up.

Performance test job is broken since this went in #3463534: Performance test gitlab job is broken - revert to use chromedriver.

The only question is deprecation message, let's update it to "deprecated in 10.0.4 and removed in 12.0.0"

Added some comments about being slightly more conservative in the 11.0/10.4 backports. Should this also go back to 10.3 to keep in sync?

Various tests are broken.

#3463286: Fix selenium performance/stampede issues in gitlab config and BrowserTestBase landed so that change needs to be incorporated here now.

Unit tests also failed on 10.4.x in WebAssertTest.

What's the status of this? #47 landed, but then this issue was reopened, without a revert. The two change records are still draft.

EDIT: ah I guess #3463534: Performance test gitlab job is broken - revert to use chromedriver is the partial revert?

Can contrib follow core's example/lead? Should it?

@Wim This was re-opened for backport, not due to regressions.

However #3463286: Fix selenium performance/stampede issues in gitlab config and BrowserTestBase landed in 11.x, and needs to be incorporated into the backport here to avoid serious performance/instability issues on pipelines. The gitlab YAML files have diverged quite a lot between branches so it is a real backport not just a cherry-pick unfortunately.

Contrib w3c testing is happening in #3463894: Update templates so 11.0 is the default/current branch.

It looks like a blocker for deprecation #3462682: Deprecate non-W3C compliant testing

PS: not clear state of w3c testing in contrib as it reported blocked too #3462681: Add W3C compliant JS testing

We never finished the backport, I think it's too late to bother with that now. Let's just keep both sets of tests in 11 and then remove non-w3c testing in 12.

This was released with 11.1