Automate Accessibility Checks for Core

Also worth looking at https://www.axe-core.org/ & https://github.com/pa11y

Would be excellent to have this as part of the patch review process. There was a TestSwarm project that was used early on on D8 using QuailJS [@1844046] but that project has died.

Some other links.
https://www.drupal.org/sandbox/kevee/2026475
https://dev.acquia.com/podcast/98-accessibility-testing-tools-and-quail-...

Marking this as part of #1299710: [meta] Automate the coding-standards part of patch review

For what it's worth, we've been evaluating Tenon.io at CU. Their API is really straightforward (Send url... get back JSON). It has been easy to integrate with both our UI for users to check content as well as with our CI, but we found that a number of the tests either weren't running or didn't catch an issue we thought would be easy to detect like https://www.colorado.edu/tenon-tests/content/114-select-element-less-5-o...

When we met with the vendor, they quickly acknowledged the issues and committed to having fixes pushed before the CSUN Assistive Technology Conference on 3/22.

At the enterprise level, a managed version of Tenon can be brought "in-house". At that point, you aren't limited by the number of api calls but only by the speed of running the tests on the backlog of requests.

@kreynen want to give a +1 on tenon.io

Karl's been a great advocate in the web accessibility community. He's actively investing in his platform.

Also: "We offer special discounts for open source projects, local governments, public school systems (primary/ secondary), community colleges, 501C(3) non-profits, web developer training schools/ bootcamps." - https://tenon.io/pricing.php

And they made sure there was a Drupal module built under the GPL - https://www.drupal.org/project/tenon_io

It's just not open source which is why I didn't list it here. Not as easy to integrate and propagate.

Not as easy to integrate and propagate.

But easier to maintain :)

Adding related links to refer to.

This is a good review from WordPress https://humanmade.com/2018/05/17/automated-accessibility-testing-during-...

Lots of stuff discussed in the #accessibility Slack channel today, with @modulist and myself. Here's a recap:

• The new Nightwatch tests open up more possibilities to integrate a11y checkers, because many of them have a JS API. Invoking them from Nightwatch may be a lot more practical than doing it from our phpunit base classes.
• W3C Accessibility Conformance Testing (ACT) formats may be useful one day.
  • It’s a standardized format for writing a11y test rules. My understanding is that Tenon, aXe, et al., would standardize on a common format to specify test coverage, and reporting test results.
  • This would allow tests to combine several tools, and avoid confusing overlaps.
  • Also may help auditors to track issue resolution...?
  • But it's currently bleeding edge (working draft)

As for the how/what/when to test, I think a couple of approaches together:

1. Pick a representative bunch of Drupal pages, load the page, run the general a11y check rule set. Test each page with all core themes.
   • Stark
   • Seven
   • Bartik (in particular, the node edit forms are used with admin theme or default theme)
   • Umami
   • Classy (base themes have a lot of potential to help/hinder accessibility)
2. Combined with more advanced functional tests for specific components, with simulated user journeys. For example:
   • Load the Views UI, run the a11y ruleset to check for problems on the page.
   • Then focus and activate a button.
   • Make some custom assertions about the component we're interested in. A dialog appears, has expected markup, focus moves to dialog. And finally run all the general ruleset AGAIN.
   • So what we're saying is that the DOM satisfies all the generic checks before-and-after user interaction. We're interested in maintaining the whole DOM in an accessible state throughout the user journey.
   • Some tests might not make sense to run on all themes. E.g. Bartik and Umami have their own responsive menu approaches.

Aside: a couple of years back, at a conference, I heard someone from a big a11y consultancy (not saying which....) describe a similar approach. It was to do with all these drive-by a11y lawsuits proliferating in the US. Turns out a lot if these came from a law office that was using the WAVE API to spider sites, then send them a lawsuit. So armed with the knowledge that WAVE reports were the risk for their clients, they made a custom assertion for Behat tests called "wave reports no errors" and did that at the start and end of every Behat test. This is about as black-hat as a11y gets (WAVE was the one generating the lawsuits, so that's the one we satisfy). But in a CI context it makes some sense I think.

An interesting point about putting generic a11y rulesets into CI, is how do we act on the results?

• A lot of the checks are advisory warnings, things to consider - but they may not be relevant.
• Also various false positives can happen.
• I don't think we should necessarily fail a patch that way.

Can you refine rulesets specifically for Drupal though?

• We could do that per test case?
• But heavy use of whitelists may be counter productive? Risk that we'd omit an important check.
• Maybe use a common set, and only omit a check in a specific scenario. It should require a comment to document WHY we were omitting a rule for a given test scenario?
• Could you start with the most essential a11y tests and expand them from there?

An interesting thing to do would be to set up a way to view the report for the current state of head, and maybe set some alert if we try to tag a new stable 8.x release candidate which has more warnings than the current supported release

• Like a quick-and-dirty regression smoke-test.
• But if Tenon/aXe add more coverage, the number would go up naturally.
• Maybe CI reports when a patch leads to more warnings than current HEAD, and adds a tag to the d.o issue?

You’d have to re-set your test every time Tenon and aXe do an update, like when 2.1 guidelines were added.

• Maybe our own ruleset whitelist would help there...
• So we don't include the "label in name" checks in CI until after we have carried out our initial audit?

@modulist deserves an issue credit for contributions in the Slack chat.

• A lot of the checks are advisory warnings, things to consider - but they may not be relevant.
• Also various false positives can happen.
• I don't think we should necessarily fail a patch that way

100% agree

Since I mentioned that we were evaluating Tenon.io in #4, I'll just add a bit about what we found in that evaluation. We decided not to license Tenon since it had too many false negatives (meaning it didn't catch obvious accessibility issues). We left our examples of things we thought should fail at https://www.colorado.edu/tenon-tests/. Some of these are more nuanced and required an understanding of what the test was actually testing, but several of the tests listed in UI just didn't fail like https://www.colorado.edu/tenon-tests/content/114-select-element-less-5-o.... That test should be really straightforward, but it would always appear to have passed.

Giving users a false sense of being WCAG compliant with tests that pass incorrectly is just as much an issue for us as tests that fail incorrectly.

We decided that the way UNL had implemented the aXe tests along with other metrics in https://github.com/UNLSiteMaster/site_master was a better fit for our needs. That project runs fewer tests with more confidence in each test and is only testing things that are likely to be something the user can fix. Instead of looking at a number of tests services like Tenon or Site Improve and going with the service that had the most tests, we are treating a combined score of failed accessibility tests, spelling errors and broken links as dead canaries that indicate a site is being poorly maintained and needs a manual review.

Tenon may still be a good fit for scanning contrib, but it wasn't a good fit for our needs.

I would also say that false positives don't worry me, but they add a potentially massive review burden and/or issue friction.

In an ideal world, I'd like some way of tracking the false positives and advisory warnings. So we could say "Manually reviewed by andrewmacpherson on 2018-10-15, works-as-designed". Preferably somewhere that non-a11y contributors could check, say before RTBC, or before committing a patch. Maybe a code comment in the relevant Drupal test file about a false positive we're expecting?

Would be good to have more folks looking at incorporating https://www.previousnext.com.au/blog/automate-your-drupal-accessibility-...

The more things can be tied into workflows like CircleCI the more likely developers will pay attention to these automated tests.

Accessibility efforts need to be able to demonstrate that the product is more accessible today than it was yesterday.

The Drupal Accessibility Scanner module has a nice mechanism to see changes over time.

Like many accessibility tools, it is following the best practices of using Deque's open source axe-core engine. Axe has been developed to avoid false positives. Axe does give warnings as well as errors, but these need to be manually validated.

To replicate & advance on the TestSwarm which was part of D7's history with QuailJS, I would recommend using Pa11y to with the axe engine (much as described in this blog post for GitHub Pages). I do not know if the Pa11y Integration module is of any use, but think not.

Ideally before every commit (and possibly with every PR) Drupal core would be tested for axe errors through the CI/CD pipeline to see if axe reports any errors. We should be able to say (at some point) that all Drupal Core is "axe clean" or has no axe errors. We would need to test sample pages against with pa11y (with axe) for the Core themes. This would absolutely need to include the home page, search and login page, but should also include sample content types and standard views.

It should also include tests for the authoring/admin side, as this is also part of our commitment to WCAG 2.1 AA. Fortunately, there are some resources for how to manage logins with pa11y that might come in handy. Also from Albert Albala's blog here. There's also docs for this too. I don't know what paths we will need to explore on the admin side, but we should start with the ones used most commonly by editors and administrators and move from there.

This will catch many common accessibility issues early. Unfortunately, it is only between 30-60% of them, but it is sizeable and important. Pa11y catches the pages as they are generated and does not evaluate dynamically changing content. Cypress-axe can do that, but first lets get pa11y set up to evaluate the basics.

Wanting to add a note about the Pa11y Integration module.

This would be good for developers.

I don't know if it helps, but I will be tagging a v2 -dev release of Editoria11y in the coming week that adds a REST API callback for issues discovered and generates a dashboard. Something like that might work well for editorial issue detection, and could be batch updated by a headless crawler if I set up a role/permission to let the crawler write to the database.

Just reposting my thoughts from Slack where @mgifford asked me. The suggested pa11y solution sounds like a good choice since it is based on aXe. If pa11y would be preferably over the other solutions brought up in the other comments and the IS i am unable to assess. At least one advantage to note is that for pa11y there is already an existing integration with gitlab out of the box available according to their docs: https://docs.gitlab.com/ee/user/project/merge_requests/accessibility_tes... which i would consider as a big plus.

and in regards of the general testing strategy i've stumble across the following article: https://blog.testproject.io/2022/02/07/accessibility-testing-best-practi... which brought up one or two good additional ideas.

The author suggests that linting would be worth a thought as an initial step. So would it make sense to add instructions to the developer documentation how to set up the coding environment for the work in drupal core and contrib and add there an instructional part about a11y linting? That way developers have already an initial feedback directly in their coding environment. That could create more initial awareness and could lower the number of warnings and errors in the next step as soon as a11y checks are added to the CI process.

And the points he makes in the Create a Testing Strategy section of the article sound reasonable and thoughtful by not starting of with the whole set of tests but go in stepwise over the weeks and introduce more of the desired set of tests. So maybe make the steps @mgifford added and suggested even more granular?

Agreed that a testing strategy could help with roadmapping accessibility testing. I like the idea of layering the approach to accessibility testing i.e. linting, accessibility unit tests with Nightwatch, pa11y for full page and sitewide testing.

I looked at what it would take to add @rikki_iki's nightwatch-axe-core package to core's Nightwatch tests and here is what I came up with.

Easy enough to add the package using yarn and it installs axe-core 4.4.2:

$ yarn add nightwatch-axe-core
yarn add v1.22.17
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
[4/5] Linking dependencies...
[5/5] Building fresh packages...
success Saved lockfile.
success Saved 2 new dependencies.
info Direct dependencies
└─ nightwatch-axe-core@1.0.3
info All dependencies
├─ axe-core@4.4.2
└─ nightwatch-axe-core@1.0.3
Done in 120.60s.

Following https://www.previousnext.com.au/blog/automate-your-drupal-accessibility-..., I had to add axe-core commands to the Nightwatch configuration. I did that by editing 'core/tests/Drupal/Nightwatch/nightwatch.conf.js' and adding two lines. Here is the diff:

diff --git a/core/tests/Drupal/Nightwatch/nightwatch.conf.js b/core/tests/Drupal/Nightwatch/nightwatch.conf.js
index 58bdfd8417..8ef5c991c2 100644
--- a/core/tests/Drupal/Nightwatch/nightwatch.conf.js
+++ b/core/tests/Drupal/Nightwatch/nightwatch.conf.js
@@ -44,6 +44,9 @@ Object.keys(collectedFolders).forEach((folder) => {
   collectedFolders[folder] = Array.from(new Set(collectedFolders[folder]));
 });
 
+// Add accessibility testing commands.
+collectedFolders['Commands'].push('./node_modules/nightwatch-axe-core/commands');
+
 module.exports = {
   src_folders: collectedFolders.Tests,
   output_folder: process.env.DRUPAL_NIGHTWATCH_OUTPUT,

I then created a example test, 'tests/Drupal/Nightwatch/Tests/exampleA11yTest.js' (copy of exampleTest.js),

module.exports = {
  '@tags': ['core'],
  before(browser) {
    browser.drupalInstall({
      setupFile: 'core/tests/Drupal/TestSite/TestSiteInstallTestScript.php',
    });
  },
  after(browser) {
    browser.drupalUninstall();
  },
  'Accessibility test': (browser) => {
    browser
      .drupalRelativeURL('/test-page')
      .waitForElementVisible('body', 1000)
      .axe()
      .drupalLogAndEnd({ onlyOnError: false });
  },
};

Running the test returned no errors (nice!):

$ yarn test:nightwatch tests/Drupal/Nightwatch/Tests/exampleA11yTest.js 

yarn run v1.22.17
$ cross-env BABEL_ENV=development node -r dotenv-safe/config -r @babel/register ./node_modules/.bin/nightwatch --config ./tests/Drupal/Nightwatch/nightwatch.conf.js tests/Drupal/Nightwatch/Tests/exampleA11yTest.js

[Tests/Example A11y Test] Test Suite
====================================
ℹ Connected to chromedriver on port 9515 (2838ms).
  Using: chrome (74.0.3729.157) on Linux platform.

✔ Running Accessibility test:

✔ Element <body> was visible after 54 milliseconds.
✔ Passed [ok]: Axe: 26 passed.

OK. 2 assertions passed. (2.163s)
Done in 260.69s.

I also injected 'axe()' in the 'search narrow form is accessible' test from 'tests/Drupal/Nightwatch/Tests/Olivero/oliveroSearchFormTest.js' and ran that and I got a few errors but that was for the entire page. Here is what the output looks like (removed the other test results):

$ yarn test:nightwatch tests/Drupal/Nightwatch/Tests/Olivero/oliveroSearchFormTest.js 
yarn run v1.22.17
$ cross-env BABEL_ENV=development node -r dotenv-safe/config -r @babel/register ./node_modules/.bin/nightwatch --config ./tests/Drupal/Nightwatch/nightwatch.conf.js tests/Drupal/Nightwatch/Tests/Olivero/oliveroSearchFormTest.js

[Olivero/Olivero Search Form Test] Test Suite
=============================================
ℹ Connected to chromedriver on port 9515 (935ms).
  Using: chrome (74.0.3729.157) on Linux platform.

...
✔ Running search narrow form is accessible:

✔ Element <#header-nav> was visible after 71 milliseconds.
✔ Element <.block-search-narrow .search-form.search-block-form> was visible after 65 milliseconds.
-----4 aXe violations-----
#1: Elements must have sufficient color contrast (color-contrast)
Impact: serious
Count: 1
See: https://dequeuniversity.com/rules/axe/4.4/color-contrast
#1.1: Fix any of the following:
  Element has insufficient color contrast of 3.68 (foreground color: #5d7585, background color: #cfe4f0, font size: 9.0pt (12px), font weight: normal). Expected contrast ratio of 4.5:1
  <span class="feed-icon__label">
    Drupal
  </span>

#2: Heading levels should only increase by one (heading-order)
Impact: moderate
Count: 1
See: https://dequeuniversity.com/rules/axe/4.4/heading-order
#2.1: Fix any of the following:
  Heading order invalid
  <h4 class="label">Member for</h4>

#3: Ensures landmarks are unique (landmark-unique)
Impact: moderate
Count: 1
See: https://dequeuniversity.com/rules/axe/4.4/landmark-unique
#3.1: Fix any of the following:
  The landmark must have a unique aria-label, aria-labelledby, or title to make landmarks distinguishable
  <div class="search-block-form block block-search-narrow" data-drupal-selector="search-block-form" id="block-olivero-search-form-narrow" role="search">

#4: All page content should be contained by landmarks (region)
Impact: moderate
Count: 1
See: https://dequeuniversity.com/rules/axe/4.4/region
#4.1: Fix any of the following:
  Some page content is not contained by landmarks
  <h2 id="primary-tabs-title" class="visually-hidden">Primary tabs</h2>

--------------------------
✖ Failed [fail]: (Axe: 4 rule violation(s). See aXe violations output for details)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) 


FAILED: 1 assertions failed and  2 passed (2.796s)
...

I think those Axe errors can be reviewed and the issues potentially fixed.

The 'axe()' function can take a context and options to focus only on the area that you want to test instead of the entire page (example shown here https://github.com/rikki-iki/nightwatch-axe-core#per-test-configuration). Here is the test code with 'axe()' function only focusing on the search fields:

  'search narrow form is accessible': (browser) => {
    browser
      .resizeWindow(1000, 800)
      .drupalRelativeURL('/')
      .click(mobileNavButtonSelector)
      .waitForElementVisible(headerNavSelector)
      .waitForElementVisible(`${searchNarrowSelector} ${searchFormSelector}`)
      .axe(`${searchNarrowSelector} ${searchFormSelector}`);
  },

Rerunning the test produced no Axe errors.

$ yarn test:nightwatch tests/Drupal/Nightwatch/Tests/Olivero/oliveroSearchFormTest.js 
yarn run v1.22.17
$ cross-env BABEL_ENV=development node -r dotenv-safe/config -r @babel/register ./node_modules/.bin/nightwatch --config ./tests/Drupal/Nightwatch/nightwatch.conf.js tests/Drupal/Nightwatch/Tests/Olivero/oliveroSearchFormTest.js

[Olivero/Olivero Search Form Test] Test Suite
=============================================
ℹ Connected to chromedriver on port 9515 (1722ms).
  Using: chrome (74.0.3729.157) on Linux platform.

...
✔ Running search narrow form is accessible:

✔ Element <#header-nav> was visible after 141 milliseconds.
✔ Element <.block-search-narrow .search-form.search-block-form> was visible after 62 milliseconds.
✔ Passed [ok]: Axe: 8 passed.
...

I think accessibility testing in core with Nightwatch and specifically with the nightwatch-axe-core package by @rikki_iki is doable and something that maybe should be pursued sooner?

Ooh glad to hear nightwatch-axe-core was simple to include as a POC. Happy to address any questions, concerns or feedback around potentially including it (happy to share access and what not). We've had it running on all our projects for the last 2 years and hardly come across any issues. It probably does need a bit of updating but this would be a good push for me to do that lol

Thanks @rikki_iki. What's your recommendation for the axe.conf.js, do you leave it the default or add specifics wcag values? Also I think the location of that file for Drupal core would be the core folder which might be fine. I was thinking maybe it should go into the core/tests/Drupal/Nightwatch but would that file be picked up? I can give it a try but wanted to ask as well.

It would be great to get a list of the URLs that are presently being covered by axe. There are some issues sneaking into Core that should have been caught by automated testing.

As with any good CI/CD process, it is important to monitor that list of URLs on a regular basis to ensure that we are using the best sampling that we can.

Phase 1 & 2 in the description identified some key URLs. It is unclear (without digging into the code) which URLs are being tested.

RE #25, I found these test case URLs.

• "/"
• "/admin/content"
• "/admin/structure"
• "/admin/structure/block"
• "/admin/structure/taxonomy/add"
• "/admin/structure/types/add"
• "/node/add/page?destination=/admin/content"
• "/search/node"
• "/user/1/edit" (tested with and without the navigation module
• "/user/login"

In these files:

• a11yTestDefault.js
• a11yTestAdmin.js
• a11yTest.js

I'm wondering if there's an error in the admin tests.

{ name: 'Create Article', path: '/user/1/edit' },

Should perhaps be:

{ name: 'Create Article', path: '/node/add/article?destination=/admin/content' },

There are some issues sneaking into Core that should have been caught by automated testing.

Some of these missed issues may be a result of the limited nightwatch_a11y_testing profile. A bunch of Axe errors appear when I use the modules from the standard profile.

I've been digging into it a little more before I create an issue.

Certainly, that create article path looks wrong.

I created this issue and fixed the path.
#3501457: Incorrect path used in a A11y Test Admin

I have also created: #3501493: Get nightwatch a11y test coverage to use Standard profile to pickup the rest of the coverage

Make some custom assertions about the component we're interested in. A dialog appears, has expected markup, focus moves to dialog. And finally run all the general ruleset AGAIN.

I can see also this for:

• Disclosure pattern in general.
• Menus.

I'd also like to have a policy / coding standard for tests regarding the locating of elements by name, or role & name, when possible.

It's closer to how users interact with the page and would build in a check for the correctness of the name (if the name is incorrect, the element won't be found). Sweeps with Axe will find missing names but won't confirm that they are correct.

It looks like Nightwatch doesn't currently offer a clean solution, but Playwright does. So, maybe after #3467492: [policy, no patch] Replace Nightwatch with Playwright. Or, we could implement the workaround described in the Nightwatch GitHub issue as a custom Nightwatch command.

Testing for Drupal core now pretty much fully within core itself. (If a custom container is needed, there is https://www.drupal.org/project/drupalci_environments, although well-supported 3rd-party containers are preferred.) If this can land in a way that contributed projects could take advantage of, there is https://www.drupal.org/project/gitlab_templates