Problem/Motivation
So far, we have very little functional test coverage for keyboard operations, distinct from pointer operations.
Prior to using WebDriver this was awkward anyway, because Goutte keyboard driving was flaky.
In the last couple of years we have started to add some keyboard test coverage:
- #2711907: [regression] Some ajax-enabled buttons are not keyboard operable - a simple test scenario, which illustrates the awkwardness of doing it with Goutte. Keyboard events were unreliable, so this test did it by evaluating some inline JS. We don't need that workaround now we use WebDriver.
- #2725259: [regression] Table Drag handles no longer respond to up/down arrow keys - the keyboard tests here involve a more complex sequence of keyboard operations.
We now have some significant experience of writing functional tests for keyboard operations. So let's start expanding our keyboard accessiblility test coverage...
Proposed resolution
1) proposed text
Add new heading to the core testing gate
Add Keyboard operation test
When is this needed?
- When adding a new UI feature.
- Changes that involves the JS keyboard, focus, or blur events.
Details
- Ensure compliance with {TBA}
Resources
{Add link to d.o doc about how to write such a test}
2) Create or add the following to a d.o doc about testing
Things we can write keyboard behaviour tests for
Here's an initial survey to find testable keyboard behaviours.
- "Mobile menu" behaviour at small breakpoints, e.g. in Bartik and Umami themes. Is the menu operable with a keyboard?
- Assert: the menu button opens and closes using either the enter or spacebar keys.
- Assert: the nested menu items are NOT in the keyboard tabbing order when the menu is closed. If they are, that's a failure of WCAG "Focus visible".
- When a dialog, opens, focus moves inside it.
- Dialogs can be dismissed by pressing ESC.
- When a dialog is dismissed, focus returns to the button that opened it. May need separate tests for each place that dialogs are used.
- Assert that the colllapse.js details works with spacebar and enter.
- Assert that keyboard focus does not move to operable controls which are concealed inside containers which are in a closed state. For example only controls in the active vertical tab can be tabbed to, not controls in the other tabs.
- The CKEditor toolbar configuration UI. Lots of interesting scenarios there.
You can tag issues with "needs accessibility review" to get help with devising keyboard scenarios and/or clarify expected behaviour.
Remaining tasks
- Catalogue existing keyboard operations we can add test scenarios for. In progress, forming a list under a separate heading above.
- There is a manual test plan somewhere. Are there any keyboard related things there?
- Find the existing issue about converting the manual test plan to Nightwatch.
Comments
Comment #2
andrewmacpherson commentedInitial brain dump, please discuss :-)
I'm not sure what the best component is, or the preferred topic tags for testing-related issues
Comment #3
andrewmacpherson commentedAdded a few example assertions about keyboard behaviour we could write.
Comment #4
andrewmacpherson commentedWe can also include assertions that dynamic aria-* properties are correctly updated to reflect the UI state, e.g. sortable table columns. Some of this might already have Functional JS tests, when testing with click(). It would be good to include these assertions in the keyboard tests too, in case we have any aria-* properties which are updated by mouse* event handlers, and omitted from key* handlers.
Comment #5
andrewmacpherson commentedComment #6
tim.plunkettFixing tags
Comment #12
mgiffordI like this idea. With automated tools and simple keyboard-only testing we can catch the vast majority of errors.
Comment #16
quietone commentedJust changing title per Special titles.
Comment #18
quietone commentedUpdated the IS with proposed text for an addition to the core testing gate. It needs work for details so changing status.