Should there be an official list of assistive technology supported by Drupal core?

Splitting this question into two parts. Still quoting verbatim from the earlier issue, though.

Related, do we need to update Drupal's list of supported browsers to include screenreaders, and other assistive software like Dragon?

It's been suggested before, and I've thought about it a lot. Short version: I'm strongly against it. It's a very different situation to browser support, and I don't think it's remotely practical to form a policy on which AT is supported vs. not supported.

We can make an informed decision about our supported browsers because:

1. There are browser usage figures, from several independent traffic data sets with a large sample size.
2. Feature support is well documented by Can I Use, MDN, etc.
3. Most of the browsers have open bug trackers, and beta/dev/nightly preview release channels for testing.
4. Polyfills are an option which helps make the decision to drop browser support easier.
5. Browsers don't cost users money to upgrade.
6. It's a fairly short learning curve when a user decides to switch to a different browser. Browser UI's have been converging for a number of years, and there isn't a great deal to distinguish them any more.

We can't say the same for assistive technology:

1. We don't have good usage stats. There are a few interesting user surveys, but they don't have anywhere near the sample size of the browser stats. There are also important privacy reasons against tracking assistive tech usage, and there's a growing consensus in the a11y community that OS and browsers should aim to prevent this. Meanwhile, there are kinds of assistive tech which even the OS can't detect.
2. Projects like a11ysupport.io may eventually be a great resource for accessibility standards support, but the project is still very young. There isn't yet anything as comprehensive as Can I Use.
3. Most commercial assistive technology doesn't have public bug trackers or preview releases. Heck, many don't even have useful release notes.
4. We can't polyfill assistive tech, or OS-level accessibility APIs, and I don't expect this situation to change. There are also some accessibility things which can't be yet be detected/polyfilled in browsers, though this may become feasible when the Accessibility Object Model matures and is well supported.
5. Some assistive tech is expensive to purchase and upgrade (like, Photoshop expensive; not some $5 app). That complicates the decision about when to drop support; there's a far greater risk of leaving users stranded than with browser support. OS vendors are bundling more assistive tech with the OS, but this has been slow and much third-party assistive tech still dominates the field.
6. Assistive technology has a much higher learning curve than most web browsers. If a user needs to change to a different screen reader because we have dropped support for one, that's likely to be very disruptive.

Previously, I've gone looking for comparable web frameworks which have a list of officially-supported assistive tech. So far I have found very little.

• The closest thing I've seen to such an assistive technology support policy is CKEditor 4 Accessibility Support page.
  • The only types of assistive tech mentioned are screen readers, and Windows high-contrast mode. There's no mention of speech control, switch access, magnifiers, or other forms of colour adjustment.
  • They officially support exactly ONE screen reader (and consequently, just one OS): "In our accessibility testing we use the latest Firefox version with latest JAWS version as our reference environment and officially supported platform. This is because we have found this combination to be most reliable and standards-compliant.". The standards-compliant part is somewhat controversial; JAWS is one of the old breed of screen readers which hooks directly into the browser process to look at the DOM, rather than dealing solely with what the OS-level accessibility APIs provide. In a sense, JAWS is cheating somewhat; VoiceOver, Narrator, and Talkback can only deal with what the OS-level accessibility APIs have implemented. Another criticism of this is hubris: if you only test/support one browser and screen reader, then there's a risk that what you're really doing is targeting that platform, and not conforming to standards yourself. This all makes their next assumption unreliable in my view...
  • "we assume that since we test in the most standards-compliant solution, other standards-compliant environments should also work correctly." It's hard to argue with this (as worded), but there's a LOT of wishful thinking going on here. Firstly, how do they know that Firefox/JAWS is the most standards-compliant combination? Secondly, "others... should also work" doesn't take actual implementations into account (browser, OS, and assistive tech), and this varies a lot. So I don't think this part of their policy really addresses WCAG's notion of only relying on accessibility supported technology, or WCAG guideline 4.1 ("maximise compatibility with existing user agents including assistive technologies").
  • I recall reading an issue in the CKEditor bug tracker, where a problem was noted with one browser/screen reader combination. The response from another commenter said something along the lines of "that isn't our support target". Effectively, it was dismissed as won't-fix, without much discussion about whether it could be addressed. I think their policy encourages this, which is a pity. (TODO: did I bookmark the issue?)
  • Overall, I feel their policy is along the lines of "best viewed in Netscape". I'm reticent of following this policy for Drupal.
• I also found an IBM open source UI widget collection (called Deliteful, or something?) which included useful testing notes about assistive tech behaviour with individual components. It didn't amount to a supported AT policy though.

===========

It would be helpful to have a known list to test against.

That's a very different proposition to a list of officially-supported AT. I'm still thinking about this. There's a big difference between "we tested a feature using X and Y" and "we only officially support X and Y".

The accessibility topic is about much more than just the capabilities of the assistive technology itself. It's also important to consider the variety of users and situations where AT is used. So rather than a list of assistive technology, a list of testing scenarios may be more appropriate. We already have an accessibility handbook page (mostly compiled by @lauriii) which has a lot of test scenarios. Assistive tech sections could be expanded there. For example, what to look out for with WCAG "Name, Role, Value". Other times, it might be better to treat assistive tech testing per feature or issue. Something like "needs manual assistive tech test steps".

My own assistive tech testing habits (not always consistent; often it's because I'm curious about a particular scenario):

• About a half-dozen combinations of browser viewport and zoom. We've used this plan for Claro, in #3087225: [META] Assess Claro's conformance with WCAG Text spacing, Text resize, and Reflow..
• Arbitrary font changes. I used to have a Firefox add-on for this, but it stopped working after v.57 Quantum. Now I use browser settings to disallow author fonts, or just tinker with font-family in dev tools.
• Windows high-contrast mode. Black-on-white and Yellow-on-black pre-set themes. Plus a custom theme with arbitrary colours chosen at whim, to catch the assumption that a pre-set theme is in use (currently I have a sort of Miami Vice pink-on-aqua). I test HC with IE11, Edge17, Firefox. Edge Chromium Canary recently added support behind an experimental flag.
• Other colour modes. Greyscale mode (Android, KDE Plasma, MacOS). Colour blindness emulators (Android dev settings, pity you can't screenshot the result).
• Screen readers: JAWS + NVDA, with IE11, Firefox, Chrome. Occasionally testing with Edge17/Narrator. Android9/Talkback/Chrome, iOS12/Safari/VoiceOver, MacOS/Safari/VoiceOver, MacOS/Chrome/VoiceOver. The exact choice varies; Windows screen readers are my main choice, because that's what most users have in the screen reader user surveys. By the same logic, though, I ought to do more iOS VoiceOver testing...
• Speech control: I manually inspect a lot of controls for WCAG "label in name" violations. Also check when "show numbers" puts a number in a location where there isn't any visible control. Mainly using Android Speech Control and MacOS Dictation. I need more practice with Dragon.
• Screen magnifiers: MacOS and KDE Plasma built-in magnifier tools. Magnifier set-ups vary a great deal. This is very informal. I'm not aware of any good published resources for magnifier walk-through tests. So far I have a short list of "bad smells" which I look for involving left/right pairings, large areas of white space, and content on hover.
• Keyboard testing. This part is my actual life. I'm a keyboard-mostly user.

Misc thoughts:

JAWS only runs on Windows, and is expensive 3rd-party assistive tech. I'm curious what proportion of CKEditor devs use Windows as their main environment, and actually have JAWS. DrupalCon events are typically a vast sea of MacBooks, and whenever I look back over old accessibility issues from the early days of D8 development, it's rare to see any mention of a screen reader other than VoiceOver. So like CKEditor, I think we have already wandered into a trap of "works in VoiceOver" to a degree. You could hear it in our use of aria-pressed for some disclosure controls, where aria-expanded is the correct property to use; I wonder if we choose that because VoiceOver announces it as a "toggle button"?

OTOH, if a developer uses a Mac as their daily machine, it's a stretch to expect them to test regularly with NVDA because that only runs on Windows. Vice versa too. But testing with at just one screen reader is better than not testing with any screen reader at all. I think it's more useful if an issue has notes about what testing has been done, rather than having a grand test document, and not knowing whether all the different browser/screen reader combinations have been done.

I've seen plenty Drupal issues which mention screen reader testing, but buried in comments. It would be good to encourage these to be noted in the main issue summary. (e.g. "Screen reader testing: VoiceOver/mac, see comments #16-20, and #42")