Provide a new library to replace jQuery UI autocomplete

Thanks for splitting this off and making each issue title & issue summary so clear, @bnjmnm! 🙏👏Looking forward to patches to review 🤓

I just fixed a few typos!

Note: I manually tested using the field_tags field at /node/add/article.

I suspect what's really needed here is to try to break this in as many ways as possible. I did some of that, and found a few problems. You as the person who worked on the code probably have a more informed hunch of where the risks are, i.e. how it could break. It'd be great if you could document your own attempts of breaking it in a comment — or better yet, share those in the form of tests. Then none of us would have to repeat that manually!

Overall, it already seems to be working very well! 👏

1. +++ b/core/core.libraries.yml
   @@ -104,16 +104,24 @@ drupal.announce:
   +  version: VERSION
   

   🐛 This needs the included asset library's version, not VERSION which means "core's version".

   It also needs the license key, just like all other 3rd party asset libraries.

2. +++ b/core/core.libraries.yml
   @@ -104,16 +104,24 @@ drupal.announce:
   +    - core/drupal.announce
   

   🥳 Oh! Exciting! I'll be testing this specifically then.

3. +++ b/core/core.libraries.yml
   @@ -471,20 +479,6 @@ jquery.ui.accordion:
   -jquery.ui.autocomplete:
   

   👎 We can't delete it, we can only deprecate it.

4. +++ b/core/lib/Drupal/Core/Entity/Element/EntityAutocomplete.php
   @@ -142,6 +142,21 @@ public static function processEntityAutocomplete(array &$element, FormStateInter
   +    // Create attribute that will be used for client side enforcement of field
   

   🤓 Nit: s/client side/client-side/

5. +++ b/core/misc/autocomplete.es6.js
   @@ -55,150 +55,89 @@
   +   * Detemines if a suggestion should be an available option.
   

   🤓 Nit: s/Detemines/Determines/

6. +++ b/core/misc/autocomplete.es6.js
   @@ -55,150 +55,89 @@
   +    const cardinality = inputElement
   +      .closest('[data-autocomplete-cardinality]')
   +      .getAttribute('data-autocomplete-cardinality');
   

   🤔 Needs to handle the case of no cardinality metadata being present. Or fail explicitly.

7. +++ b/core/misc/autocomplete.es6.js
   @@ -55,150 +55,89 @@
   +      'data-autocomplete-first-character-blacklist',
   

   👎 Let's use different terminology — see #2993575: [meta] Remove usage of "blacklist", "whitelist", use better terms instead.

8. +++ b/core/misc/autocomplete.es6.js
   @@ -55,150 +55,89 @@
   +    // Do not show items already added to field.
   +    if (currentValues.indexOf(suggestionValue) !== -1) {
   +      return false;
        }
   
   @@ -213,40 +152,113 @@
   +        // Screenreader announces the number of results found, and if it is
   +        // the first set of results for a field, additional instructions on
   +        // navigating the results are provided.
   ...
   +            'There is one result available.',
   +            'There are @count results available.',
   

   🐛 These operate independently of each other, but the announcing of the number of read results needs to be aware of omitted suggestions.

   Otherwise this results in incorrect announcements.

   For example: terms "bar" and "baz" exist, "bar" is already entered, type "b" to search. You see 1 suggestion ("baz") but the screen reader announces there are two results.

9. +++ b/core/misc/autocomplete.es6.js
   @@ -213,40 +152,113 @@
   +        // The first time autocomplete results appear on a field, the
   +        // screenreader provides detailed instructions on how to navigate the
   +        // options. After the first time, this variable is set to true so the
   +        // screenreader does not provide lengthy instructions each time.
   +        let autocompleteInstructionsRead = false;
   

   👍 This is a nice touch! This will need to be verified by an accessibility maintainer in the end, but I can confirm it works as advertised :)

10. +++ b/core/misc/autocomplete.es6.js
    @@ -213,40 +152,113 @@
    +        // Add class previously added by jQueryUI for backwards compatibility.
    +        autocompleteElement.classList.add('awesomplete-input');
    

    🤔 It seems unlikely that jQuery UI was adding the class awesomeplete-input? Did you mean to add ui-autocomplete here? That would also allow some test changes to be reverted. Although I'm not sure that's really worth it.

11. 🐛 When typing a single character, that yields autocomplete suggestions. Good. But when you delete that single character, the autocomplete suggestions remain visible. That seems like a bug in our implementation, because https://leaverou.github.io/awesomplete/'s demo doesn't do this. Note this only happens for any entry that isn't the first — it does not happen when choosing a first entry!
    The current jQuery UI-based solution also does not do this.
12. 👍 I explicitly tested with accented characters and emoji in an attempt to break it 😏, this worked fine.

We need a dependency evaluation for Awesomplete here. Thanks!

This is looking really good @bnjmnm! I think awesomeplete is an excellent choice, and I'm a big fan of Lea Verou too.

I've tried to address @Wim Leers feedback:

1. Changed version to 1.1.5 and added the license info
2. I did some testing with Mac OS voiceover on
3. Restored the jquery.ui.autocomplete asset lib, added deprecation message.
4. Added hyphen
5. Fixed spelling
6. I think this is covered in
   

               +++ b/core/lib/Drupal/Core/Entity/Element/EntityAutocomplete.php
               @@ -142,6 +142,21 @@ public static function processEntityAutocomplete(array &$element, FormStateInter
               +    // Create attribute that will be used for client side enforcement of field
               +    // cardinality.
               +    $name = explode('[', $element['#name'])[0];
               +    if (!empty($complete_form[$name]['widget']['#theme']) && $complete_form[$name]['widget']['#theme'] === 'field_multiple_value_form') {
               +      $cardinality = 1;
               +    }
               +    elseif (!empty($complete_form[$name]['widget']['#cardinality'])) {
               +      $cardinality = $complete_form[$name]['widget']['#cardinality'];
               +    }
               +    else {
               +      $cardinality = -1;
               +    }
               +
               +    $complete_form[$name]['#attributes']['data-autocomplete-cardinality'] = $cardinality;
               +
           

   However I also added an additional expression to set cardinality to -1 if the wrapper element with that attribute is not found.

7. Changed to "denylist" per the variable names in the latest patch in #2993575: [meta] Remove usage of "blacklist", "whitelist", use better terms instead
8. This was a really good catch, at first I was only looking at the markup that awesomeplete generates and it seemed accurate. This bug is with the value that gets sent to "announce".
   jQuery UI autocomplete doesn't filter the lists as well as awesomeplete, either- after selecting an item it was still available in the list.
   

   Changed the value passed here to be the same count awesomeplete has after evaluating the cached autocomplete list.

9. I like this too, there also seems to be something in "announce" preventing some other repetitions but I didn't dig into it.
10. Changed this back to ui-autocomplete-input, but I left awesomeplete in JSWebAssert and EntityReferenceAutocompleteWidgetTest since that is the class that gets added to the wrapper div.
11. This does seem a little buggy, I may be misunderstanding but it seems to work the same as http://leaverou.github.io/awesomplete/#multiple-values
    
    select one word, type a character then hit backspace and the full list appears.

#9 I'll open an issue to start the dependency evaluation, and post the reply here.

I'm concerned that changing this library in a minor release will be disruptive because this completely replaces the suggestions markup, meaning that any CSS targeting the old autocomplete suggestions wouldn't work anymore. How do we make sure that themes have enough time to prepare for this update? It almost seems like there we should try to find a way for themes to decide which markup they support, and we would deprecate the jQuery UI autocomplete option to be removed in Drupal 9.

I'm also concerned about the potential disruption caused by adding the new <div class="awesomplete"> element. This change could break selectors, for example .form-type-entity-autocomplete > input would be affected.

It seems like the default design is also different, meaning that anyone relying on the default designs would also see changes in the UI after this change. I suggest that we try to customize the design to be similar to jQuery UI to minimize potential disruption.

We should try to move as much of the markup as possible to theme functions to make it easier for themes to customize the design of the autocomplete suggestions.

I'm concerned that changing this library in a minor release will be disruptive because this completely replaces the suggestions markup, meaning that any CSS targeting the old autocomplete suggestions wouldn't work anymore. How do we make sure that themes have enough time to prepare for this update?

Can we ship both implementations, and then make it a site-level configuration option for which one gets used in 8.8 and 8.9 (defaulting to jQuery UI)? Then complete replacement in Drupal 9?

Maybe we could ship awesomplete as a module replacing the core/drupal.autocomplete library via the library alter hook?

I reviewed the patch from this angle. Besides adding the library, two PHP files are changed here. The change to FormState just avoids an error, we could ship it without a problem. And adding a new data-autocomplete-cardinality attribute to EntityAutocomplete looks like a minor and non-breaking markup change. So both of those could stay and be present even if awesomplete is not enabled. This would also put minimal burden on the awesome people working on the patch here since the necessary plus code is just an info and a very small module file (famous last words, I know) and would avoid a configuration form and all that.

Using a module means we'll eventually need to deprecate/uninstal/remove the module when we do the change 100% in core, but we're doing that for other modules (like migrate multilingual) already so seems viable to me.

Using a module means we'll eventually need to deprecate/uninstal/remove the module when we do the change 100% in core, but we're doing that for other modules (like migrate multilingual) already so seems viable to me.

Another example of that is the Field Layout (experimental) module #3007167: [policy] Deprecate field_layout module and move it to contrib, so if this was done as a module it could also be marked as experimental, which might be appropriate given this will require a lot of manual experimenting by many users to find all of the edge-case issues and incompatibilities.

Tagging for future UX call.

I think this is a smart way to be more careful about BC @bnjmnm. There are a lot of places where jQuery UI classnames exist in core, such as in stable and seven. If we can get this patch landed we should add follow-up issues for eventually removing/changing the jQuery UI css.

If we put awesomplete in an optional module, what would the process be for updating all the places where the old classnames are used, and what would be the motivation for ever changing them in themes like Claro?

Besides classnames, the biggest difference in markup is that the jQuery UI suggestion list is appended to the body (then positioned with javascript), compared to the awesomplete markup which keeps the list as a direct sibling to the input.

In #18 testMarkup enforces the jQuery UI approach.

+++ b/core/tests/Drupal/FunctionalJavascriptTests/EntityReference/EntityReferenceAutocompleteWidgetTest.php
@@ -106,4 +116,112 @@ public function testEntityReferenceAutocompleteWidget() {
+    $suggestion_list = $page->find('css', 'body > .ui-autocomplete');

Why is it important to show that the suggestion list is a direct descendent of the body? I don't think that's necessarily a better way and I didn't find any examples of css which depended on this placement. If that body > is removed then it would be simpler to make this test pass without altering how awesomplete works.

If we put awesomplete in an optional module, what would the process be for updating all the places where the old classnames are used, and what would be the motivation for ever changing them in themes like Claro?

If awesomplete is shipped in an optional module, themes could support both, jQuery UI and awesomplete simultaneously. We should make core themes support both as part of the initial patch.

Why is it important to show that the suggestion list is a direct descendent of the body? I don't think that's necessarily a better way and I didn't find any examples of css which depended on this placement.

I don't think there's any reason for us to try to make awesomplete work that way. I think it was just an implementation detail in jQuery UI. I think we could change this selector to allow placing the suggestions list anywhere in the markup.

I'm not sure about trying to make awesomplete fully backwards compatible with jQuery UI. If we ever want to remove the bc layer we'll be faced with the same problem again.

If we go for the optional module approach, this isn't necessary - themes can just target both sets of markup until they/Drupal completely drops jQuery UI support.

This patch moves all the awesomplete code into an experimental core module and restores the changes to core.libraries.yml and misc/autocomplete.
I left @bnjmnm's changes to EntityAutocomplete and FormState, as well as some of the new tests, but I removed the markup test and the tests and new module from #22 which was added for BC.

The "loading" class is now being toggled, and I fixed a small issue with how list items were assigned an #id.

The module overrides the core library and replaces the dependency with awesomplete. I used the minified js and the "base" css so that it was easier to override with Drupal styles, I copied a few lines from Seven to give the dropdown list a default look, as well as some padding that the jQuery UI Autocomplete gets from "menu".

I looked into overriding the markup but it gets generated by awesomplete directly and is very minimal, there's an open feature request for custom templates

The previous patch failed because the InstallUninstallTest was looking for a specific string pointing to online documentation, I created a doc page and updated the help text and link.
https://www.drupal.org/docs/8/core/modules/experimental-awesomplete-auto...
This test didn't always finish on my local, it takes around 15-20 min to run for me. I experimented with local changes to the test to only install this new module and it seemed to be fixed but I'm not 100% confident this will pass now.

The remaining warnings were from the deprecation message, in https://www.drupal.org/project/drupal/issues/3067251#comment-13264865
I suppressed that deprecation warning in \Drupal\Tests\Listeners\DeprecationListenerTrait::getSkippedDeprecations()
That seemed like a valid approach here, too, since having this deprecation warning for core/jquery.ui.autocomplete appear to developers is one of the goals of this issue.

There are some other minor changes and nit fixes in this patch to comply with the linter and hopefully avoid more warnings from testbot.

Added drupal/awesomplete_autocomplete to composer.json under "replace"

1. +++ b/core/misc/autocomplete.es6.js
   @@ -257,32 +272,26 @@
      autocomplete = {
   ...
   -      source: sourceData,
   -      focus: focusHandler,
   -      search: searchHandler,
   -      select: selectHandler,
   ...
   -      minLength: 1,
   ...
   +      minChars: 1,
   

   Could we try to keep changes to this minimal since this is the API surface to the autocomplete? For example, changing minLength to minChars seems unnecessary.

   Also, removing source, focus, search and select seems potentially disruptive. How are we expecting people to extend this in future?

   If we can't replace these with anything, we should try to document how to achieve that (if it's even possible). And we should rename this to something else than autocomplete so that modules and themes can support both autocomplete libraries at the same time without having knowledge about this module.

2. +++ b/core/tests/Drupal/FunctionalJavascriptTests/EntityReference/EntityReferenceAutocompleteWidgetTest.php
   @@ -17,11 +19,19 @@ class EntityReferenceAutocompleteWidgetTest extends WebDriverTestBase {
   +  public static $modules = ['node', 'taxonomy'];
   

   Should we add test coverage for awesomplete as well?

3. +++ b/core/modules/awesomplete_autocomplete/awesomplete_autocomplete.info.yml
   @@ -0,0 +1,5 @@
   +description: 'Provides a replacement for jQuery UI Autocomplete which will be deprecated in Drupal 9.0.'
   

   This seems a bit unclear. Is jQuery UI Autocomplete deprecated in Drupal 9.0 or is this module deprecated in Drupal 9.0?

4. +++ b/core/modules/awesomplete_autocomplete/js/autocomplete.es6.js
   @@ -0,0 +1,317 @@
   +  const extractTermID = term =>
   

   Autocomplete could be used for other types of entities than terms. Maybe extractEntityId would be a better name?

@catch and I discussed that since the experimental module we'd be adding here is equal parts improved product feature and security hardening, unlike most experimental modules, this could be added as a beta module in 8.9.x/9.0.x as well as 8.8.x or 9.1.x.

The patch in #35 looks good! It mostly addresses the feedback in #29 and reorganizes some of the js in a more sensible way.

#29.1 Since minChars is a property awesomplete is expecting in the options- we'd have to add a map function to rename the keys before the options are passed in.

It's not really possible to extend it in the same way, as the jQueryUI functions send different arguments

There isn't a 1:1 match for every option or method from jQuery UI autocomplete to awesomplete.

I do like this improvement to the options- separating out that object. It would be better if modules or themes could override these options, but I'm don't think adding the awesompleteOptions object to the Drupal global means it can be overriden here. Even if that global is changed by another module or theme it would already have been passed into the Awesomplete instance.
If we want this to be override-able it would need to be moved to a separate js file that can be replaced with libraries_override for example.

#29.2 The test coverage looks perfect, just extending the existing test but with this module installed. Clever! I ran it locally and it excecuted the 2 tests. I also paused to verify in the testing browser that the awesomplete markup was present.
From the comments in the test file:

If awesomplete_autocomplete is moved to core, this test can be removed as EntityReferenceAutocompleteWidgetTest will then be testing with Awesomplete

That seems ideal, it should function exactly the same as the old autocomplete so re-using the existing test makes sense.

#29.3 Adding the standard deprecation message to the description is an improvement.
#29.4 There were a few other places "term" is still being used, in comments and extractLastTerm

    +++ b/core/modules/awesomplete_autocomplete/js/autocomplete.es6.js
    @@ -0,0 +1,336 @@
    +   * Helper splitting terms from the autocomplete value.
    ...
    +    // We will match the value against comma-separated terms.
    ...
    +   * @function Drupal.autocomplete.extractLastTerm
    ...
    +   * @param {string} terms
    ...
    +  const extractLastTerm = terms => autocomplete.splitValues(terms).pop();
    ...
    +    return Awesomplete.FILTER_CONTAINS(suggestion, extractLastTerm(input));
    ...
    +      extractLastTerm(input),
    ...
    +    const term = autocomplete.extractLastTerm(input.value);
    ...
    +    if (term && term.length > 0) {
    +      if (autocomplete.cache[inputId].hasOwnProperty(term)) {
    +        awesomplete.list = autocomplete.cache[inputId][term];
    ...
    +        xhr.open('GET', `${apiUrl}?q=${term}`);
    ...
    +            autocomplete.cache[inputId][term] = results;
    ...
    +    extractLastTerm,
    

nit: There are also a couple comments where function names don't match.

    +++ b/core/modules/awesomplete_autocomplete/js/autocomplete.es6.js
    @@ -0,0 +1,336 @@
    +   * @function Drupal.autocomplete.splitValues
    ...
    +  const autocompleteSplitValues = value => {
    ...
    +   * @function Drupal.autocomplete.extractLastTerm
    ...
    +  const extractLastTerm = terms => autocomplete.splitValues(terms).pop();

    +++ b/core/.eslintrc.json
    @@ -19,7 +19,8 @@
    +    "Awesomplete": true

Good catch adding this to the eslint file!

Thanks for fixing those naming issues, and the approach of moving those options into drupalSettings is smart.
This all still looks good, and there weren't any functional changes in this last patch.

There seems to be a regression here.

With a term reference field, the current behavour is that if you manually type the full term into the autocomplete (for example in the middle of the list, or before autocomplete has a chance to, um, complete), then we'll query the database for the term to look it up when you submit the form. With awesomplete, it just says it's unable to find a matching entity.

This is related to #2881892: UX: Hide entity ID in autocomplete widget - I was testing the patch to see if it helps with that, but it appears to be even more dependent on the ID being in the textfield values.

With awesomplete, it just says it's unable to find a matching entity

@catch can you elaborate on this?

Awesomplete displays the label in the list like "News" but puts the value with the id into the input like "News (3)". If you type in the label only without letting Awesomplete fill in the value, saving still seems to find the referenced entity.

I'm sure I'm missing something, but I'm not seeing a regression in behavior.

The new tests @bnjmnm added pass for me locally and just went green from the testbot, looks good unless that test isn't proving the right thing!

If you type in the label only without letting Awesomplete fill in the value, saving still seems to find the referenced entity.

This is what I was testing but it wasn't on a clean 8.x install, so it might be my local rather than the code - good to have the extra test coverage though regardless! If I get a chance to retest I'll do so though.

@zrpnr has also posted #3089455: [META] Awesomplete roadmap so that we can split out anything that's not alpha-blocking into that meta issue. I've posed some questions there about how we should handle the path to stability, mostly not directly related to this patch.

This could be added to 9.1 now that it's open, too late for 8.9/9.0 but main thing is being able to deprecate remaining jQuery UI libraries for Drupal 10.

It looks good :) need a reroll and a few things:

1. +++ b/core/core.libraries.yml
   @@ -496,6 +496,7 @@ jquery.ui.autocomplete:
   +  deprecated: The "%library_id%" asset library is deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. See https://www.drupal.org/project/drupal/issues/3076171
   
   +++ b/core/tests/Drupal/Tests/Listeners/DeprecationListenerTrait.php
   @@ -147,6 +147,7 @@ public static function getSkippedDeprecations() {
   +      'The "core/jquery.ui.autocomplete" asset library is deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. See https://www.drupal.org/project/drupal/issues/3076171',
   

   Need to be updated, it's not removed from D9

2. +++ b/core/modules/awesomplete_autocomplete/awesomplete_autocomplete.info.yml
   @@ -0,0 +1,5 @@
   +core: 8.x
   

   need to use core_version_requirement

3. +++ b/core/modules/awesomplete_autocomplete/awesomplete_autocomplete.libraries.yml
   @@ -0,0 +1,23 @@
   +  version: 1.1.5
   

   Still up to date :)

4. +++ b/core/modules/awesomplete_autocomplete/js/autocomplete.es6.js
   @@ -0,0 +1,342 @@
   +        const xhr = new XMLHttpRequest();
   +        xhr.open('GET', `${apiUrl}?q=${searchTerm}`);
   +        xhr.onload = () => {
   

   I like the effort but we should stick with jquery for that. Having a dependency on jQuery is not making me happy but we have to keep things consistent. If you want to change how we do something, it needs to be discussed and the plan for how everything in core will be impacted need to be detailed before we make a move.

   Chances are another element on the page needs jQuery since we still rely on it a lot.

5. +++ b/core/modules/awesomplete_autocomplete/js/autocomplete.es6.js
   @@ -0,0 +1,342 @@
   +    attach(context) {
   +      const autoCompleteInputs = context.querySelectorAll(
   +        'input.form-autocomplete',
   +      );
   ...
   +      Array.prototype.forEach.call(autoCompleteInputs, autocompleteElement => {
   

   Need to use .once()

6. +++ b/core/modules/awesomplete_autocomplete/js/autocomplete.es6.js
   @@ -0,0 +1,342 @@
   +    detach(context) {
   +      const autoCompleteInputs = context.querySelectorAll(
   +        'input.form-autocomplete',
   +      );
   +      Array.prototype.forEach.call(autoCompleteInputs, autocompleteElement => {
   +        autocompleteElement.removeEventListener('input');
   +      });
   +    },
   

   Need to make sure the detach trigger is "unload", otherwise it will wreak havoc when ajax requests are made elsewhere on the page.

7. +++ b/core/modules/awesomplete_autocomplete/js/autocomplete.js
   @@ -0,0 +1,190 @@
   diff --git a/core/modules/awesomplete_autocomplete/js/awesomplete.min.js b/core/modules/awesomplete_autocomplete/js/awesomplete.min.js
   

   Can we put that in /assets with the rest of the third party code?

8. Doesn't work on claro, probably because of the override.

Working on it.

This change addressed about nothing of the current autocomplete issues. So, here is not asking for crazy NEW features. Even the `LeaVerou/awesomplete` is worse than the current autocomplete on my quick test.

Therefore, this awesomplete is: INPUT + DROPDOWN LIST

awesomplete is very lite:
https://github.com/LeaVerou/awesomplete/blob/gh-pages/awesomplete.js

above patch is about rewritten or provding the core part of the `LeaVerou/awesomplete`:

autocomplete = {
cache: {},
splitValues: autocompleteSplitValues,
extractLastEntityReference,
inputListener: autocompleteInputListener,
firstCharacterDenylist: ',',
};

now, you can jump out of the box and doing your own version with:
#1899236: Add new Splitbutton render element to eventually replace Dropbutton

You only need to adopt the INPUT to trigger a dropdown list I think.

Then you have the same experience on dropdown things in Drupal. :) Nope?

Unassigned this issue as of now if someone wants to pick this up. Otherwise, I will recheck after someday.

@bnjmnm,

Correct! Awesomplete is also a dropdown list builder like the Splitbutton does. Ajax and other parts are fed from customized coding. Awesomplete has no keyboard shortcuts, and you need to keep it the same experience for Drupal globally.

Awesomplete has very similar problems with the Core one:

RED: no deduplicated
GREY: the problem of change / start from the middle.

(sidenote: I think I've provided patches somewhere in Core threads to address these autocomplete issues)

+ Patch coding, Only 2 extra major features required to turn Splitbutton into Autocompelete:
- INPUT handler
- item selection handler (IMO, Splitbutton required this feature alone also)

I'm currently working on writing a custom implementation (library) on my own for Gin. Maybe this can be used/modified & optimized for a more global use.

It's basically using the default dropdown for all AT/Keyboard users (+ native select implementations on mobile like the revolver on iOS) but provides pointer users a custom solution which also works with keyboard search, filter etc.

Quick WIP Demo: https://twitter.com/saschaeggi/status/1291351015279796225?s=20

@saschaeggi

😍😍

that's same to my suggestion on above few comments I believed.

more:
https://www.drupal.org/project/drupal/issues/1899236#comment-13701448

So if Drupal implemented my idea, you will get the same thing with shared codebase

It's good to see you're the Lead of the Drupal Design. Probably, you & your team can figure out more sharable component and codebase :)

@bnjmnm quick note on the Gin select state: It's a early draft for sure.

I couldn't open it via keyboard navigation unless a screenreader is enabled
should be possible to open via keyboard (native select)

If the element is focused and clicked, the native select options appear instead of the styled ones.
based idea on this principle here https://css-tricks.com/striking-a-balance-between-native-and-custom-sele...

If opened at the bottom of the page, the option position does not adjust to be visible in the viewport. This can be fine for something like an autocomplete or dropbutton, but with a select there's an expectations the options will appear in the viewport.
This is just not yet implemented

Maybe this helps, Cheers :)

What does "gin select" refer to?

I recall being fairly impressed with Awesomeplete when I tried it a while ago. Since then I've found some situations where it mis-reports it's value. That's a hard blocker.

Hear a reroll patch.

Seems like something is missing from the patch given that it went from ~47kb to ~12kb.

Had a look, don't know how to fix the composer issue.

1. +++ b/core/composer.json
   @@ -54,6 +54,7 @@
   +        "drupal/d9_autocomplete": "self.version",
   

   not a fan of the name. maybe we can make a vanilla js module and put all the overrides in there (autocomplete, ajax, tabledrag, etc.)?

2. +++ b/core/modules/d9_autocomplete/d9_autocomplete.info.yml
   @@ -0,0 +1,4 @@
   +description: 'Provides a replacement for jQuery UI Autocomplete. jQuery UI Autocomplete is deprecated in Drupal 8.8.0 and will be removed in Drupal 9.0.'
   

   versions should be deprecated in D9.2, removed in D10 no?

3. +++ b/core/modules/d9_autocomplete/js/autocomplete.es6.js
   @@ -0,0 +1,492 @@
   +  Drupal.Autocomplete = class {
   

   Lacks significant docs :) and candidate for "extraction", might want to declare it outside the Drupal namespace already.

4. +++ b/core/modules/d9_autocomplete/js/autocomplete.es6.js
   @@ -0,0 +1,492 @@
   +      $(this.input).on('input', () => this.inputListener());
   +      $(this.input).on('blur', (e) => this.blurHandler(e));
   +      $(this.input).on('keydown', (e) => this.inputKeyDown(e));
   +      $(this.ul).on('mousedown', (e) => e.preventDefault());
   +      $(this.ul).on('click', (e) => this.itemClick(e));
   +      $(this.ul).on('keydown', (e) => this.listKeyDown(e));
   +      $(this.ul).on('blur', (e) => this.blurHandler(e));
   

   we could use the object version to avoid having several $(this.input) and $(this.ul) calls.

5. +++ b/core/modules/d9_autocomplete/js/autocomplete.es6.js
   @@ -0,0 +1,492 @@
   +      const separator =
   +        numItems < cardinality || parseInt(cardinality, 10) === 0 ? ',' : '';
   +      const before = this.input.value.match(/^.+,\s*|/)[0];
   +      this.input.value = `${before}${item}${separator}`;
   

   Should probably be configurable, or do we expect people to override the whole function?

6. +++ b/core/modules/d9_autocomplete/js/autocomplete.es6.js
   @@ -0,0 +1,492 @@
   +          const xhr = new XMLHttpRequest();
   

   we should get the fetch polyfill in core sonner or later. As an external lib I'd go with using fetch only.

7. +++ b/core/modules/d9_autocomplete/js/autocomplete.es6.js
   @@ -0,0 +1,492 @@
   +      // Wrapped in an `<a>` for backwards compatibility.
   

   we're going back to where with that? if it's for under IE11 I'd get rid of it.

8. +++ b/core/modules/d9_autocomplete/js/autocomplete.es6.js
   @@ -0,0 +1,492 @@
   +    announceSuggestionCount(count) {
   

   as an external lib I'd see an event being triggered for this.

9. +++ b/core/modules/d9_autocomplete/js/autocomplete.es6.js
   @@ -0,0 +1,492 @@
   +        ? this.input.getAttribute('data-autocomplete-first-character-denylist')
   

   maybe we can put the setting as a json string in a generic data-autocomplete attribute? like we did for #states

In any case it seems to be working just fine :)

1. Removing needs release notes and needs release manager review since I think there are couple of steps we probably want to take before working on those. Leaving needs accessibility review and needs issue summary update there since they seem still relevant.
2. We should probably expand our test coverage to explicitly test some of the features provided for improved accessibility. This would both act as documentation on the expected behavior, and ensure that it remains consistent as time passes.
3. Should users be able to navigate back to the input from the autocomplete list using arrows? This seems to be how Accessible Autocomplete and jQuery UI Autocomplete work.

4. +++ b/core/core.libraries.yml
   @@ -465,6 +465,7 @@ jquery.ui.autocomplete:
   +  deprecated: The "%library_id%" asset library is deprecated in drupal:9.1.0 and is removed from drupal:10.0.0. See https://www.drupal.org/project/drupal/issues/3076171
   

   This should point at a change record.

5. +++ b/core/modules/d9_autocomplete/js/autocomplete.es6.js
   @@ -0,0 +1,492 @@
   +      const wrapper = this.input.closest('[data-autocomplete-cardinality]');
   

   I don't think this is going to work with IE 11. We should support IE 11 until #3155358: [policy, no patch] Drop IE11 support from Drupal 10.0.x has been resolved.

6. Even though we probably want to set some restrictions on how much of the markup and the attributes can be overridden, we probably should add some theme functions / API for adding additional classes and wrappers.

1. +++ b/core/modules/d9_autocomplete/d9_autocomplete.libraries.yml
   @@ -0,0 +1,36 @@
   +  drupalSettings:
   +    autocompleteOptions:
   +      inputClass: ui-autocomplete-input
   +      ulClass:  ui-menu ui-widget ui-widget-content ui-autocomplete ui-front
   +      loadingClass: ui-autocomplete-loading
   +      itemClass: ui-menu-item
   

   As part of #3067523: Add Drupal JavaScript theme functions for password confirm widget I think we set precedent that we prefer not to use drupalSettings for this type of UI settings.

2. +++ b/core/modules/d9_autocomplete/d9_autocomplete.libraries.yml
   @@ -0,0 +1,36 @@
   +      liveRegion: false
   

   Seems like we already force this to be false so we could probably remove this from here.

3. +++ b/core/modules/d9_autocomplete/js/drupalautocomplete.es6.js
   @@ -0,0 +1,866 @@
   +  constructor(input, options = {}) {
   

   It would be helpful to have docs for the constructor parameters.

4. +++ b/core/modules/d9_autocomplete/js/drupalautocomplete.es6.js
   @@ -0,0 +1,866 @@
   +        'When autocomplete results are available use up and down arrows to review and enter to select.  Touch device users, explore by touch or with swipe gestures.',
   

   Two spaces after the first sentence.

5. +++ b/core/modules/d9_autocomplete/js/drupalautocomplete.es6.js
   @@ -0,0 +1,866 @@
   +    // This f
   

   This comment seems incomplete 😇

6. +++ b/core/modules/d9_autocomplete/js/drupalautocomplete.es6.js
   @@ -0,0 +1,866 @@
   +    this.preventCloseOnBlur = true;
   ...
   +      this.preventCloseOnBlur = true;
   ...
   +    this.preventCloseOnBlur = true;
   ...
   +      this.preventCloseOnBlur = true;
   

   Why are we setting this twice 🤔

7. +++ b/core/modules/d9_autocomplete/js/drupalautocomplete.es6.js
   @@ -0,0 +1,866 @@
   +    if (previousItem) {
   ...
   +    } else {
   +      this.input.focus();
   +      this.close();
   +    }
   

   Why are we closing the tray when the input is being focused again? Current behavior is that the tray remains open even when the input gets focused using keyboard arrow navigation.

8. +++ b/core/modules/d9_autocomplete/js/drupalautocomplete.es6.js
   @@ -0,0 +1,866 @@
   +   * Announces to assistive tech when an item is highlighted
   

   Nit: missing a full stop 🧐

Not a complete review, starting with checking that the points in #82 were addressed in #85:

1. The entire drupalSettings block was removed
2. liveRegion removed
3. docblock added to constructor
4. extra space removed
5. incomplete comment removed
6. extra preventCloseOnBlur removed
7. close() removed
8. full stop added

I tested patch #85 in a fresh installation and I faced no issues. None at all I must say.

I have a UX suggestion though: It would be great to automatically add a comma (,) once the user selects a term when using "autocomple tags-style" widget, so that he/she can proceed typing the next term without the need to add the comma.
Just a suggestion, nothing more. This was never the case with previous versions of Drupal anyway.

The patch works and works as expected for the frontend user/editor. As far as I am concerned that part is RTBC by me.

Other things to consider:
There is a composer.lock change in the patch file, I can't understand why.
There are references to IE10-11. I suppose that's needed if we wanna added the new functionality in Drupal 9?
The module name "Drupal 9 Autocomplete", "d9_autocomplete" ties this to Drupal 9, which is not the case. Maybe "drupal_autocomplete" would make more sense in the long run?

Re-rolled.

Updated composer.lock file.
Mistakenly reverted the changes.

Tagging for release manager review since I think we are finally ready for that 🙂Thank you for updating the IS @bnjmnm!

A risk of few sites enabling the module

That is mitigated if you enable it by default in the standard or minimal profiles, so new installs get it. And high-volume adoption would likely raise awareness in 9.x supporting contrib.

What is Module & Lib way?(I think I got it now after reading the below 2 comments) For me, I suggested making it a standard lib and add a module layer. (Both)

About the BC things, I suggested not to do it in the new module (or provide very limited support only, mainly handle the Events). And move current code into a contri-module as BC for old sites if needed.

There's a very long time ago, I suggested Drupal define its own API (Interface) for some (controversial) JS components. Then, we can plugin our own libs and interact with other libs more easily. I can predict still a lot of sites will drop the new lib and add Chosen-alike in the future. (The Core Needs are very different from the real-world)

Except for the development time, the shim sounds like the best option here - we'll know everyone is using the same thing, so it's a lot more predictable.

A risk of few sites enabling the module

That is mitigated if you enable it by default in the standard or minimal profiles, so new installs get it. And high-volume adoption would likely raise awareness in 9.x supporting contrib.

This is tempting, but has the potential to break contrib modules and themes that aren't updated yet (possibly in a non-obvious way for people installing core for the first time).

As far as I'm concerned, the experimental module approach is off the table at this point. We originally went that direction over a year ago when we thought we'd be adopting an autocomplete library that had new features and such. Awesomplete turned out not to be more of a BC solution rather than anything adding new features, so from that point on the module approach wasn't really on the table from a product perspective. For something to be exposed to the site builder as a module, it should include features or enhancements valuable to the site owner that don't exist with it disabled.

So, let's go ahead with the shim approach. As much as possible, we should keep the use of jQuery limited to the shim, and make the shim self-contained so that it can be cleanly deprecated for a clean removal in Drupal 10. We'll want to thoroughly document the primary API, the shim API, and the upgrade path as we go through it.

Thanks everyone!

Retitling to reflect the current scope. Thanks!

There are shims a little all over the place, in autocomplete-init as well as jqueryui.autocomplete.shim.es6.js can we put all in one place?

I would reduce the API we support from jQueryUI do we need all those options this in the ally_autocomplete script?

1. +++ b/core/.eslintrc.json
   @@ -20,7 +20,8 @@
   +    "DrupalAutocomplete": true
   
   +++ b/core/misc/a11y_autocomplete/autocomplete.es6.js
   @@ -0,0 +1,1032 @@
   +    this.count = document.querySelectorAll(
   +      '[data-drupal-autocomplete-input]',
   +    ).length;
   

   So this is a roundabout way to have a counter and have a unique number for this specific instance. Not a fan of indirectly making a counter like that.

2. +++ b/core/misc/a11y_autocomplete/autocomplete.es6.js
   @@ -0,0 +1,1032 @@
   +        'When autocomplete results are available use up and down arrows to review and enter to select. Touch device users, explore by touch or with swipe gestures.',
   

   Need to have that translatable somehow

3. +++ b/core/misc/a11y_autocomplete/autocomplete.es6.js
   @@ -0,0 +1,1032 @@
   +    // If jQuery is available, use it to add events listeners. Otherwise use
   +    // vanilla JavaScript.
   

   We should go with native only. the point is to remove jquery dependency at some point so why not start now if that works?

4. +++ b/core/misc/a11y_autocomplete/autocomplete.es6.js
   @@ -0,0 +1,1032 @@
   +  implementWrapper() {
   

   It's hard to follow that the HTML will look like, might want to have comments explaining the whole structure. And while we don't have to use Drupal.theme having a "theme" function here might be a good thing, makes it easier to override too

5. +++ b/core/misc/a11y_autocomplete/autocomplete.es6.js
   @@ -0,0 +1,1032 @@
   +    // Delay the announcement by 500 milliseconds. This prevents unnecessary
   +    // calls when a user is navigating quickly.
   

   Is that actually helping? a user navigates quickly but doesn't know if hitting keyup/down does anything like this no?

6. +++ b/core/misc/a11y_autocomplete/autocomplete.es6.js
   @@ -0,0 +1,1032 @@
   +      this.cache[inputId] = {};
   

   how is it invalidated?

7. +++ b/core/misc/a11y_autocomplete/autocomplete.es6.js
   @@ -0,0 +1,1032 @@
   +window.DrupalAutocomplete = DrupalAutocomplete;
   

   not a11y_autocomplete?

8. +++ b/core/misc/autocomplete-init.es6.js
   @@ -0,0 +1,341 @@
   +      const pluralMessage =
   +        maxItems <= this.totalSuggestions
   +          ? 'There are at least @count results available. Type additional characters to refine your search.'
   +          : 'There are @count results available.';
   +      return Drupal.formatPlural(
   +        count,
   +        'There is one result available.',
   +        pluralMessage,
   +      );
   

   This is not translatable,
   Need to have on call to formatPlural and a different one to Drupal.t() with the @count parameter

Would welcome a merge request to simplify review.

+++ b/core/misc/autocomplete-init.es6.js
@@ -0,0 +1,341 @@
+    detach(context) {
+      context.querySelectorAll('input.form-autocomplete').forEach((input) => {
+        const id = input.getAttribute('id');
+        Drupal.Autocomplete.instances[id].destroy();
+      });
+    },

Need to check the trigger parameter and check that it's "unload", otherwise this will destroy autocomplete on ajax requests (trigger = "serialize"), like adding a paragraph or adding a media.

proposed resolution is not up to date

Following a meeting with bnjmnm, lauriii, gabor, and myself, todo:

• Need to have a way to use autocomplete with and without the shim at the same time
• Define/document the Drupal API supported by a11y_autocomplete, create the @drupal/a11y_autocomplete package in the project: https://www.drupal.org/project/a11y_autocomplete

Just sharing this link that crossed my Twitter feed that may be helpful: https://adamsilver.io/blog/building-an-accessible-autocomplete-control/

Thanks for your work.

This wasn't working in IE11 so I added the polyfills as dependencies to the a11y_autocomplete library

Still digging into this MR, but wanted to mention something that stood out to me:
in the removed autocomplete.js there was a property: firstCharacterBlacklist
which is set by data-autocomplete-first-character-blacklist

In the new version, that property is changed to firstCharacterDenylist and set by data-autocomplete-first-character-denylist

in core/modules/link/src/Plugin/Field/FieldWidget/LinkWidget.php should the old data attribute be changed to the new one?
blacklist --> denylist

Or should the shim convert the old property name to the new one?

Make the shim be the default, but leave jQuery UI autocomplete in for the rest of 9.x and make it reasonably easy to switch to it if necessary. Warnings could be triggered if there are attempts to use the shim in ways that are unsupported, and they could provide steps on how to use jQuery UI autocomplete instead. Other jQuery UI libraries were deprecated in 8 before being completely removed in 9.

Seems reasonable to me. I certainly don't think supporting all the possible uses of $widget is viable, and adding a contrib module to revert to jQuery IU seems like it would limit contribs adoption of this replacement.

Fantastic work here btw!! It's really exciting :)

added #3217881: Reduce public API surface to the patch, and updated the core implementation accordingly, seems to be working as expected.

pushed a pretty significant update, tried to decouple things as much as possible between the vanilla script and the jquery shim, cleaned up the library definition too.

once this is more or less good I think we need to split this issue up:

1. Add autocomplete and polyfill libs (no integration yet)
2. Add the shim library and and tests
3. Finally make the actual change to the drupal.autocomplete library declaration and relevant tests, deprecation notices, etc.

We're missing styling for the vanilla autocomplete I think, right now the dropdown is transparent :p

Much better, nightwatch test works now. jquery ui test stuff failed because I changed the library definition back to what it was.

we might need to split this issue a bit once it's closer to green.

1. add the autocomplete lib
2. add the new autocomplete init
3. Add the shim layer & tests
4. Replace all uses of jquery-era autocomplete and trigger deprecations

what do you think?

We need to specifically document what areas of the API get BC and which do not, this should be done in the draft change record.

I'd like to try this out myself. Say, using a taxonomy reference with a large vocabulary. I can only see LinkWidget in the diff, but the video seems to show an entity reference. What's involved in getting this running? Say with a term reference field and a large vocabulary.

I found the video in #139 hard to follow.

Re. #140:

Firefox is notably absent from the demos.

The demos sounded remarkably verbose to me, with much redundant information. I think some of the custom announcements may not be necessary. It's worth doing a detailed comparison.

It also sounds like you've tried to emulate VoiceOver's optional training hints. I fear those may annoy a lot of users; VoiceOver's training hints can be turned off, and other screen readers don't have anything so verbose.

The Core autocomplete walkthrough ends with an example of using OSX voice control to operate the autocomplete.

This was very brief.

The choice was made using the show-numbers tool. Is it possible to make a choice without resorting to that?

Has this been tested with any other speech control apart from Apple's?

There are lots of mentions of "the shim", but the issue summary doesn't explain it well. Is that explained in another issue?

Awesomplete was literally the only library without significant accessibility regressions compared to jQueryUI and one that could be added with the least disruption so that was the library we chose.

It was then observed that the amount of customization required for the Awesomplete implementation was significant enough that we were using very little of the library itself, and it may be simpler to make the solution entirely custom.

My recollection is different. The main problem with Awesomeplete was that in some situations it would misreport the control's value to assistive tech, which was a critical blocker.

Discussed with @catch, @nod_ and @bnjmnm the approach for the BC layer on this issue. The overall approach seemed like it was going to the right direction. However, there were few changes that we thought would be useful to make:

1. Remove data-drupal-10-autocomplete since it isn't needed anymore now that markup BC layer has been moved to Stable
2. Ensure that the shim only triggers deprecation warnings when the shim is not used meaning that the initialization of the shim shouldn't trigger deprecation warnings.
3. Make the shim more specifically target the drupal.autocomplete use case to avoid breaking use cases where jQuery UI autocomplete is initialized directly.

Thanks for the thorough review and background info on the implementation Ben.

@andrew thanks for taking the time to take a look. I'm working on the new standalone Autocomplete implementation so I'm following this very closely. I'm having a hard time understanding the specific actions you're asking for though.

• Do you mean Firefox testing is required to pass the accessibility core gate or is it something you would have liked to have?
• Putting my core committer hat on, I'd like to have a list of required/supported software/hardware setup that explain what we need to test for to pass the accessibility core gate. This should be it's own spin off issue though.
• from what Ben said the verbosity is to mimic the implementation which went through extensive user testing. If it is too verbose, what do you think should be the exact level of verbosity? Right now m happy to tweak the implementation but at my level it's not actionable feedback on this topic and I'm not sure of what the detailed comparison would entail
• the numbers tool I don't know what it's referring to so I'll defer to Ben on that. I'm not sure of what you are asking though, is this a core gate requirements?
• for speech control I wasn't aware we were even testing for that in core so guidance on the specific list of software to test for would be welcome

It seems the level of accessibility review differs depending on the issue, the core gate doc are pretty vague too so it might be worth revising them to add some more details.

The last test failure from the dontbreakstuff branch happens when the form-autocomplete class is added. even before the shim is changed to make use of the class to know if it needs to call the old autocomplete or our version. Not sure what's happening.

The commit in question: https://git.drupalcode.org/project/drupal/-/merge_requests/1356/diffs?co.... the previous commit passed the tests.

yeah not planning on removing this feature. Just trying to see if there are other problems.

now the question is how much of EntityReferenceAutocompleteWidgetTest::testAutocompleteOptions do we want to keep knowing the lib is pretty well tested already. We should have some but maybe not as much if some of them can be included upstream (like for the ignored char search trigger thing?

Removed some test cases for options that are not supported by the library anymore (sort/displaylabel) should be green, we can see what else we want to test for in the core patch vs. the library itself.

All right finally back to green, thanks for your patience.

Added a few things to make integration easier with things like link it, fixed a few edge cases (avoid double initialization of Autocomplete, introduced a few helpers to make the shim code less verbose/brittle.

On the test coverage there are 2 night watch tests that don't test what they are supposed to otherwise they are good.

As far as feature go I'd like to add back the classes options because the shim and modules like link it rely heavily on it.

Now it's 10.0/9.4 material, patches are hidden because work happens in merge requests

Umm so there is some issue with the latest merge from 9.3.x haven't had time to track it down and I'm off tomorrow. Will pick it up on Friday if nobody beats me to it.

I think we might need to add some more features to the BC layer, namely supporting wiget's _search and _response as boostrap theme is using them: https://git.drupalcode.org/project/bootstrap/-/blob/8.x-4.x/js/misc/auto... and with 65k+

With latest patch + the merge request to add the _search and _response overrides bootstap theme works as expected.

I think the last commit did it \o/

I propose autocomplete library, free from Jquery.
1) Good with accessibility
2) light
3) working with most type of data - static file, axios, jquery ajax, Promises fetching data.
4) A lot of hooks and options. could be rewrited.

https://tomik23.github.io/autocomplete/#started
https://github.com/tomik23/autocomplete

I'm starting to review this MR. One thing I'm tripped up on at the moment is that the MR has lines like the following...