Change autocomplete to use updated JQuery and smaller file

On api.drupal.org, the JSON file (currently) makes sense, to avoid computing the options every time. On your site, it makes more sense to compute it on the fly probably and return just the first 10-20 options.

So I agree -- an option would be good. We could have an option in the API admin page that chooses one method or the other, and maybe a third option that would disable it completely?

Works for me. Obviously, making the single-file the default (for backwards compatibility) would be desirable. :)

I do want to properly solve this so we can handle more modules on API.drupal.org, ideally without the complexity of extra options.

What about splitting up the file by starting letter, and loading as soon as that is typed? To keep startup fast, we could have another file with the first N suggestions that is loaded on focus like today.

Or, maybe searching all projects doesn't make sense. If we expect people will be more interested in a single project, and make them easy to get to, it could work.

psynaptic's site has all the core and contrib code in one branch/project, so breaking it up by project will not help there. Regarding search in multiple projects, we still need to figure out what the plan is for
#990108: Links and search do not make sense and are not working across projects

I don't think that making multiple files is a very good idea either. There's no particular reason why splitting the file up into 26 or so pieces will necessarily make it small enough either. Autocomplete in Drupal normally works on demand (type a few letters and it figures out what the possibilities are), so why would it be so terrible to do that here too? And why are configuration options so bad?

If we have two different implementations, that is twice as much to maintain, one could easily go unmaintained. In general, we shouldn't need to be asking users about implementation details, even if API module admins are likely to know the internals. Ideally, the module just works well.

I do want api.drupal.org to be fast and get out of the way of developers. A static file is very quick, and with the whole database, it is as fast as the browser. But, that can't scale well. We have decent servers, so Drupal core's autocomplete won't be slow, but you can't get around network latency.

Another possibility is to keep the first N suggestions in a file like we have now, and do HTTP requests to get more-specific information. The first 1000 or so suggestions can happen immediately, while we ask the server for more in the background.

Good points...

It seems like autocompletes on drupal.org itself (such as if you have to change the project for an issue) are fast enough to deal with. So how about if we just scrap using the file entirely, and the obsoleted jquery-autocomplete plugin that we distribute with the API module, and use the Drupal Core functionality?

There's not much point it having a ton of custom code around in the API module that we have to maintain, if it's there to solve something that isn't even a problem now.

hm, what kind of errors did the core method throw?

Well, hm. Any ideas then? Can we live with the javascript HTTP 0 errors?

My understanding is that those happen when a page load interrupts an AJAX call. I believe if you type 'l' or 't' and press return faster than the server responds, you get the annoying error. Even if that is worked around, HTTP requests are still slow.

I think the best autocomplete widget to use would be http://jqueryui.com/demos/autocomplete. I don't know the full history, but I think it might be the updated, official version of the plugin currently included in API module. Since it is in JQuery UI, it should be well supported and likely to work well with Drupal.

From the documentation there, "You can pull data in from a local and/or a remote source." We can go with both. The local data source would be pulling the 100 or 1000 shortest identifiers from a static file, like API module already does. Then a remote data source can kick in to get more specific suggestions. The idea is that many searches will be satisfied by the first set of suggestions and be really fast. If the network connection and server are doing well, they fill in the specific suggestions before the user types enough to run out of initial suggestions.

How would that work? If I started typing, I would only get something from the list of 100 shortest? And then if I kept typing, when would it switch to being a dynamic database query? That doesn't sound very good or workable. For instance if we said 3 characters typed and the three they typed happened not to appear in the 100 shortest file, they would have nothing showing. And as things are now, as you type you generally have a mix of shorter and longer results, after just about 2 characters, so what really is the savings in starting with a file? How often do people look for t() or l() -- everything else is longer. So I don't see a really compelling reason to complicate things by using both a file and a dynamic query...

Another thing to consider... Using Jquery UI plugin in D6 would make things much more complicated. jQuery in Drupal 6 is not a high enough version to support JQuery UI, so we would probably need to require http://drupal.org/project/jquery_update -- and then we'd probably also need to require people to download the jQuery UI plugin separately (it's in enough use that distributing with the API module would not be smart IMO). Which is all quite lame.

"For instance if we said 3 characters typed and the three they typed happened not to appear in the 100 shortest file, they would have nothing showing." The dynamic results would show up as soon as the HTTP request completes.

Ah, that makes sense. Sounds like a good approach, if we can get around the deployment complications mentioned in the last paragraph of #13.

Since I'm probably doing the Drupal 7 port shortly, and this will be vastly simpler to deploy in D7, let's postpone this until the D7 version.

Just so summarize... the plan here is:

1. We will switch to using the current version of the JQuery Autocomplete plugin, instead of the old version we're currently using... hmmm... After reading through this issue where D8 core is considering switching to this plugin:
#675446: Use jQuery UI Autocomplete
I am not so sure. It looks like the current JQuery Autocomplete plugin
a) requires a higher version of JQuery than is in Core D7
b) is not accessible, whereas the current D7 Drupal Core auto-complete core functionality is.

So I think we might want to just switch to the core Drupal auto-complete and live with the consequences?

2. If we can make the Drupal auto-complete work this way, ideally we would want the autocomplete to use a static file with, say, the 1000 shortest items in the branch, and then an AJAX request to get longer items.

Update/clarification:

Accessibility - based on the latest comments on #675446: Use jQuery UI Autocomplete:
- The JQuery UI Autocomplete plugin from JQuery UI version 1.9 should be acceptably accessible. It is still in the early development stages though.
- JQuery UI version 1.8's autocomplete widget is not sufficiently accessible to pass muster of the Drupal accessibility team.
- The Drupal Core 7.x autocomplete widget is accessible.
- I doubt that the old autocomplete widget we currently have in the API module is accessible.

Functionality:
- The autocomplete widget we are using now in the API module only supports having a file for input (I think). I personally experience trouble on api.drupal.org sometimes where the autocomplete doesn't work (I filed an issue about it long ago and it was not reproducible, and no one could figure out why -- I often have trouble where I type in a search term and hit enter, and the search doesn't happen, I just get back to the landing page -- this might or might not be related to the autocomplete widget though).
- Drupal Core 7.x autocomplete only supports using a Drupal path to return the autocomplete options for a given string input. As pointed out above, this leads to HTTP errors when the user types ahead of the autocomplete AJAX return.
- The JQuery UI widget supposedly supports both, which might alleviate the HTTP errors.

So... any thoughts about what we should do? My inclination is still to use the core autocomplete functionality and ditch the files.

If we go with core, I think people will type fast and press enter before the server has a chance to respond, triggering the HTTP 0 error. Can we fix or work around that in core?

I talked to effulgentsia in IRC, and it sounds like it's this issue we're talking about:
#1232416: Drupal alerts "An AJAX HTTP request terminated abnormally" during normal site operation, confusing site visitors/editors
A full fix doesn't look like it's coming anytime soon, but nod_ suggests using the patch in comment #85 to fix it in the meantime... It looks like that patch is for D8, but we could probably make a D7 version if it won't apply (with -p1). The patch looks fairly straightforward -- it just logs the error to the JS console instead of putting it in an alert, which should be fine.

Another related issue is
#1060776: Queue AJAX requests rather than letting the browser queue them: Fix HTTP Error 0
This one doesn't have a fix at the moment, but looks like it's probably the root of the autocomplete problem.

So... should we go forward?

Fixing this issue somehow is another pre-requisite to getting all of contrib on api.drupal.org (the existing auto-complete is not workable with that much data).

I just committed a change that uses Drupal Core autocomplete instead of all that custom code. Reduced the code base considerably... The functionality is slightly different from what it was previously, but the advantage of not having to maintain all that custom code, plus it will work for contrib...

What about speed difference when using Drupal Core autocomplete widget. I suspect it's clearly slower than using JSON cached file. Do you plan to work out autocomplete return speed issue? Probably some database caching (APC, memcached, etc.) should be evaluated. Would love to hear opinion on this one.

We can't use a file and expect it to scale, so I don't think that's an option. If you have an idea for how caching could work, let me know, but I don't have any ideas. Drupal Core uses these types of autocompletes all the time... what's the problem?

I mean that Drupal Core autocomplete is slow because it makes a AJAX request and returns JSON object. This is significantly slower than parsing cached JSON file. Now talking about user experience - user has to wait autocomplete result for .5 second and every new character typed makes a new AJAX request.

Ideally, would be nice to have autocomplete like Google has - elegant, speedy and intuitive. Boost module can work with JSON caching, but again even when caching JSON, Drupal Core autocomplete repopulates autocomplete popup on every AJAX request. When I think it should be visible all the time and just update results of autocomplete list.

These expressed ideas about Boost and etc. are clearly out of scope of this issue, but just wanted to say my opinion and drawbacks of this solution you just commited.

Yes, we were aware of some drawbacks... but there really wasn't much choice except trying to do something that would be very complicated and hard to maintain.