Search API Autocomplete support

Autocomplete could be considered the child of the suggestor issue.

At least personally I never tried out search_api_autocomplete with the solr backend due to the requirements on my project.
Someone should at least give a try and adapt the solr backend with the changes we made in the search_api_DB backend. I think this should be not too much of an effort.

i'm alter search_api_solr plugin and implement getAutocompleteSuggestions

http://pastebin.com/YjMe0Mw9

but i have warning, because current getAutocompleteSuggestions method use argument type without right namespace

Nice, I jsut started debugging this and fixed the namespace issues. I took your code and put it into a patch for the SearchApiSolrBackend and it does not look that bad, thx for your work! Here the patch.

The latest patch looks good. But at least it requires some tests.

But spellchecking / suggestions will be revised in general, because we don't leverage the power of this feature correctly. Nevertheless we can commit this patch as intermediate solution if some test coverage is added.

+++ b/src/Plugin/search_api/backend/SearchApiSolrBackend.php
@@ -24,6 +24,8 @@ use Drupal\search_api\Query\QueryInterface;
+use Drupal\search_api_autocomplete\Entity\SearchApiAutocompleteSearch;
+use Drupal\search_api_autocomplete\Suggestion;

In my first review I missed that search_api_autocomplete is not a included in search_api but a separate module.
We can't add such a dependency.
We need to discuss how to integrate with that module.

In my first review I missed that search_api_autocomplete is not a included in search_api but a separate module.
We can't add such a dependency.
We need to discuss how to integrate with that module.

namespaces not for dependencies, namespaces for right argument type in methods
without installed search_api_autocomplete all must work ok
without right namespaces mathod can't be called, because generate exception about wrong argument type

Instead of adding a dependency into Search API Solr it would be best (IMHO) to use a bridge-module to integrate Search API Autocomplete.
The attached patch adds this bridge.

I talked to Thomas to get some background information before starting further discussions here.

mkalkbrenner: hi, can you tell me something about the current state of search_api_autocomplete?

drunken_monkey: dawehner has ported it, I have only given it a cursory glance yet. As with Solr and all other extension modules, I currently focus entirely on Search API itself. From what I gather, it has several larger problems, but generally works.

mkalkbrenner: thanks for the infos. Are there open issues within search_api_db regarding autocomplete?

drunken_monkey: I don't think so, no.

mkalkbrenner: So the assumption is that search_api_db could be used as blueprint for the Solr based autocomplete implementation and remaining problems can be solved within the search_api_autocomplete module?

drunken_monkey: Yes, more or less. But you can also just use Solr's D7 implementation of autocompletion. It hasn't really changed (yet).

With the combined effort of many people, the Search API Autocomplete recently got into a working state: #2700691: [search_api_autocomplete] Search API Autocomplete.

I recently tried the patch in #10, but couldn't quite make it work... Has somebody successfully made Search API, Search API Autocomplete and Search API Solr Search work together?

@ressa: Thanks for pointing me to #2700691: [search_api_autocomplete] Search API Autocomplete. I wasn't aware of this issue. And to be honest I think it's a bad practice to have these detailed information discussed there instead of within a meta issue in the search_api issue queue.

But it seems to make sense now to start working on the Solr part :-)

I think I won't accept the approach proposed in #10 because it will complicate the autocomplete support of search_api_multilingual.

I think I won't accept the approach proposed in #10 because it will complicate the autocomplete support of search_api_multilingual.

I agree with markus here, yes, its sad that its coupled with the class, but php totally allows us to do that, so we should not cripple ourselves. Searchapi has simply that design.

@ressa: Thanks for pointing me to #2700691: [search_api_autocomplete] Search API Autocomplete. I wasn't aware of this issue. And to be honest I think it's a bad practice to have these detailed information discussed there instead of within a meta issue in the search_api issue queue.

I also cannot agree more :)

I agree that having a "Drupal 8 Contrib Porting Tracker" issue for a module separated from the modules' own issue queue is a problem for those looking for info on the status of a module. Perhaps it could be worth considering moving them into the modules' own issue queue? Many modules already have a "D8 version of this module?" issue, so it is the natural place to look for info on that, and track any progress.

I started by reviewing the existing implementation that doesn't work anymore. It's a strange implementation that uses search queries in combinations with facets.

I asked myself, why we didn't use one of the built-in features. Candidates are:

• spellchecker, like @Hydra proposed in #5
• suggester, the best and most powerful solution but difficult to handle
• terms, the easiest and fastest solution that is fully compatible to the DB backend for the suggest_suffix part of autocomplete

From my point of view, suggest_suffix should be solved using terms and suggest_words using suggester.
But then I realized and remembered that all of these advanced cool Solr features work per Index aka Core. And Search API has this concept of creating multiple virtual indexes in one Solr core. That's simply incompatible and caused the weird implementation we found in the code :-(

After a short discussion I decided to not skip these advanced features. The opposite is true and I plan to leverage them more and more to really make a difference regarding features, performance and scaleability compared to the DB backend.

Therefore we need to avoid that multiple Search API indexes are stored in one Solr Core.
I added a first patch that prevents the user from assigning multiple indexes to one core via the user interface and brings dramatically easier implementation of suggest_suffix.
It works on Solr 6 having comparable results to the DB backend. But Solr 5 and 4 have at least "some" suggestions which might lead to "no results" unless we extend their schema as well.
So this first patch is just a start and it is still work in progress. But it would be great to get some feedback already.

Great start for a first patch, it works - and even with Search API Autocomplete :-)

For others who want to test it, here is a quick way of getting it up and running on a fresh install:

Download Search API Solr Search and required modules

composer config repositories.drupal composer https://packages.drupal.org/8
composer require "drupal/search_api_solr 1.x-dev"

Patch Search API Solr Search and enable module

cd modules/search_api_solr/
wget https://www.drupal.org/files/issues/2790249_terms.patch
patch -p1 < 2790249_terms.patch 
drush en search_api_solr_defaults -y

Update schema.xml and restart

sudo cp solr-conf/6.x/* /var/solr/data/NAME_OF_YOUR_SOLR_CORE/conf/
sudo ls /var/solr/data/NAME_OF_YOUR_SOLR_CORE/conf/ -l
sudo service solr restart
sudo service solr status

Search API Solr fulltext search View available at http://example.com/solr-search/content

Optional: Add Search API Autocomplete

drush dl search_api_autocomplete;
drush en -y search_api_autocomplete;

Flush caches and activate, at admin/config/search/search-api/index/default_solr_index/autocomplete

Like I wrote, the Search API Autocomplete Search works, the node number count seems to be off, though.

• Drupal 8.2.3
• Search API 8.x-1.0-beta3
• Search API Autocomplete 8.x-1.0-alpha1
• Search API Solr Search 8.x-1.x-dev, patched

NOTE: I first tried downloading the modules with drush, but I got a Solarium error and WSOD:

drush dl search_api;
drush dl search_api_solr --dev;
cd modules/search_api_solr/
wget https://www.drupal.org/files/issues/2790249_terms.patch
patch -p1 < 2790249_terms.patch 
composer install
drush en search_api_solr_defaults -y

After running composer install, only two Composer packages are installed:

Package operations: 2 installs, 0 updates, 0 removals
  - Installing symfony/event-dispatcher (v3.1.7)
    Loading from cache

  - Installing solarium/solarium (3.7.0)
    Loading from cache

The composer require "drupal/search_api_solr 1.x-dev" command, which give you a working installation, updates a lot of packages:

Package operations: 30 installs, 0 updates, 0 removals
  - Installing symfony/dom-crawler (v2.7.6)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/css-selector (v2.8.4)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/browser-kit (v2.7.6)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing fabpot/goutte (v3.1.2)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing behat/mink (v1.7.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing behat/mink-browserkit-driver (v1.3.2)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing behat/mink-goutte-driver (v1.2.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing mikey179/vfsstream (v1.6.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing sebastian/version (1.0.6)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing sebastian/recursion-context (1.0.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing sebastian/global-state (1.0.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing sebastian/exporter (1.2.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing sebastian/environment (1.3.2)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing sebastian/diff (1.3.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing sebastian/comparator (1.2.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing phpunit/php-text-template (1.2.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing doctrine/instantiator (1.0.5)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing phpunit/phpunit-mock-objects (2.3.8)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing phpunit/php-timer (1.0.7)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing phpunit/php-token-stream (1.4.8)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing phpunit/php-file-iterator (1.4.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing phpunit/php-code-coverage (2.2.4)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing phpdocumentor/reflection-docblock (2.0.4)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing phpspec/prophecy (v1.5.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing solarium/solarium (3.7.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing drupal/search_api (1.0.0-beta3)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing drupal/search_api_solr (dev-1.x 4a56b0d)
    Cloning 4a56b0dc94619266d7812151a3ec9773aa8b907b

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing jcalderonzumba/gastonjs (v1.0.2)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing jcalderonzumba/mink-phantomjs-driver (v0.3.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing phpunit/phpunit (4.8.27)
    Loading from cache

Now I added spell suggestions to the autocomplete box, but just for Solr 6.x.
And I improved the check for multiple enabled indexes on the same "server".

Works beautifully, I now get suggestions on miss-spelled words - thanks!

Quick way of getting it all up and running, for testing:

drush dl drupal --drupal-project-rename=solr.local --verbose;
cd solr.local;
drush si standard --db-url='mysql://root:root@localhost/solr_local' --account-pass=admin --site-name='solr.local' -y;
drush pmu update search -y;
drush dl admin_toolbar -y && drush en admin_toolbar_tools -y;

composer config repositories.drupal composer https://packages.drupal.org/8;
composer require drupal/search_api;
composer require "drupal/search_api_solr 1.x-dev";

cd modules/search_api_solr/;
wget https://www.drupal.org/files/issues/2790249_18.patch;
patch -p1 < 2790249_18.patch;

# exchange d8 with the name of your Solr core
sudo cp solr-conf/6.x/* /var/solr/data/d8/conf/;
sudo service solr restart;

drush en -y search_api_solr search_api_solr_defaults;

drush dl devel;
drush en -y devel_generate;
drush generate-content 20;

drush dl search_api_autocomplete;
drush en -y search_api_autocomplete;
drush cache-rebuild;

This patch is now the candidate for a final review. Changes compared to #18:

• added some test coverage for the autocomplete feature
• fixed config schema
• removed several doublets from the suggestions
• improved support of old Solr configs (and versions)
• respect the individula autocomplete settings regarding search field overwrites

Great work, the patch works fine - thanks!

The issue I had with the number count was because I was re-using a Solr index, which had old records in it. Once I made a new core, the suggestion count was perfect.

I have tested the patch in #20 with the current -dev and Solr 6.x. The autocomplete is working for me, but there is a bit of unexpected behavior. This could be in my setup, but for example: a title field with value: "Breaking Bad" will either autocomplete "breaking" or "bad" but not the field value as a whole.

I can confirm this behavior:

The autocomplete is working for me, but there is a bit of unexpected behavior. This could be in my setup, but for example: a title field with value: "Breaking Bad" will either autocomplete "breaking" or "bad" but not the field value as a whole.

The DB backend doesn't suggest that in a reliable way, too. The DB backend just randomly picks words. So we're kind of compatible here.

Nevertheless Solr perfectly supports this kind of "phrase" auto completion using it's suggester component. Unfortunately there's no way to configure that component in a common way. A dedicated suggester needs to be configured per field in solrconfig.xml. But due to the fact that the number of fields is increased dynamically by Search API it's impossible to configure suggesters upfront. On the other hand, defining one especially for your use-case is simple. But it will require a small piece of custom code to get the data from it.
And suggesters need to be maintained. That means that you need a cron job to periodically "build" them. They are not real-time.

Search API Solr Multilingual contains a generator for the solrconfig.xml. We could use this one to create the required configuration every time you configured a search page that includes autocomplete. But that is out of scope for this issue here and requires a new feature request.

BTW I'll rename "suggest_words" to "suggest_corrections" and reserve "suggest_words" for the potentially upcoming new feature.

I slightly modified the patches again and committed them since the tests are passing on Solr 4,5 and 6.
But I opened a follow up: #2831842: Prepare usage of Solr's suggester component for autocomplete suggestions and re-enable multiple "indexes" per Solr Core later

I have just tested the latest dev-release of the module, and it's so nice to have Search API, Search API Autocomplete and Search API Solr Search (DEV) work together out-of-the-box - thank you @mkalkbrenner!

@ressa can you outline which versions of the modules you have that are working out of the box?

Sure, these are the modules:

• Search API - version: '8.x-1.0-beta3'
• Search API autocomplete - version: '8.x-1.0-alpha1'
• Search API Solr search - version: '8.x-1.0-beta1+24-dev'

EDIT: You can also just take the commands from #2790249-19: Search API Autocomplete support, leaving out the two lines of patching. That should get you up and running.