Support type-specific boosting with a processor to configure function_score queries (8.0.x and 9.0.x)

@andyg5000, thanks!

It looks like we're getting a test failure in the \Drupal\Tests\elasticsearch_connector\Unit\SearchAPI\Query\QueryParamBuilderTest::testBuildQueryParams() test.

Specifically the mock query object isn't expecting the new call to getOption('elasticsearch_connector_type_boost_functions') added in the patch. You'll probably have to add something like...

@@ -74,6 +74,8 @@
       ->willReturn($fields);
 
     $query = $this->prophesize(QueryInterface::class);
+    $query->getOption('elasticsearch_connector_type_boost_functions')
+      ->willReturn([]);
     $query->getOption('offset')
       ->willReturn(0);
     $query->getOption('limit')

... to fix that test.

QueryParamBuilderTest::testBuildQueryParams() is only supposed to test a basic flow through QueryParamBuilder::buildQueryParams(), so you'll also need to add a separate test where getOption('elasticsearch_connector_type_boost_functions') returns something, and then checking that the query is modified in the way that you expect. I recommend testing your specific use-case in this separate test, so that you can be confident that future changes to the module won't cause your use case to regress!

Finally getting back to this issue after a while; sorry for the delay, and thank you in advance for your patience with me.

I haven't had a chance to rebase, re-review the code, or fix the test... just updating the issue summary with all the details so that I don't have to hop between this issue and #2980257: Type-specific boosting is being ignored completely with elasticsearch_connector; and also for documentation purposes (i.e.: for folks who stumble across this issue in the release notes / future). Thanks for understanding!

We are attempting to use the "Type-specific boosting" feature of Search API in combination with elasticsearch_connector, and it turns out that this is not working at all. I'm not even sure if it was ever implemented in the code.

Some notes on what happened here...

Search API Processor plugins can run at three different stages: Preprocess index (i.e.: when content is being indexed, i.e.: copied to Elasticsearch), Preprocess query (before a search query is sent to the server), and Postprocess query (after a search query response is received, before displaying results).

The Type-specific boosting processor (\Drupal\search_api\Plugin\search_api\processor\TypeBoost, machine name type_boost) is built into Search API (i.e.: upstream from Elasticsearch Connector) but - at time-of-writing - it is still hard-coded to run at Preprocess index time.

The problem is that, beginning in Elasticsearch 5.0.0 (released in 2016), boosting at index-time was deprecated; but it still worked on indexes created before Elasticsearch 5.0.0 until at least Elasticsearch 7.11, but it was completely removed in Elasticsearch 8.0.0.

TL;DR this means that Search API's Type-specific boosting plugin stopped working for new sites around Elasticsearch 5.0.0 / 2016; but may have continued to work for existing sites depending on how they managed their indexes.

@sokru, @siliconvalet, @sonnykt, @tvosenek, @a.dmitriiev, @ebeyrent, and myself removed some other vestiges of index-time boosting in #3183164: Index-time boost is deprecated / merge request !1.

I was wondering if it was possible to hide Search API's upstream TypeBoost (type_boost) processor for Indexes with a Server with an Elasticsearch backend. It is possible to hide it (by subscribing to \Drupal\search_api\Event\GatheringPluginInfoEvent), but not for a specific Index/Server/Backend. Likewise, it's possible to alter the class associated with that plugin definition, and/or change the stages associated with the plugin definition... but again, not for a specific Index/Server/Backend.

(I don't want to assume that site builders only have one Search API Server backend enabled on their site at one time - perhaps they're evaluating multiple search backends to see which fits their use-case, or perhaps they're transitioning to/from Elasticsearch)

Anyway, all this is to say that I'm satisfied with the solution in this patch, i.e.: of providing a clearly-marked, Elasticsearch-specific processor plugin for this plugin.

The test failure we're seeing in Drupal\Tests\elasticsearch_connector\Unit\SearchAPI\Query\QueryParamBuilderTest::testBuildQueryParams() has to do with test doubles (a technique used for writing unit tests). I wrote a longer explanation of test doubles in #3542571-10: Add OriginalIndexId to IndexParamsEvent earlier today (see that comment for more information).

Similar to how we fixed the test in 3542571, we'll have to update the mock QueryInterface to expect a call to getOption('elasticsearch_connector_type_boost_functions', [])).

But, unlike 3542571, there is no other test coverage for function_score queries, or (elasticsearch_)?type_boost plugins, so we'll still need to write a test for the functionality added in this patch before I can merge it (tests will ensure that the functionality your site depends on won't break in the future as new patches are accepted).

Okay, I've added tests, and this is ready for community review: @andyg5000, would you be able to re-test the patch to ensure it still works? If so, please change the issue status to "Reviewed & tested by the community"

Some steps for manual testing...

1. Set up a local 8.0.x environment
2. Log in as an administrator.
3. Go to /test-elasticsearch-index-search. Under Fulltext search = foo, then click the Apply button. Look at the 0 ID column: the rows should be in the order 4, 2, 5, 1. Note also the 5 Category column: the rows should be in the order article_category, item_category, article_category, item_category, i.e.: reflecting roughly equal boosts to every content type. Note also the numbers in the 1 Relevance column; they should be pretty close to each-other.
4. Go to /admin/config/search/search-api/index/test_elasticsearch_index/processors. Under Enabled, check Type-specific boosting (Elasticsearch). Scroll down to Processor settings, and open the Type-specific boosting (Elasticsearch) vertical tab. Set Default boost for items from this datasource = 1.00. Set Boost for the Entity Test Bundle bundle to 1.00. Set Boost for the item bundle = 21.00. Set Boost for the article bundle = 1.00. Click the Save button.
5. Go to /test-elasticsearch-index-search. Under Fulltext search = foo, then click the Apply button. Look at the 0 ID column: the rows should be in the order 2, 1, 4, 5. Note also the 5 Category column: the rows should be in the order item_category, item_category, article_category, article_category, suggesting the 21.00 boost to the item content type is giving items a bigger boost. Note also the numbers in the 1 Relevance column; the two items should have much larger relevance than the articles.

I will create a merge request for the 9.0.x branch shortly.

(updating issue summary)

Created a 9.0.x merge request. Added a tag so I remember to merge the 8.0.x branch too.

I haven't heard back from anyone in 7 days, but I want to include this change in the next release, so I'm going to merge this without community review.

Merged to 9.0.x; merging to 8.0.x next.

Merged to 8.0.x. I'll update this issue when these code changes are released.

The changes in this issue were released in elasticsearch_connector-9.0.0-alpha3, and elasticsearch_connector-8.0.0-alpha7