Compatibility with elasticsearchconnector 8.x-7.x

I confirm that patch is needed for elasticsearch_connector version 7.x. It doesn't use the 'type' as an array key.

I think there is another issue. The SearchApiDataType (text_ngram) is showing as none supported. Seems like the default = true flag needs to be added to elasticsearch_connector_autocomp/src/Plugin/search_api/data_type/TextNgramDataType.php like it's done in the elasticsearch_connector module.

New patch attached with the change I was talking about.

Hi,
I tried the last patch which solved the "not supported" issue but introduced a new one in combination with search_api_autocomplete. The search_api_autocomplete module queries the current search for fulltext fields to apply the autocomplete onto them. To retrieve the fulltext fields it uses the \Drupal\search_api\Entity\Index::getFulltextFields method which in turn uses the \Drupal\search_api\Utility\DataTypeHelper::isTextType function to determine if the current field is a text field.

You can see in the below code, that a field is only a text type if its type is "text" or it is not a default type and its fallback type is "text".

https://git.drupalcode.org/project/search_api/-/blob/8.x-1.x/src/Utility...

public function isTextType($type, array $textTypes = ['text']) {
  if (in_array($type, $textTypes)) {
    return TRUE;
  }
  $dataType = $this->dataTypeManager->createInstance($type);

  if ($dataType && !$dataType->isDefault()) {
    return in_array($dataType->getFallbackType(), $textTypes);
  }
  return FALSE;
}

I think the reason for that behavior lies in the definition of the "default" property as stated in the comment here:

https://git.drupalcode.org/project/search_api/-/blob/8.x-1.x/src/Utility...

// We know for sure that we do not need to fall back for the default
// data types as they are always present and are required to be
// supported by all backends.
if (!$dataType->isDefault() && (!$server || !$server->supportsDataType($typeId))) {
  $this->dataTypeFallbackMapping[$indexId][$typeId] = $dataType->getFallbackType();
}

I found another hint for how this could work in the search_api field config which says:

The data types which can be used for indexing fields in this index. Whether a type is supported depends on the backend of the index's server. If a type is not supported, the fallback type that will be used instead is shown, too.

So I think in this case the field type is special to the Elasticsearch engine and should therefore not be generally supported (as the default flag would indicate). Instead the server itself should declare its support for this type. However as far as I understand this is done inside the \Drupal\elasticsearch_connector\Plugin\search_api\backend\SearchApiElasticsearchBackend::supportsDataType function which checks against a static list of types (only object at the time of writing) without any possibilities to alter this.

I know that this is not an issue with this module, but I wanted to inform you that I think setting the default flag to true would not be the correct solution. I will possibly add an issue to the elasticsearch_connector module to see if this can be solved.

Changing this to needs work. We need to decide on a viable solution and implement that.

I'm happy to receive a patch with a proposed resolution, any solution is better than none (generally)

Fixed it, just changing it to false (without quotes I might add) in the annotation is enough to make the ngram field useable again as text fallback from a views fulltext filter or argument.

Tested and works for me :-)

#9 +RTBC - Looks good to me as long as we're not worried about being backward compatible with the older versions of Elasticsearch Connector. The ngram data type does still show as unsupported, but I'm not sure that this was supposed to resolve that.

The patch works for me.
But the status on the data types still show warning.

From https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis... I'm created a new patch to make this, wonderful, module work with Elasticsearch 7.13.x.

The original module really doesn't work because $params['type'] on elasticsearchConnectorPrepareIndexMapping() method (web/modules/contrib/elasticsearch_connector_autocomp/src/EventSubscriber/DefaultSubscriber.php:67).

The patch here helps to managed that issue, but the original module doesn't handle ngram as tokenizer (handle as filter) this has two implications:

1. I have to add {query: {..., "analyzer": "ngram_analyzer"} to queries
2. The highlight return (from ES) doesn't highlight part of word (only entire word)

PS: For who can't recreate the index, ElasticSearch has a query "query_string" that helps with this, eg:

GET /drupal_nodes/_search
{
  "query": {
        "query_string" : {
            "query" : "*app*",
            "fields" : ["*"],
            "analyze_wildcard" : true,
            "allow_leading_wildcard": true
        }
   }
}

this query will find: apple, rappi, whatsapp, etc

Follow the patch:

diff --git a/src/EventSubscriber/DefaultSubscriber.php b/src/EventSubscriber/DefaultSubscriber.php
index 541b299..4723f0f 100644
--- a/src/EventSubscriber/DefaultSubscriber.php
+++ b/src/EventSubscriber/DefaultSubscriber.php
@@ -64,16 +64,16 @@ public function elasticsearchConnectorPrepareIndexMapping(Event $event) {
     if ($ngram_index_analyzer_enabled) {
       foreach ($index->getFields() as $field_id => $field_data) {
         if ($field_data->getType() == 'text_ngram') {
-          $params['body'][$params['type']]['properties'][$field_id]['type'] = 'text';
-          $params['body'][$params['type']]['properties'][$field_id]['boost'] = $field_data->getBoost();
-          $params['body'][$params['type']]['properties'][$field_id]['fields'] = [
+          $params['body']['properties'][$field_id]['type'] = 'text';
+          $params['body']['properties'][$field_id]['boost'] = $field_data->getBoost();
+          $params['body']['properties'][$field_id]['fields'] = [
             "keyword" => [
               "type" => 'keyword',
               'ignore_above' => 256,
             ],
           ];
-          $params['body'][$params['type']]['properties'][$field_id]['analyzer'] = 'ngram_analyzer';
-          $params['body'][$params['type']]['properties'][$field_id]['search_analyzer'] = 'standard';
+          $params['body']['properties'][$field_id]['analyzer'] = 'autocomplete';
+          $params['body']['properties'][$field_id]['search_analyzer'] = 'autocomplete_search';
         }
       }
     }
@@ -82,6 +82,8 @@ public function elasticsearchConnectorPrepareIndexMapping(Event $event) {
   }

   /**
+   * https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenizer.html
+   *
    * Called on elasticsearch_connector.prepare_index event.
    *
    * @param \Symfony\Component\EventDispatcher\Event $event
@@ -99,23 +101,27 @@ public function elasticsearchConnectorPrepareIndex(Event $event) {
 {
 	"settings": {
 		"analysis": {
-			"filter": {
-				"ngram_filter": {
-					"type": "{$settings['ngram_config']['ngram_type']}",
-					"min_gram": {$settings['ngram_config']['min_gram']},
-					"max_gram": {$settings['ngram_config']['max_gram']}
-				}
-			},
 			"analyzer": {
-				"ngram_analyzer": {
-					"type": "custom",
-					"tokenizer": "standard",
-					"filter": [
-						"lowercase",
-						"ngram_filter"
-					]
-				}
-			}
+        "autocomplete": {
+          "tokenizer": "autocomplete",
+          "filter": [
+            "lowercase"
+          ]
+        },
+        "autocomplete_search": {
+          "tokenizer": "lowercase"
+        }
+      },
+      "tokenizer": {
+        "autocomplete": {
+          "type": "{$settings['ngram_config']['ngram_type']}",
+          "min_gram": {$settings['ngram_config']['min_gram']},
+          "max_gram": {$settings['ngram_config']['max_gram']},
+          "token_chars": [
+            "letter"
+          ]
+        }
+      }
 		}
 	}
 }
diff --git a/src/Plugin/search_api/data_type/TextNgramDataType.php b/src/Plugin/search_api/data_type/TextNgramDataType.php
index 311a448..e4305e6 100644
--- a/src/Plugin/search_api/data_type/TextNgramDataType.php
+++ b/src/Plugin/search_api/data_type/TextNgramDataType.php
@@ -11,7 +11,8 @@
  *   id = "text_ngram",
  *   label = @Translation("Fulltext (ngram)"),
  *   description = @Translation("Indexes field using the index's ngram analyzer (only useful if ngram analysis is enabled.)"),
- *   fallback_type = "text"
+ *   fallback_type = "text",
+ *   default = "true"
  * )
  */
 class TextNgramDataType extends DataTypePluginBase {

#14 to patch for composer. I changed the annotation for TextNgramDataType like in #9 so fields indexed using this data type are returned by Index::getFulltextFields.

For a specific project the fulltext search needed to work with keyword "b2b". To make tis possible we needed to add the digit class to the token_chars setting. The previous patch only support the letter class. I extend the patch to make the token_chars and custom_ token_chars setting configurable.

Patch #15 fixed all my issues after update from elastic 6.x.

Created a patch which works with D 9.x Elastic 7 and php 8.1

Please review it.

Litle finetuning to #19

Using exposed filter (>=, <= operators) for date fields and got:
Error: Call to undefined method Drupal\search_api\Query\ConditionGroup::getField() in elasticsearch_connector_autocomp_elasticsearch_connector_search_api_query_alter()

#20 patch modified.

The patch needs updating to work with elasticsearch_connector 8.x-7.x.

PrepareIndexEvent::getIndex doesn't exist, it should be PrepareIndexEvent::getIndexName instead. There may be other changes needed. If I have time today I will see if I can update the patch.

For one of our projects, we use the patch from #15, which worked fine. But due to deprecation of the boost-parameter, in combination with Elasticsearch 8, an error will occur:

[error]  Message: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Unknown 
parameter [boost] on mapper 
[rendered_item]"}],"type":"mapper_parsing_exception","reason":"Failed to 
parse mapping: Unknown parameter [boost] on mapper 
[rendered_item]","caused_by":{"type":"mapper_parsing_exception","reason":"Unknown 
parameter [boost] on mapper [rendered_item]"}},"status":400}

When the boost-parameter is omitted, it works fine. Therefore I've rerolled the patch from #15, without the boost property.
So this is not a reroll of #21 and therefore this still needs work.

Re-rolling patch #24 for latest version of dev

We have tested and are using the patch 3072676-25.patch from comment # 25.