ES Connector 7.x has an alpha version for a while now: https://www.drupal.org/project/elasticsearch_connector/releases/8.x-7.0-.... This version support ES7.

Would it be recommended to open an additional 8.x-7.x branch for module, which has the updated composer dependency?

CommentFileSizeAuthor
#9 Screenshot from 2020-12-09 13-55-03.png55.49 KBrobindh
#4 3159624-4.patch1.64 KBrobindh
#3 3159624-3.patch1.07 KBrobindh
#2 3159624-2.patch320 bytesrobindh

Issue fork search_api_elasticsearch_attachments-3159624

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

robindh created an issue. See original summary.

robindh’s picture

robindh commented
Status: Active » Needs review
StatusFileSize
new 3159624-2.patch320 bytes

Here's a patch with the updated dependency

robindh’s picture

robindh commented
StatusFileSize
new 3159624-3.patch1.07 KB

Since the removal of mapping types in ES7 (https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-...), the
indexMapping() function throws an error on mapping import. This patch should fix it.

robindh’s picture

robindh commented
StatusFileSize
new 3159624-4.patch1.64 KB

For some reason the ingest pipeline stops when encountering a document without attachment field.
This also prevents these documents from being indexed.

Adding ignore_missing => TRUE to the pipeline config does prevent this though.

slejnej’s picture

slejnej commented

Hi,

this patch can only be applied if you add plugins in specific order. For example we're already using connector 7 with ES7 and want to add the attachment indexing but not possible because composer complains about dependencies to connector 6.

Any idea when it is going to be released so we don't have to hack around cloning repos?

Thanks

morleman@gmail.com’s picture

morleman@gmail.com commented

Hi,

Thanks for the patch robindh, as I was able to get past the mapping type removal now, however I still receive an error when enabling Attachments:

:{"type":"illegal_state_exception","reason":"[nested] nested object under path [es_attachment] is not of nested type"}}}]},"status":400} in Elasticsearch\Connections\Connection->process4xxError() (line 638 of /var/vamed-public/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php).

Currently installed: Elasticsearch Server 7.4.0 w/ ingest attachments plugin , Elasticsearch Connector: 8.7.0-alpha3, Search API Elasticsearch Attachments 8.x-6.0-dev (w/ 3159624-4.patch)

-Had to manually add the patch to the files as had issues with git and composer... perhaps that did it?

Since the Elasticsearch 6.0 is unsupported November 20, 2020, will there be an update to Elasticsearch Attachments for ES 7?

Driss BENTOUNSI’s picture

Driss BENTOUNSI commented

Hello,
As already mentioned in #5, i just hack around to install the module when you use ES7 and Elastic connector 8.7.
But even with this work around i have 3 issue :

Notice : Undefined index: caused_by dans Drupal\elasticsearch_connector\Plugin\search_api\backend\SearchApiElasticsearchBackend->indexItems() (/var/www/web/modules/contrib/elasticsearch_connector/src/Plugin/search_api/backend/SearchApiElasticsearchBackend.php ligne 506)

Notice : Trying to access array offset on value of type null dans Drupal\elasticsearch_connector\Plugin\search_api\backend\SearchApiElasticsearchBackend->indexItems() (/var/www/web/modules/contrib/elasticsearch_connector/src/Plugin/search_api/backend/SearchApiElasticsearchBackend.php ligne 506)

field [es_attachment] not present as part of path [es_attachment]. for id: entity:file/100:fr

When i enable Attachments.

Do you have any idea ? i cant downgrade to 6 as you know Elasticsearch 6.0 is unsupported

Thanks

robindh’s picture

robindh commented
Status: Needs review » Needs work

Hi, I must have missed #5 & #6 - I'll try to make some time later this week, to try and improve the patch.
I should add that I am not a maintainer for this module though, so I can't create a new branch / release even if the patch works.

I'll put the issue status back to 'needs work' for now

robindh’s picture

robindh commented
Status: Needs work » Needs review
StatusFileSize
new Screenshot from 2020-12-09 13-55-03.png55.49 KB

I've tested the patch more thoroughly now, using the following module versions:
Drupal core: v8.9.11
Search API: v1.18
Elasticsearch: v7.10.0
Elasticsearch ingest-attachment plugin: v7.10.0
Elasticsearch connector: v7.0-alpha3
Search API Elasticsearch Attachments: v8.6.0, using the patch in #4

I used a fresh drupal install, using the umami installation profile.

Indexing of attachments seems to work correctly for node & file datasources.
In the case of the node datasource; I added a media reference field on a node content type. The contents of media items (of bundle 'document') were added as es_attachment within each elasticsearch document. File datasource works out of the box. Files get indexed with their contents (also within the es_attachment field). Media datasource is not supported by default! To make this work, there is a patch included in this issue: https://www.drupal.org/project/search_api_elasticsearch_attachments/issu...

Adding a custom search implementation within a controller worked fine for me as well + the highlight processor is working.

I did notice that enabling the "Elasticsearch attachments" processor can throw this error: error-screenshot
This can be fixed by clearing the index via the UI, since index mappings will then be re-imported.

I did not encounter any of the errors posted above while testing; possibly there is an issue with a custom query / mapping alter that makes it incompatible with this module? I'd recommend trying to debug using kibana.

I'm moving the issue back to 'needs review', since the patch seems to work fine. Feedback / improvements are always welcome though!

jOpdebeeck made their first commit to this issue’s fork.

ronaldmulero’s picture

ronaldmulero commented

Works to index Content, Media, and attachments in Drupal 9 after applying these 3 patches:
* https://www.drupal.org/project/search_api_elasticsearch_attachments/issu...
* https://www.drupal.org/project/search_api_elasticsearch_attachments/issu...
* https://www.drupal.org/project/search_api_elasticsearch_attachments/issu...

Elasticsearch 7.4.2 with ingest-attachment plugin
Drupal 9.1.6
Elasticsearch Connector 8.x-7.0-alpha3
Search API 8.x-1.19
Search API Elasticsearch Attachments 8.x-6.0 (patched)

jlstrecker’s picture

jlstrecker
they/them
Primary language English
Location Athens, Ohio, USA
commented

I also tested the latest patch successfully. I would mark this RTBC, except that it should probably go on a new 8.x-7.x branch. @dakku, could you create that branch?

I tested with the 8.x-6.x branch plus these changes:

I successfully indexed nodes with PDF attachments (no errors), and spot-checked a few to make sure they showed up in search results, on a site with:

  • Elasticsearch 7.17.4 with ingest-attachment plugin
  • Drupal 9.3.14
  • Elasticsearch Connector 8.x-7.0-alpha3
  • Search API 8.x-1.23
olarin’s picture

olarin commented
Version: 8.x-6.0 » 8.x-7.0-alpha1
Status: Needs review » Fixed

I've committed these changes to a brand new 7.x branch and made an alpha release thereof.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.