Problem/Motivation

Setting up and maintaining an ElasticSearch cluster is non-trivial. The creators of ElasticSearch, Elasticsearch B.V., sell an Elastic Cloud service, which provides turn-key ElasticSearch clusters ("Hosted deployments"), and it's 30-day trial is helpful for site builders who want to evaluate ElasticSearch, but don't want to spend time setting it up. Elastic Cloud also adds features that are difficult to self-host but desirable for larger organizations (especially Web Crawlers for federated search).

Elastic Cloud Hosted Deployments and the official elasticsearch/elasticsearch library support connecting to an ElasticSearch Server using a Cloud ID and API Key; but ElasticSearch Connector (currently) doesn't expose this connection method by default.

Proposed resolution

Add an ElasticSearch Connector plugin (similar to the existing BasicAuthConnector and StandardConnector) that allows users to establish a connection to an Elastic Cloud Hosted Deployment acting as a Search API Server by entering a Cloud ID and API key.

Remaining tasks

  1. Write a patch - merge request !91 created by @mparker17 in #2
  2. Review and feedback - done by @kdborg in #6, @jeroent in #8, and now by @mparker17
  3. RTBC and feedback - done my @mparker17 in #15
  4. Commit - done by @mparker17 in #16
  5. Release - released by @mparker17 in 8.0.0-alpha5

User interface changes

Adds a new ElasticSearch Connector type when adding a Search API server with an ElasticSearch backend.

API changes

Only API additions.

Data model changes

None.

CommentFileSizeAuthor
#9 3522107-8.patch21.6 KBjeroent
#6 elasticserach_connector-elastic_cloud-3522107-6.patch21.16 KBkdborg@gmail.com

Issue fork elasticsearch_connector-3522107

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

mparker17 created an issue. See original summary.

mparker17 opened merge request !91

mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Issue summary: View changes
Status: Active » Needs review
Issue tags: +Needs tests

I've added some code in !91; it needs tests; but reviews are welcome.

mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Issue tags: -Needs tests

We don't have any automated tests for the basicauth, and no specific tests for the standard connector either. Further, search_api_opensearch doesn't have any specific automated tests for its basicauth or standard connectors either.

search_api_solr has test connectors (i.e.: which serve as test doubles in tests), \Drupal\search_api_solr_test\Plugin\SolrConnector\BasicAuthTestSolrCloudConnector, and \Drupal\search_api_solr_test\Plugin\SolrConnector\BasicAuthTestSolrConnector, but the search_api_solr module works in a slightly different way (i.e.: the connectors make the connection to the backend directly; but that's not the case in elasticsearch_connector and search_api_opensearch).

Looking into adding tests for the connectors we have seems non-trivial, so I've created some follow-up issues...

  1. #3532389: Have ElasticSearchConnectorInterface::getClient() return an Elasticsearch ClientInterface instead of a Client
  2. #3532403: Add tests for the Connectors

... and I'm going to suggest that we merge this without tests.

mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Issue summary: View changes

Updating the issue summary

kdborg@gmail.com’s picture

kdborg@gmail.com commented
StatusFileSize
new elasticserach_connector-elastic_cloud-3522107-6.patch21.16 KB

I've created a patch from the change link. The change is connecting to an Elastic Cloud instance and working well.

mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Status: Needs review » Reviewed & tested by the community

Thanks! I'm glad to hear it is working well!

When someone else (i.e.: who didn't write the code) tests the code, and finds it to be working, we mark the issue as "Reviewed & tested by the community", which I'm going to do now!

jeroent’s picture

jeroent
Primary language Dutch
Location 🇧🇪
commented

We are also using this patch on a project. However, in Elastic Cloud, we are using a serverless project.
This patch was working fine for us, with some extra changes.

In BackendClient, we got errors on the /_close and /_open API calls, so we just had to disable those.
But perhaps this is not the right place and this should be handled in a follow-up issue.

jeroent’s picture

jeroent
Primary language Dutch
Location 🇧🇪
commented
StatusFileSize
new 3522107-8.patch21.6 KB
mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented

(sorry for the noise: I rebased onto the latest 8.0.x, so that the phpstan pipeline wouldn't warn us about a deprecation that was fixed in 8.0.x)

mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Status: Reviewed & tested by the community » Needs review

Hmm... because of the new change in #9, I'm going to need to re-review (and test if the change from #9 works).

mparker17 changed the visibility of the branch 3522107-elastic-cloud-connector to hidden.

mparker17 changed the visibility of the branch 3522107-elastic-cloud-connector to hidden.

mparker17 changed the visibility of the branch 3522107-elastic-cloud-connector to active.

mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Issue summary: View changes
Status: Needs review » Reviewed & tested by the community

Apologies for the delay.

I've rebased the code in this branch onto the latest 8.0.x.

I've reviewed the changes in #9 and they look fine to me.

Sounds like this has been reviewed by @kdborg in #6, @jeroent in #8, and now I'm using this patch in production, so I think we can move it along to RTBC.

Assigning credit.

  • mparker17 committed 8b4b3dc4 on 8.0.x 
    [#3522107] feat: Connect to an Elastic Cloud Hosted Deployment acting as...
mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Issue summary: View changes
Status: Reviewed & tested by the community » Fixed

I wanted to publish some Elasticsearch Connector documentation mentioning the connection types that the module makes available, and I didn't get any responses saying I shouldn't merge this, so I've merged it.

Thanks everyone for your contributions! I am hoping to make a new release next week, which should include this code.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Issue summary: View changes

Released in 8.0.0-alpha5!

mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Issue summary: View changes

(making issue summary more consistent with newer issues)