Problem/Motivation
Currently looking to add search/indexing to our drupal site. Deciding to experiment with AWS OpenSearch Serverless I came across this plugin however, the plugin is unable to connect to said serverless collection. Having tried both creating an iam user and using permanent creds from said iam user. Or grabbing session creds from the task role and neither result in a successful connection. We then created a standard aws opensearch domain and connection was successful immediately both through basic HTTP auth with a master user and with using the same permanent creds using the IAM user as master user. The motivation is to use serverless opensearch instead so we dont have fixed compute we wont need for a long while, while search gets established.
Steps to reproduce
Create an opensearch serverless cluster in a private vpc with an opensearch serverless vpc endpoint, create network, data-access and encryption policies allowing for ecs task role/iam user access for data. Have an ecs cluster/service/task running in the same vpc with the ecs task role being granted aoss:APIAccessAll and aoss:DashboardAccessAll. Allow connection access to the vpce via a security group with https allow inbound rule for said ecs service. Use session creds from curl http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI in the ecs container in the aws sig connector, should result in a could not connect to opensearch in plugin. otherwise create an iam user with the same policy for aoss and use permanent creds/keys in connector setup.
Proposed resolution
Allow aws sig connector to differentiate between aws opensearch and opensearch serverless, by allowing users to specify which service type they want for the opensearch client builder https://git.drupalcode.org/project/search_api_opensearch/-/blob/3.x/modu... using https://github.com/opensearch-project/opensearch-php/blob/1866e6ee95c150...
Remaining tasks
User interface changes
API changes
Data model changes
| Comment | File | Size | Author |
|---|---|---|---|
| #18 | 3479880-18.patch | 4.4 KB | vijaycs85 |
Issue fork search_api_opensearch-3479880
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
dlund commentedI think I can implement this. I think the aws signature connector is missing the SigV4 service definition shown here. here in the opensearch client builder on this line here
Comment #3
dlund commentedComment #4
dlund commentedHaving done some more research. I have managed to connect to the serverless collection with the base php opensearch client as it is supported. I have tried adding the extra method to the client instantiation in this module but still the serverless cluster is not available or able to connect according to the generic error message (this should also be showing the actual exception not defaulting to an unhelpful generic message). I can only assume that in order to check for cluster health/availability that this plugin is using an API call not supported in aws opensearch serverless therefore will look to be unavailable. https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ser...
Comment #5
vijaycs85+1. I can confirm there are two issues:
1. Reading credentials from
AWS_CONTAINER_CREDENTIALS_RELATIVE_URIenvironment variable and get the key, secret, token & expire.2. Service setup.
For the first issue, if we don't pass the details via the form or settings.php, it would use
CredentialProvider::defaultProvider()and reads & fills the details. However for the service setup, there is no option at the moment. Here is the simple working script on ECS:Comment #6
kim.pepperSince we updated the opensearch-php library to 2.4.x you can use the
\Drupal\search_api_aws_signature_connector\AwsSigningHttpClientFactoryand pass your ownAws\Credentials\CredentialProviderin the service definition.We might be able to make it an option in the settings form.
Comment #7
vijaycs85Thanks @kim.pepper. For our setup, we are just missing service, so doesn't make sense to have custom credential provider. I have attached the patch to introduce the service field in settings form.
Comment #8
kim.pepperShould we default this to 'es'?
Comment #10
kim.pepperCreated a MR from the patch at #7
Comment #12
kim.pepperI think we need to handle existing sites a bit better.
Comment #13
vijaycs85you mean add an update hook?
Now this is a dropdown, so helps what services are available.
yeah, makes sense. updated.
Ideally yes as the library sets default to es.
P.S: I am pushing changes to both MR and patch as I need to apply on my project in latest 2.x.
Comment #14
kim.pepperYep.
Comment #15
vijaycs85Added update hook.
Comment #16
kim.pepperNW for the code style fails
Comment #17
kim.pepperComment #18
vijaycs85rerolled against latest 2.x
Comment #19
vijaycs85rerolled and fixed PHPCS issues on the MR at 3.x as well.