Problem/Motivation

OpenSearch supports neural search using text embeddings: https://opensearch.org/docs/latest/search-plugins/neural-search/

The purpose of this issue is to add neural search capabilities to this module.

Proposed resolution

OpenSearch can generate text embeddings for you, but we decided to do it on the Drupal side instead. The main reason is to allow us to use AI models that are not supported by OpenSearch.

Remaining tasks

- Create neural backend plugin type
- Implement ollama backend plugin
- Implement OpenAI backend plugin
- Generate text embeddings during indexing and add the vector to the indexed data
- Create views filter for neural search (which generates the vector for the search query and queries OpenSearch based on that)
- Create configuration UIs
- Add test coverage

Issue fork search_api_opensearch-3472769

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

slashrsm created an issue. See original summary.

slashrsm’s picture

slashrsm commented
Version: 2.x-dev » 3.x-dev
slashrsm’s picture

slashrsm commented
Issue summary: View changes
Status: Active » Needs review

Still work in progress, but the idexing side of things work.

slashrsm opened merge request !64

kim.pepper’s picture

kim.pepper
Primary language English
Location 🏄‍♂️🇦🇺Sydney, Australia
commented
Status: Needs review » Needs work

Thanks Janez. Looks like a great start. Besides the linting errors, I can see there are still a lot of hard coded values, and we're missing tests. I know it's a complex area, but it would be good to have some basic docs on setup and links for further info.

acbramley’s picture

acbramley commented

Quick n dirty review for now as this is obviously still WIP - great to see movement in this area though!

slashrsm’s picture

slashrsm commented
Issue summary: View changes

slashrsm changed the visibility of the branch query_side to hidden.

slashrsm’s picture

slashrsm commented

After watching Driesnote and looking into AI module a bit I realized that we are basically re-implementing their provider plugins here. In order to avoid that I decided to depend on the AI module for providers. Updated MR assumes/uses #3477750: Provide embeddings vector size , which add vector size function that we rely on.

maximilianmikus’s picture

maximilianmikus commented

I was looking into adding OpenSearch as a vector database provider and I found this issue by chance. I was wondering if it wouldn't be better to put this functionality in its own provider module? I started a project just for that before I found this issue by chance.

damienmckenna’s picture

damienmckenna
Location TN, USA
commented

FYI the separate provider module has been deprecated in favor of this issue, though the current MR doesn't apply against the 3.x branch.

kim.pepper’s picture

kim.pepper
Primary language English
Location 🏄‍♂️🇦🇺Sydney, Australia
commented

A recommended approach for vector indexing is an ingest pipeline. I wonder if this issue could be expanded to include support for that?

kim.pepper opened merge request !107

kim.pepper’s picture

kim.pepper
Primary language English
Location 🏄‍♂️🇦🇺Sydney, Australia
commented
Status: Needs work » Needs review

Started work on a more integrated approach. At this stage all the MR does is set index.knn = TRUE when creating the index.

In order to have knn enabled on an index, we need to set that option when creating the index. We can't change it after.

This meant we needed to refactor the addIndex() method to not create then update settings, but to pass the settings at creation time. This refactoring could potentially be split out into a separate issue.

kim.pepper’s picture

kim.pepper
Primary language English
Location 🏄‍♂️🇦🇺Sydney, Australia
commented

Ran into a bit of an issue with the pipelines. In order to have Opensearch generate the text embeddings, you need to specify text field to embedding field mappings when creating the pipeline. I don't think it would be easy to dynamically create a pipeline like this with search api.

I'm going to check out the https://www.drupal.org/project/ai_vdb_provider_opensearch module to see if the built-in AI Search would work.

mstrelan made their first commit to this issue’s fork.

mstrelan’s picture

mstrelan commented

Updated the MR so choosing the Text Embedding (Vector) data type creates the field as a text field. Then the field mapper is decorated to create a paired knn_vector field. That will let us dynamically create the ingest pipeline.

mohit_aghera made their first commit to this issue’s fork.

kim.pepper opened merge request !109

kim.pepper’s picture

kim.pepper
Primary language English
Location 🏄‍♂️🇦🇺Sydney, Australia
commented

Chatting with @mohit_aghera in slack and we have a new approach based on a semantic field type in OpenSearch 3.1. This greatly simplifies the amount of work needed to support neural search. See https://docs.opensearch.org/docs/latest/field-types/supported-field-type...

There is a bug with remote text embedding models, which was recently fixed and merged but not yet in a release. https://github.com/opensearch-project/neural-search/issues/1426

mstrelan closed merge request !109

kim.pepper’s picture

kim.pepper
Primary language English
Location 🏄‍♂️🇦🇺Sydney, Australia
commented
Status: Needs review » Fixed

We have moved this to it's own project https://www.drupal.org/project/search_api_opensearch_semantic with the latest from MR !107.

Please continue any work over there.

Marking this issue as Fixed to provide contribution credits.

kim.pepper closed merge request !107

kim.pepper closed merge request !64

kim.pepper’s picture

kim.pepper
Primary language English
Location 🏄‍♂️🇦🇺Sydney, Australia
commented
Related issues: +#3540868: Add a BeforeIndexCreateEvent

Created #3540868: Add a BeforeIndexCreateEvent for the new event.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.