Integrate a service or library to compare vectors of floats

So the previous one I found:

  public static function cosine($a, $b, $base) {
    $a = array_fill_keys($a, 1) + $base;
    $b = array_fill_keys($b, 1) + $base;
    ksort($a);
    ksort($b);
    return self::dotProduct($a, $b) / (self::magnitude($a) * self::magnitude($b));
  }

No matter how I pass the arguments to it, comparing "Test Article" input against a vector that was generated as "Test Article" or "This is a test article" at best only reaches 0.50000x score value - shouldn't it be much higher?

Looking at the test file:

https://github.com/mlwmlw/php-cosine-similarity/blob/master/test.php

It too looks like it expects the inputs as words and not floats.

Trying to use the NlpTools as this:

$score = $sim->similarity($embedding, $compare);

throws this exception (which I do not understand at the moment):

throw new \InvalidArgumentException("Vector \$A is the zero vector");

based on the docs, it does not look like it works with a vector of numbers like we get back from OpenAI: http://php-nlp-tools.com/documentation/similarity.html

Might be on to something here.

With this class: https://github.com/suryamsj/Cosine-Similarity/blob/main/function/CosineS...

I get this back. In this case, I am comparing a vector of "Test Article" assuming user input (search, wherever). This node that was scored has a title of "Test Article" and its body mentions the phrase "test article" among other words:

A 1.0 hit is what I would expect for an exact match and a high-ish score (body) otherwise.

The class needs to be converted into English (for docs purposes) and if we could put some test cases in to prove it is scoring correctly given an input, that would probably get us moving here.

Made short work of that... I asked OpenAI to translate the Indonesian comments into English:

Method 'calculate': Get the similarity value (similarity). Range of results: 0 - 1. Formula: sum (ai * bi) / (root (sum (ai ^ 2)) * root (sum (bi ^ 2)))

Method 'getTop': Calculate the top value (number to be divided). Formula: sum (ai * bi)

Method 'getDivider': Calculate the divider value. Formula: root (sum (ai ^ 2)) * root (sum (bi ^ 2))

Method 'calc': Static function to simplify calling the calculate function without having to initialize the class first.

If this is indeed correct, I can adapt this class into Drupal.

I am not 100% certain of this, but in the end I think this feature will necessitate an integration with a data storage that supports vector search, like Pinecone or Milvus. Unless there is a reasonable way to search with MySQL that is performant, I am unsure this will work at scale (100+ entities).

Some readings:

- https://www.pinecone.io/learn/vector-database/
- https://docs.pinecone.io/docs/query-data
- https://frankzliu.com/blog/a-gentle-introduction-to-vector-databases
- https://milvus.io/

Without a vector search, this does not leave many avenues for local storage to accomplish this. You'd be stuck loading several records just to loop and compare, where something like Pinecone can do that heavy lifting a million times faster.

https://docs.pinecone.io/docs/openai

Solr 9 appears to have dense vector search added, but it only supports a vector 1024 in length whereas OpenAI returns one that is 1536.

https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-searc...

I think what is worth trying here is to build a thin API client to interact with Pinecone and see what happens:

https://docs.pinecone.io/reference/describe_index_stats_get

Fortunately, the API is very lean.

https://docs.pinecone.io/docs/query-data

We already have the vector data, so it should be a matter of slotting in an HTTP client for query/comparison. We can store entity id, bundle, etc like so:

https://docs.pinecone.io/docs/insert-data#inserting-vectors-with-metadata

Seems like I am on the right track with this approach, whether or not Pinecone is used. I would need help with dynamic client design (i.e. multiple client, multiple settings, multiple config) but we just need 1 right now.

I wrote a rough client just to interact with Pinecone. Here is an example:

public function query() {
  return $this->client->post(
    'https://foo.pinecone.io/query',
    [
      'headers' => [
        'Content-Type' => 'application/json',
        'API-Key' => 'xxxxx',
      ],
      'json' => [
        'includeMetadata' => 'true',
        'namespace' => '',
        'topK' => 5,
        'vector' => [
          0.36112685691390006,
          0.35447963086021944,
          0.8178008998144455,
          0.06058116493124188,
          ... + 2044 more values

it responds with the top 5 hits for the vector:

That is... very cool. But this is example data.

What I should do now is insert the queue worker data into Pinecone instead of the local Drupal database. If the upsert there is successful, I should save the remote id in Drupal instead (similar to how Search API tracks items).

Yes. I should probably diagram it out, but:

1. "Entity" is saved
2. Job item is created
3. Queue worker processes the job
4. Data is saved (to db)

This is all working today. What I think we need to do instead, is:

4. Data is upserted to Pinecone, and the remote id record is saved to the db
5. Complete the client

Then, it would be a matter of integration modules (like one that bolts on functionality to Views) to provide input UI, take the input, vectorize it with OpenAI, then use the Pinecone client to do a search like I did above and get topK number of items back. From there, it would be that modules responsibility to ensure access levels are checked. This is very similar to how Search API modules work.

I also think we should save input that is vectorized from users, because the vector never changes (unlike a ChatGPT response), and that would really make this faster.

So then, its (user interface) -> vectorized input -> vector is queried against Pinecone -> response -> (module does something with the response).

The black and white response in my previous comment is the HTTP response from Pinecone using an example data set they provided. Ideally, our packed object would be in the metadata, which is possible to do:

https://docs.pinecone.io/docs/insert-data#inserting-vectors-with-metadata

There would be our entity id, bundle, etc, all items needed to run an entity load.

I'm wondering what happens if you ask to embed a giant block of text. I am thinking a body field on a content type that is a ton of text. OpenAI max token limit is 8196 for embeddings. I am also wondering if that will take longer to respond to queries. Hmm..

We also need to strip_tags and trim all strings before embedding.

Also a good read, and I think what we are trying to achieve here: https://www.pinecone.io/learn/metarank/

I committed some initial progress here to proof out this idea.

https://git.drupalcode.org/issue/openai-3339686/-/blob/3339686-integrate...

This is the client for Pinecone. Their API is very simple.

I amended the queue worker to send this data to Pinecone:

https://git.drupalcode.org/issue/openai-3339686/-/blob/3339686-integrate...

Almost instantly, its in Pinecone:

Given the breadth of options (Pinecone, Milvus, Typesense, etc) that someone could use to store vectors, I am not sure any of that code belongs in this module. A long term solution would be a connection manager or deriver that lets a module define a connection, then tell the embeddings module to use that connection object. I do not know how to do that yet, but also do not want to impede progress.

The next step is to find a scenario to query against something - any thoughts?

The 'id' key in Pinecone is the entity UUID from Drupal. It's possible we don't need any additional metadata at all.

Bah, scratch that. I need to use a unique id per row since we are indexing fields on entities and not the total entity worth of text. They require the id field, it is not an autoincrement.

https://docs.pinecone.io/reference/upsert

I meant, how can we prove this idea has legs - maybe a page in the admin that has an input field and lets you 'search' for matching content?

On the id front, the id can be a string in Pinecone, so I whipped up something quick to satisfy that.

  /**
   * Generates a unique id for the record in Pinecone.
   *
   * @param \Drupal\Core\Entity\EntityInterface $entity
   *   The entity being upserted.
   * @param string $field_name
   *   The field name of the vector value we are storing.
   * @param int $delta
   *   The delta on the field where this value appeared.
   *
   * @return string
   *   The identifier of this record.
   */
  protected function generateUniqueId(EntityInterface $entity, string $field_name, int $delta): string {
    return 'entity:' . $entity->id() . ':' . $entity->getEntityTypeId() . ':' . $entity->bundle() . ':' . $field_name . ':' . $delta;
  }

Kind of awkward Pinecone doesn't handle/generate its own records. But this should suffice.

Instead of embed 'on the fly' when someone performs a search, perhaps a better idea would be to try this first:

https://github.com/RichDavis1/PHPW2V

Take the user input, word2vec it with 1536 dimensions. If the math/output is right, it should be good enough to compare to what we got in OpenAI/Pinecone without incurring HTTP API call time and API usage hits.

Its hard to say though because they don't go into much detail here: https://platform.openai.com/docs/api-reference/embeddings

But we can always take a sample input string and tweak the arguments to word2vec and see if they get close.

edit: looks like that lib only saves to a file. Darn. Ok, I won't try to prematurely optimize it then, we will cross that bridge when we get there.

I whipped together a quick demo of searching input in Pinecone:

One thing I noticed is that if you embed several fields on a node, you have to filter out duplicate node ids from the response or you will see it over and over many times. I have no good idea on how to mitigate that yet. I also noticed that even for a direct hit, superfluous words like 'and a the in' etc can influence other results much more.

I spent a little time on this this weekend looking into a path forward here.

I took a survey of 10 or so known 'vector databases' (Pinecone, Milvus, etc). I thought I could make a plugin object and plugin manager for all the connections that could be built. This may be a possibility:

interface VectorClientInterface {
  
  public function query(array $parameters);

  public function insert(array $parameters);

  public function update(array $parameters);

  public function delete(array $parameters);

}

Where we don't care how you do it, as long as you do it (textbook interface). Beyond that, integrators would be on their own, but we need them to at least ensure the plugin does these four things. I cannot make it more specific than this - different databases have variable list of arguments and in some cases they don't use a vector of floats at all as the comparison.

Yeah - I will probably have to shelve that behavior. I've never created dynamic plugins that store their own configuration before (a la Search API backends). I have it mostly there... just cannot get the configuration to save into state like other Drupal configuration.

Basically, if you were to code a new plugin using the annotation I made, it would appear on the form with its own settings (as defined by that plugin) since each service has different requirements in its API.

From this point, the user would select which plugin is 'active' - and that would be loaded dynamically in the queue worker.

I think for now I will have to shelve the work in a branch, until I can finish that or someone helps me figure out how to finish a complete plugin manager / collection / config manager implementation. I have never really done that before.

For now I have merged in my WIP to dev. The client plugins can be circled back on later, but the work is there.

There is a UI now on the embeddings config that lets you try different strings and search against what is stored in Pinecone.

I get odd results sometimes and I am not entirely sure why. For instance, I filled a node with gibberish typos. When I search "Drupal" or "Drupal 10", its returned with a relevancy of .8-.85, which is very high for something that doesn't mention this or anything close to it at all.

There are two interfaces to try... there is a 'Search test' on the embedding setting section. I added a new POC feature too that tries to figure out if your content is 'too similar' to other content in regard to SEO. If you have a Pinecone account you can try it out.

I am not 100% on if the vectors are being generated/stored efficiently or if I just do not have good content (I really don't have much). Sometimes the results are very similar, sometimes the results have a match that don't make any sense.

I think step one was enforcing namespaces when saving, which I did in #3341713: Add a namespace for Pinecone when upserting instead of leaving it blank. I would need to read up more on similarity search accuracy.

If I am not mistaken, it sounds like entities should be submitted as an entire string of text instead of individual field values, based off the examples I am reading:

https://github.com/openai/openai-cookbook/blob/main/examples/vector_data...

Sure, but:

"input
string or array
"

Right now the code is getting an embedding per field value, instead of all values as one embedding. I am trying to figure out why I get relevancy scores that are just not relevant at all. For example:

There should not be any results... at least any results scored that high. There is nothing close to "Amigo" in any of those nodes. That is why I wonder if the implementation is correct.

Even if I type in "zzzzzzzzzzzzzz", I still get results that are ranked high.

Also, in the Pinecone examples, they use stop words in some examples:

https://docs.pinecone.io/docs/semantic-text-search

Like not embedding them and storing that... that would probably help too.

Hmmm.... this could be a bug with Pinecone: https://community.pinecone.io/t/cosine-similarity-same-datasets-one-in-m...

Note for later, we may leverage this module to support multiple clients for vector db connections: https://www.drupal.org/project/http_client_manager

Most of these APIs are HTTP based. No need to reinvent the wheel.

Yes, this is mainly experimental. OpenAI returns a vector of 1536 length. There are not much in the way of PHP libraries that do this (embedding) that I could find. Solr has deep vector searching in 9.1, but its length is 1024 (plus, we'd need 9.1 support in Search API Solr). Right now the 'similar title' function of openai_content uses this, but so far I have found it to not be nearly as accurate as Solr on even the loosest comparisons. I cannot tell if I am using it correctly or if Pinecone is not that accurate. Earlier in the thread I had considered finding an external library that could be called with FFI somehow to generate an embedding, but was unsuccessful so far.

If you can really build all the things Pinecone advertises with a vector database on its homepage, I lean towards the direction of perhaps I am not searching it right - but you can see the API, its very simple and straightforward. They do mention that it is generally useful for augmenting search and or 're-ranking', but the examples in their docs appear to work far better than what I was able to try and do. I could also be misunderstanding the use case, or its also possible the data is not being compared correctly (I have seen a couple threads in Pinecone asking the same).

If this can work (not strictly for search) I am still interested in the possibilities around content recommendations and the like. It could prove a low-cost solution in an otherwise generally high cost service (content recs, content personalization). We have the appropriate data, I am just not connecting the dots in accordance with what their docs show. We can poke at this while waiting for Search API Solr to update and deep vector field support is in.

Adding some links here as notes, clearly we set out on the right track and its 'almost there'. I think some of the work I did in the other thread on improving the summary suggestion will potentially help accuracy of this feature too.

https://www.crunchydata.com/blog/whats-postgres-got-to-do-with-ai

https://vimota.me/writing/gpt3-klaviyo-automation

This is actually really really encouraging - we had the right idea to begin with. It's just a matter of making it hum now. How exciting!

Also a note for myself - to get around the embedding call limitation (60 per minute) I need to add a sleep() call to the end of the queue worker process.

Yes. I have read pg_vector, its a required extension for Postgres to do this. So far, I have seen no indication MySQL/MariaDB offer this, or, will offer this. Some have posted stored procedure routines, but thats.... they are difficult to install via module and usually not allowed at most managed hosts. So far, only Solr will be offering vector search functionality in 9.1 ( but its limited to 1024 length - OpenAI returns 1538).

We don't have to find one (no PHP library seems to exist anyway). I did not know enough upfront when starting this issue. OpenAI can convert text to a vector, then that vector can be stored in Pinecone (or any client a consumer wants to integrate to).

The only question right now is, is it being done right? Like you said, sometimes I get good results, and other times the results don't make sense. I have not been able to narrow that down.

I think we can stick with the search aspect, but "similar content title" feature probably needs to go away, its not as accurate as I would think.