Make use of Milvus' Grouping functionality

Example group by field added to make it easier for people to test how this behaves in the meantime.

Postponed on #3526390: Improve the AI Search recursive retrieval of a specific quantity of results

Thanks for the progress here! I added feedback to #3526390: Improve the AI Search recursive retrieval of a specific quantity of results

Added a nitpick + a question, thanks!

Had a proper test of this but unfortunately Grouping is just not working as advertised at the docs page https://milvus.io/docs/grouping-search.md

I did set the group size to 1 per my review comment and tried with both strict true and false (though from my understanding of the docs, false should be fine).

You can see in this screenshot the debug of params + debug of the results in ::searchWithGrouping() shows all items have same drupal_entity_id yet are not grouped together:

From Slack message I think this is needs review again as well

Okay updated ddev container #3549396: Update docker examples to give this a retest as discussion in slack

Gave this a retest, Milvus 2.5.18 and not working for me still sorry. Here's how I am testing:

1. Checkout merge request from #3526390: Improve the AI Search recursive retrieval of a specific quantity of results
2. Checkout this MR
3. Index a content item with many chunks
4. Do a vector search for similar content
5. See that multiple chunks from the same drupal_entity_id are returned

How are you testing this out yourself? Before MR can you confirm you're getting 2 chunks from same drupal_entity_id, then checking out this MR and you're subsequently getting just 1 chunk from drupal_entity_id?

Added this to ::searchWithGrouping() to demonstrate:

      $response = $this->makeRequest('vectordb/entities/search', [], 'POST', $params);
      dvm($params);
      dvm(Json::decode($response));

Screenshot of Milvus version:

Maybe there is some setup needed to achieve the below bit of the docs here?

Indexing: This grouping feature works only for collections that are indexed with these index types: FLAT, IVF_FLAT, IVF_SQ8, HNSW, HNSW_PQ, HNSW_PRQ, HNSW_SQ, DISKANN, SPARSE_INVERTED_INDEX.

Okay I dove into this in more detail and I can see that the field actually needs to be created separately, and not just as an attribute in the generic $meta field (the 'dynamic' field) where we were putting drupal_entity_id, drupal_long_id, and any other filterable attributes.

I documented in comments the approach in more detail. To make use of this you actually have to fully drop your existing collection as the field seems to need to be created at the moment the collection is created in the first place.

I think last step is to handle not forcing reindexing, but instead warning the site builder they should reindex when not supported which to me means:

1. We need to store cached or static whether the collection has the field 'drupal_entity_id' by checking the describe collection which returns the field schema https://milvus.io/docs/view-collections.md
2. We then need to check if the field exists in the supportsGrouping + in the insertIntoCollection (the latter needs to conditionally set drupal_entity_id param only if supportsGrouping)
3. We should show on the Milvus Search API Server status page whether grouping is supported, and if not, have a message like:
   

   In order to support grouping, your collection needs to be dropped as it was created either prior to AI VDB Provider Milvus supporting grouping, or it was created manually without the 'drupal_entity_id' field in some other way. After dropping, resave the Milvus Search API Server to recreate the collection automatically with the 'drupal_entity_id' field needed for grouping and re-index your content.

Setting to Needs Review to get review on the approach, after which we can go back to Needs Work to implement the above (if others are in agreement)

Excellent thank you!

I did create a follow-up in AI Search here #3549911: Recursive retrieval using the supported grouping may still need recursion as while this will mostly work, there are some edge cases where it still needs some recursive retrieval