When an indexed entity is deleted, only 10 of its associated chunks are removed from the database. This leaves orphaned chunks behind, which can cause errors when attempting to load the deleted entity.
1. Index an entity that results in more than 10 chunks. 2. Delete the entity. 3. Observe that only 10 chunks are deleted; the remaining chunks persist in the index. 4. Attempt to load the deleted entity, which may lead to errors due to the leftover data.
The deleteItems function relies on getVdbIds, which in turn uses querySearch to retrieve IDs. However, querySearch limits its results to 10 items, causing incomplete deletion.
I have provided a patch that removes this limit, ensuring all associated chunks are properly deleted when an entity is removed.
| Comment | File | Size | Author |
|---|---|---|---|
| #2 | delete-all-chunks-from-index-3521461-2.patch | 536 bytes | ayrmax |
Show commandsStart within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
changes
Comments
Comment #2
ayrmax commentedComment #3
scott_euser commentedHi! Thanks for the contribution, can you make a merge request instead so tests run + easier for us maintaining to review/comment. Thank you!
Comment #5
scott_euser commentedComment #6
gxleano commentedThanks @ayrmax and @scott_euser!
I've been testing the issue and it seems to be working as expected, so moving to RTBC.
Comment #7
gxleano commentedComment #8
scott_euser commentedI suppose its fine too; I searched all code that calls querySearch and they all seem to pass a limit so it probably has no other consequences.
I don't understand why the 16,384 limit passed already is not fine though as that seems to match Milvus limit documentation: https://milvus.io/docs/limitations.md - is it indicating that this is a symptom of a wider problem of us not handling limit/topk correctly in Milvus?