Problem
Deletion of existing records in pgvector table fails due to a bug in SQL DELETE query. Both dev and alpha2 versions have this bug.
The bug is in the function PostgresPgvectorClient->deleteFromCollection(), line 239:
query: "DELETE FROM {$escaped_collection_name} WHERE drupal_entity_id IN {$prepared_ids};"
instead of 'drupal_entity_id' it should have 'id'.
Steps to reproduce
1. Use SELECT in PostgreSQL to see all records for a particular drupal_entity_id
2. Open that node for editing via Drupal UI
3. Hit Save button
4. Use SELECT again to see all records for the same drupal_entity_id
5. Verify that you have a duplicate set of records (number of records doubled) since none got deleted.
Proposed resolution
Replace the query on line 239 of PostgresPgvectorClient.php by this:
query: "DELETE FROM {$escaped_collection_name} WHERE id IN {$prepared_ids};"
User interface changes
None
| Comment | File | Size | Author |
|---|---|---|---|
| #6 | 3553475-deletion-of-chunks-1.0.x.patch | 1.2 KB | calmforce |
| #2 | 3553475-deletion-of-chunks-1.0.x.patch | 689 bytes | calmforce |
Issue fork ai_vdb_provider_postgres-3553475
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
calmforce commentedI made the change: replaced drupal_entity_id by id in the DELETE query.
Attached is the patch file.
To verify that the patch works:
1. Install the patch.
2. Use SELECT in PostgreSQL to see all records for a particular drupal_entity_id
3. Open that node for editing via Drupal UI
4. Hit Save button
5. Use SELECT again to see all records for the same drupal_entity_id
6. Verify that you DO NOT have a duplicate set of records, the number of records stay the same if no editing of the node was done.
This is the fix in GitLab:
https://git.drupalcode.org/issue/ai_vdb_provider_postgres-3553475/-/compare/1.0.x...3553475-deletion-of-chunks
Comment #3
calmforce commentedComment #5
calmforce commentedThe patch tested locally, chunks properly deleted.
Created the Merge Request:
https://git.drupalcode.org/project/ai_vdb_provider_postgres/-/merge_requests/14
Comment #6
calmforce commentedThe updated patch file with limit = 16384 is attached.
Comment #7
nikro commentedAdditional investigation
Confirmed against current
1.0.xwith normal Search API dynamic indexing.The existing patch fixes the immediate mismatch by changing the final delete to use
id, and raises the lookup limit. There is a simpler path:deleteItems()already receives the Search API item IDs (for exampleentity:node/1174:en), whiledeleteFromCollection()already deletes ondrupal_entity_id.The proposed follow-up changes
deleteItems()to pass those item IDs directly todeleteFromCollection(). This avoids the primary-key lookup entirely. In particular, it avoids both the original identifier mismatch and a fixed lookup limit.The resulting statement is effectively:
That removes every stored chunk for the item before the fresh chunks are inserted.
How to test
drupal_entity_id.We verified the equivalent scenario in a sandbox: re-saving an item with 341 chunks left 341 rows, not 682, and a search for a marker in the final chunk still returned the item.
Comment #8
nikro commentedHm, had issues pushing my changes. Pushed it over the original MR: https://git.drupalcode.org/project/ai_vdb_provider_postgres/-/merge_requ...
The old changes are in a separate branch (pre-rebase, as I rebased on latest 1.0.x).
Comment #9
arianraeesi commentedComment #10
nikro commentedFollow-up: progressive indexing passes
I added a second, focused commit to the same MR. The first commit makes deletion target the correct
drupal_entity_id; this also means the Postgres provider must honor AI Core'sskipDeleteItemIdslist. ItsindexItems()override previously ignored that list and deleted the item's earlier chunks again on every progressive continuation pass, leaving only the final slice.The follow-up excludes in-progress item IDs from deletion. The first pass still removes all stale vectors after an entity update, while later passes retain the new chunks already written. This belongs to the same issue because it completes the delete-before-reindex lifecycle.
Tested through a normal Drupal node save and Search API indexing: the initial article progressed
10 → 20 → 30 → 33chunks. After replacing the body, it progressed10 → 20 → 27. No stale marker remained, and the replacement tail marker was indexed.Comment #11
arianraeesi commented