When chunking content for AI Search (chunking being fundamental for vector databases), sometimes an item is so long that it cannot be indexed in a single go and needs have multiple runs in the batch to complete it. This is typically worked around by e.g. using LLM to shorter content (high token usage and cost) or having much larger chunk size (less representative vectors, so less accuracy).
We have attempted with subbatch here #3487487: Improve AI Search Module Indexing to Handle Long-Running Chunk Embedding Processes at first but the UX for subbatch is very confusing, the end user sees progress bar after progress bar and has no real concept of when it'll finish in a big site.
If we could override the ::create() method of the service, we could extend it to have our own implementation that modifies the run so e.g. when indexing 10 items, if item number 4 needs 2 runs, we can then have it hit that again.
Described above
Change IndexBatchHelper to a service so it can be overridden by AI Search module (submodule of AI module)
MR
Show commandsStart within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
changes
Comments
Comment #3
scott_euser commentedThis looks much bigger than it is because of the interface; but really could be without interface and leave IndexBatchHelper completely untouched if you prefer.
Considered modifying CommandHelper constructor but that is also used by search_api_solr for example and given the IndexBatchHelper is already just calling static method ::create() from there, figured \Drupal::service() would be okay there, but feel free to disagree if you think coordinated release is better like we did with the Search API Fields indexing page.
Thanks!
Comment #5
drunken monkeyThanks for posting this suggestion!
I don’t quite understand the whole reasoning, but in principle this should always have been a service, just like the other helper services in the
Utilitynamespace. So, yes, let’s do it.However, just making this a service without any changes doesn’t make much sense to me. Definitely, all those methods should be instance methods, not static.
I rewrote the MR in that direction and added a change record. Please tell me what you think.
Still needs a bit of work to get rid of static
\Drupal::…()calls in the service class, but ran out of time today. Feel free to tackle it if you’re able, otherwise I’ll get to it tomorrow.Comment #6
drunken monkeyComment #7
drunken monkeyOK, DI is complete and pipelines should be passing. Handing off for review.
One thing, now that we’re creating a new class anyways: Would you say
IndexingBatchHelperis a better name thanIndexBatchHelper? The latter sounds a bit awkward to me, or maybe more like it handles some operation on an index entity, not an indexing operation.Also, having a slightly different name might help in distinguishing the old and the new class. (Other naming suggestions also welcome.)
Comment #8
drunken monkeyComment #9
scott_euser commentedThanks! Will take a look asap, much appreciated!
Ultimately the reasoning for the change is because e.g. a single entity can take multiple passes from the batch to complete when generating embeddings as it can be very long running for huge processes.
So indexing 5 items might take 8 passes in the batch if 1 item needs 3 extra passes before LLM has generated all embeddings.
First step which we do programmatically is Chunk the content using an embedding strategy (usually a combination of breaking into chunks + re-adding things like title, short description as contextual content so the chunk doesn't loose its overall semantic meaning once converted to an embedding (set of vectors). This is quick so we quickly know that e.g. a giant article had 45 chunks, and we can estimate that roughly 10 chunks can be processed per batch.
So it means we need to revisit the progress and slow it down for that long running item, potentially something like this:
Whereas currently the index helper progress start/finish is determined and fixed from the start of the batch setup. So its that method that I'd want to extend to manipulate the progress specifically for AI Search.
We had originally tried sub-batch but the UI is very confusing as progress bar then keeps jumping from 0 to 100% and its really hard as a user to understand how close to the overall total we actually are.
Comment #10
scott_euser commentedThank you! This is perfect, I am able to then override it fine and AI Search to handle the chunking now.
I did the rename to IndexingBatchHelper - makes sense to me.
Comment #11
drunken monkeyExcellent, thanks a lot!
Merged.
Thanks again!