Custom Tokenizer

After several hours of research and not finding any PHP library, I concluded there are two possible approaches:

• Re-use LiteLLM: They provide a /utils/token_counter endpoint specifically for this purpose.
  This approach will be slightly slower since it operates over HTTPS, but it will be accurate.
• Build and use SentencePiece locally: For those who prefer a faster solution, they can compile and install SentencePiece on their system and configure it as an executable. This will offer the best performance but requires server-level changes.

We will need to implement both options to provide customers with flexibility.

Even more than that - we need to create a custom Tokenizer that is aware of the custom embeddings we use, particularly in the context of the Search API.

This tokenizer should be injected into the system - potentially via a decorator - so that we can override or “hijack” the default tokenizer and replace it with our own implementation when needed.

This approach will ensure that tokenization is consistent with the embedding model’s expectations and that downstream processes relying on accurate token boundaries continue to function correctly.

I'll take this over - in a separate branch.

Testing Instructions for TokenAware Text Chunking Implementation

Code Review - Overview

This MR introduces a token-aware text chunking system with custom tokenizer support for Dropsolid AI integrations. The implementation includes:

• New Files Added:
  • src/Service/TokenAwareTextChunker.php - Core chunking service with density-based probing
  • src/Service/TokenAwareTextChunkerInterface.php - Service interface
  • src/Plugin/EmbeddingStrategy/TokenAwareEmbeddingStrategy.php - Embedding strategy plugin
  • src/Tokenizer/DropsolidXlmRobertaTokenizer.php - Custom XLM-RoBERTa tokenizer
  • src/Tokenizer/DropsolidTokenizerDecorator.php - Tokenizer decorator for AI module integration
  • src/Form/TokenizerSettingsForm.php - Admin configuration UI
• Files Removed: Old RecursiveTextChunker and RecursiveCharacterTextSplitter implementations

Prerequisites

Required Access: You must have access to Dropsolid's infrastructure services:

• Dropsolid LiteLLM service (for HTTP-based tokenization)
• Dropsolid Vector database service
• It's assumed Dropsolid.ai (litellm etc) are configured and working...

Testing Steps

Step 1: Configure the Tokenizer

1. Navigate to /admin/config/ai/ai-dropsolid/tokenizer
2. Select "LiteLLM (HTTP) tokenizer – default" from the "Tokenizer mode" dropdown
3. Ensure your LiteLLM settings are configured (should already be set up if you have access)
4. Click "Save configuration"
5. Expected Result: You should see a success message showing demo tokenization with approximately 24 tokens counted for the test string

Step 2: Create Search API Server

1. Navigate to /admin/config/search/search-api/add-server
2. Enter a server name (e.g., "Dropsolid AI Server")
3. Select "AI Search" as the backend
4. In the backend configuration:
   • Embedding engine: Select eu-e5-large-embeddings-selfhosted
   • Tokenizer for token counting: Select Dropsolid AI XLM-Roberta-base from dropdown
5. Expand "Advanced: Embedding strategy configuration":
   • Embedding strategy: Select Token-Aware Embedding Strategy (Separator-Based Chunking)
   • Maximum chunk size: 480 or 500 tokens
   • Overlap: 70, 80, or 100 tokens (your choice)
   • Context content maximum percentage: 25% or 30%
6. Click "Save"

Step 3: Create Search API Index

1. Navigate to /admin/config/search/search-api/add-index
2. Enter an index name (e.g., "Dropsolid Content Index")
3. Select your newly created server from Step 2
4. Choose datasources (e.g., "Content" for nodes)
5. Configure which fields to index (e.g., Title, Body, etc.)
6. Click "Save"

Step 4: Test Chunking with Content Preview

1. Navigate to your index: /admin/config/search/search-api/index/[your_index_name]/fields
2. Click on the "Preview contents to be vectorized" tab
3. Use "Search for an item by title" field to find a content item
4. Select a document with substantial text content (long body field)
5. Expected Results:
   • Text should be chunked into multiple segments
   • Each chunk should be ≤ 480-500 tokens (as configured) - note - it's pretty hard to identify but you can at least see that links should not be cut-off between chunks (please use some content with a ton of URLs)
   • Consecutive chunks should have 70-100 tokens overlap (as configured)
   • Chunks should break on natural boundaries (paragraphs, sentences, words)
   • No URLs should be split mid-way

Step 5: Go wild and index stuff

Process:

• Start indexing content, use 5-10 nodes per run.
• You will have to wait for the job to finish, it will not be blazing fast; but not super slow either - eventually you should be able to see all the content indexed w/o any errors

Technical Notes

• Tokenization Efficiency: The new implementation uses only 3-4 tokenization API calls per document (density probes + optional validation) instead of N calls for N chunks
• Separator Hierarchy: Text splits on \n\n → \n → . → \t → in order of preference
• Density Probing: Samples text at 10%, 50%, and 90% positions to estimate token density before chunking

Troubleshooting

If tokenization test fails:

• Verify LiteLLM service credentials in /admin/config/ai/provider/litellm
• Check that you have network access to Dropsolid's LiteLLM endpoint
• Review recent log messages for API errors

Thanks for the review!

Added 1 change and merged it in!