The 2026 Drupal Business Survey is open. Your response will help shape the future of the Drupal business ecosystem.Problem/Motivation
The token-aware chunker in TokenAwareTextChunker::chunkText() clamps its guard density to 0.36 tokens/char whenever the probes report lower values. When dense inline assets (e.g. data URIs, repeated embeds) fall outside the probe windows, the guard never reflects their higher density.
On validation failure the code only scales the character budget by a fixed 0.9 factor, reusing the previous guard. Because snapping rules can elongate the chunk, the second pass can contain even more characters (and tokens) than the first, yet the loop exits after MAX_VALIDATION_ATTEMPTS, returning an oversized chunk that breaks the configured token ceiling downstream.
The actual result, in rare cases (but some) is that, even though the limit is 350 tokens, we first try a chunk that has 420 tokens, and the 2nd attempt instead of going down, goes UP to 489 - which blows up past 512 token limit.
Comments
Comment #4
nikro commentedLe fix:
- After each failed validation we measure the actual density (tokens / characters) of the riskiest chunk, apply the existing variance buffer, and promote that value to the new guard density.
- We rebuild the chunking plan from this guard (character budget, overlap, stride) and scale the next character budget proportionally to the ratio effective_limit / tokens, ensuring the immediate retry targets a chunk size safely under the token limit.
- If the proportional scale does not shrink the budget, we fall back to the 0.9 reduction safeguard. The retry loop still validates only one chunk per attempt and does not perform additional sampling beyond the original probes.
Ran this on >5k docs, no issues.