Problem/Motivation
Field values extracted by a crawl can contain double-encoded UTF-8. A
correctly-encoded multi-byte character (e.g. an en dash, non-breaking
hyphen, or curly quote) gets re-encoded a second time somewhere in the
fetch/parse pipeline, producing a visibly wrong character (e.g. â
in place of a dash) plus invalid raw control bytes embedded alongside it in
the stored string.
This happens during extraction itself. It's visible directly in a crawl
target's field mapping Preview output, before any node is
created or updated. It isn't a display-only glitch: the corrupted text
contains actual invalid control characters, which can cause external
services that consume the content afterward to reject it outright. We saw
an AI embeddings API return a hard error on any text containing this
corruption.
On one real-world crawl target, roughly 43% of resulting articles (139 of
326) had at least one instance of this in their body content.
Steps to reproduce
- Configure a crawl target against a page containing an em dash, en dash,
non-breaking hyphen, or curly quotes in its body content. - On the target's field mapping page, use Preview against
that page's URL. - Expected: the extracted value renders the punctuation
correctly. - Actual: the extracted value shows a corrupted character
(visuallyâ) in place of the punctuation, and the underlying
string contains invalid control bytes alongside it.
Proposed resolution
Two parts:
- Add an automatic normalization step in the extraction pipeline that
detects and reverses double-encoded UTF-8 in every extracted field value,
for every crawl target, before it's written to a node. This should detect
the double-encoding pattern itself, not match a fixed list of known-bad
characters, so it also covers punctuation not yet seen in testing. - Identify why the fetch/parse step produces bytes that end up
double-encoded in the first place, and fix that at the source so the
corruption stops being generated at all. This may involve how fetched page
content's charset is detected/handled during parsing.
Remaining tasks
- Confirm the exact point in the fetch/parse pipeline where the
double-encoding is introduced. - Add a regression test with a fixture page containing known problem
characters (em dash, en dash, non-breaking hyphen, curly quotes) asserting
correct extraction. - Implement the fix per "Proposed resolution" above.
- Document the fix in the changelog.
Issue fork crwlr-3615876
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #3
abhisekmazumdarComment #4
nikro commentedComment #5
abhisekmazumdarWorth noting the root cause is different from what the issue description
guessed. The initial suspicion was the
crwlr/crawlerlibrary'sown charset-recovery logic misfiring on already-valid UTF-8 during the
initial page parse. That turned out not to be it: a page fetched and
parsed with no field processors attached came out clean every time.
The actual cause is in this module's own
HtmlAbsoluteUrlsfield processor. It re-parses the extracted HTML fragment with
DOMDocument::loadHTML()to resolve relative URLs. That methoddoes its own input-encoding detection independent of the encoding passed to
DOMDocument's constructor, and with no charset declared in thefragment it's given, it silently assumed ISO-8859-1, corrupting any
multi-byte UTF-8 character in the process. Confirmed by isolating each
processor in the pipeline against real crawled content: the value was
still clean after extraction and after the two other configured
processors, and only came out corrupted after
HtmlAbsoluteUrlsran.
Fix has two parts:
<meta charset="utf-8">on the wrapper markupHtmlAbsoluteUrlsparses, whichloadHTML()actuallyrespects. This is the real fix.
FieldMapperService, runafter the full field-processor pipeline, that detects and reverses
double-encoded UTF-8 on any extracted value regardless of cause. This is a
safety net, not a substitute for the fix above: any other processor that
re-parses HTML through a component with its own charset handling is a
fresh opportunity for the same class of bug.
Comment #6
nikro commentedTested - recrawled and issues are gone. I think the fix is solid too. Thanks!
Comment #8
nikro commentedMerged.