Problem/Motivation

Field values extracted by a crawl can contain double-encoded UTF-8. A
correctly-encoded multi-byte character (e.g. an en dash, non-breaking
hyphen, or curly quote) gets re-encoded a second time somewhere in the
fetch/parse pipeline, producing a visibly wrong character (e.g. â
in place of a dash) plus invalid raw control bytes embedded alongside it in
the stored string.

This happens during extraction itself. It's visible directly in a crawl
target's field mapping Preview output, before any node is
created or updated. It isn't a display-only glitch: the corrupted text
contains actual invalid control characters, which can cause external
services that consume the content afterward to reject it outright. We saw
an AI embeddings API return a hard error on any text containing this
corruption.

On one real-world crawl target, roughly 43% of resulting articles (139 of
326) had at least one instance of this in their body content.

Steps to reproduce

  1. Configure a crawl target against a page containing an em dash, en dash,
    non-breaking hyphen, or curly quotes in its body content.
  2. On the target's field mapping page, use Preview against
    that page's URL.
  3. Expected: the extracted value renders the punctuation
    correctly.
  4. Actual: the extracted value shows a corrupted character
    (visually â) in place of the punctuation, and the underlying
    string contains invalid control bytes alongside it.

Proposed resolution

Two parts:

  1. Add an automatic normalization step in the extraction pipeline that
    detects and reverses double-encoded UTF-8 in every extracted field value,
    for every crawl target, before it's written to a node. This should detect
    the double-encoding pattern itself, not match a fixed list of known-bad
    characters, so it also covers punctuation not yet seen in testing.
  2. Identify why the fetch/parse step produces bytes that end up
    double-encoded in the first place, and fix that at the source so the
    corruption stops being generated at all. This may involve how fetched page
    content's charset is detected/handled during parsing.

Remaining tasks

  • Confirm the exact point in the fetch/parse pipeline where the
    double-encoding is introduced.
  • Add a regression test with a fixture page containing known problem
    characters (em dash, en dash, non-breaking hyphen, curly quotes) asserting
    correct extraction.
  • Implement the fix per "Proposed resolution" above.
  • Document the fix in the changelog.

Issue fork crwlr-3615876

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

abhisekmazumdar created an issue. See original summary.

abhisekmazumdar’s picture

Assigned: abhisekmazumdar » Unassigned
Status: Active » Needs review
nikro’s picture

Assigned: Unassigned » nikro
abhisekmazumdar’s picture

Worth noting the root cause is different from what the issue description
guessed. The initial suspicion was the crwlr/crawler library's
own charset-recovery logic misfiring on already-valid UTF-8 during the
initial page parse. That turned out not to be it: a page fetched and
parsed with no field processors attached came out clean every time.

The actual cause is in this module's own HtmlAbsoluteUrls
field processor. It re-parses the extracted HTML fragment with
DOMDocument::loadHTML() to resolve relative URLs. That method
does its own input-encoding detection independent of the encoding passed to
DOMDocument's constructor, and with no charset declared in the
fragment it's given, it silently assumed ISO-8859-1, corrupting any
multi-byte UTF-8 character in the process. Confirmed by isolating each
processor in the pipeline against real crawled content: the value was
still clean after extraction and after the two other configured
processors, and only came out corrupted after HtmlAbsoluteUrls
ran.

Fix has two parts:

  • Declare <meta charset="utf-8"> on the wrapper markup
    HtmlAbsoluteUrls parses, which loadHTML() actually
    respects. This is the real fix.
  • Added a generic repair step in FieldMapperService, run
    after the full field-processor pipeline, that detects and reverses
    double-encoded UTF-8 on any extracted value regardless of cause. This is a
    safety net, not a substitute for the fix above: any other processor that
    re-parses HTML through a component with its own charset handling is a
    fresh opportunity for the same class of bug.
nikro’s picture

Assigned: nikro » Unassigned
Status: Needs review » Reviewed & tested by the community

Tested - recrawled and issues are gone. I think the fix is solid too. Thanks!

nikro’s picture

Status: Reviewed & tested by the community » Fixed

Merged.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

  • abhisekmazumdar committed 6647b5f4 on 1.0.x
    Issue #3615876: Repair double-encoded UTF-8 in extracted field values
    

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.