i will debug this a bit further to see what is breaking down but thought i would mention now in case anyone knows the answer already.

i have a lot of H1s with the odd apostrophe character in them (’) - i am not even sure how to get that from my keyboard; so i simply copied/pasted from the html doc i am trying to import. when it get's imported it shows as � (my browser shows this as a black diamond with a ? inside it).

i use the import_html search/replace feature and tried to replace the ’ with a ' and it does not find anything.

Comments

liquidcms’s picture

liquidcms commented

windows character map seem to think it is a U+02BC: Modifier Letter Apostrophe

i have tried using regex but not sure how to specify this so no luck so far.

liquidcms’s picture

liquidcms commented

a little digging in to unicode and seems this should work: /\x{02BC}/u

but fails silently giving:

Processing 1 files for import. Using default configuration settings.
Imported 0 items

liquidcms’s picture

liquidcms commented

in the end i simply hacked in this line:

function parse_in_xml_string($xmlsource, $do_tidy = 0) {
  // pre clean
  $xmlsource = str_replace('’', "'", $xmlsource);

i am sure there is an easier way or else possibly should be a hook here to allow custom precleaning functions?

liquidcms’s picture

liquidcms commented

oops, my bad.. there is a hook for this already

function mymodule_import_html_preprocess($profile, $xmlsource) {
  // pre clean
  $xmlsource = str_replace('’', "'", $xmlsource);

  return $xmlsource;
}
dman’s picture

dman commented

Funny characters were the bane of my life.
Swithing between 'unknown' encoding then trying to standardize on UTF8 or XML character encoding and then doing it again every time I passed it in and out of the XML DOM and foreign scripts and then trying to make the eventual HTML not horrible to work with in plaintext editors later (with like everything doubtful char-encoded) was a terrible balance to keep.

My tendency was towards html character entity encoding everywhere (which was at least consistent) though I certainly would agree that we can replace ’ with ' rather than its html-entity equivalent. Though type purists disagree strongly.

There is probably a catch-all preprocess routine somewhere with a very small list of obvious replacements that this one (?) could probably be added to. A bunch of common-sense stuff like this is in the heuristics somewhere - though I preferred to expose hooks - like you found - than make best-calls about what is probably expected.

dman’s picture

dman commented
Status: Active » Closed (works as designed)

The preprocess hook is indeed the one to use.