Automatically fix HTML-encoded (broken) embedded JSON

I tracked down the faulty JSON to the node having " instead of the double quote. This was hard to figure out because it is displayed as a double quote so it looks correct. The newlines were more obvious but did not cause a problem. I suppose it might be possible to have the error message be more specific but I am not sure it is worthwhile.

Below is sample code to correct this problem. If you know the node IDs then you can use $nid = $row->getSourceProperty('nid'); to restrict the fix to just those nodes.

function hook_migrate_MIGRATION_ID_prepare_row(Row $row, MigrateSourceInterface $source, MigrationInterface $migration) {
  $body = $row->getSourceProperty('field_text_body');
  $body[0]['value'] = preg_replace('/"/', '"', $body[0]['value']);
  $row->setSourceProperty('field_text_body', $body);
}

Would it be worth adding this as documentation? What do you think, maintainers?

IMHO it'd be better to incorporate automatic fixes like #4 for "likely JSON malformations".

Based on #4, it looks like we're dealing with a HTML encoding problem? Which actually makes me wonder if there is a deeper rooted problem here.

It'd be really helpful if both of you could share examples of the input that is crashing, because that may or may not confirm my hypothesis 🤓🤞

To clarify:

IMHO it'd be better to […]

… because automatically fixing what we can is better than expecting users futzing around with migration definition tweaks!

Let's assume we move forward here with what I wrote in #6 — which means this becomes a feature request.

OTOH, to then do this, we really should have some concrete examples. Can you provide some, @DamienMcKenna and @trobey? 🙏

Especially, you could also add it to the process plugin's test here: https://git.drupalcode.org/project/media_migration/-/blob/2fe44b9b32/tes...

Perfect, thanks so much @DamienMcKenna! 👍

BTW does this work in Drupal7 Media?

That is reassuring on the one hand and disturbing on the other hand 😅 But hey, of course you want your new site to actually work as expected unlike the old one, so …

Last time I touched this was when I fixed #3098833: Handle media_wysiwyg token JSON with line breaks, but that broken JSON (with linebreaks) could be "fixed" in Drupal 7 after an entity resave performed on Drupal UI.

I just paired with @huzooka to figure whether we could safely automatically fix this kind of broken JSON. Good news 🤓🥳

<?php

$input = <<<INPUT
<p>Foo?</p>
[[{ &quot;fid&quot;:&quot;123456&quot;,&quot;view_mode&quot;:&quot;default&quot;,&quot;type&quot;:&quot;media&quot;,&quot;attributes&quot;:{ &quot;class&quot;:&quot;css-class1
css-class2 css-class3&quot; ,&quot;alt&quot;:&quot;Overridden alt attribute for embed&quot; ,&quot;title&quot;:&quot;Overridden title attribute for embed&quot; },
&quot;fields&quot;:{ &quot;format&quot;:&quot;default&quot; ,&quot;alt&quot;:&quot;Overridden alt attribute for embed&quot;,&quot;field_file_image_alt_text[und][0][value]&quot;:&quot;Media alt attribute&quot; ,&quot;title&quot;:&quot;Overridden title attribute for embed&quot;,&quot;field_file_image_title_text[und][0][value]&quot;:&quot;Media title attribute&quot;} }]]
<p>Bar baz!</p>
INPUT;

$pattern = '/\[\[\s*(?<tag_info>\{.+\})\s*\]\]/sU';
$matches = [];
var_dump(preg_match($pattern, $input, $matches));
var_dump($matches);

$tag_info = $matches['tag_info'];

var_dump('BEFORE');
var_dump(json_decode($tag_info));


function fix(string $parsed_potentially_broken_json): string {
  $r = str_replace("\n", " ", $parsed_potentially_broken_json);
  // This is what \Drupal\Component\Utility\Html::decodeEntities() does.
  return html_entity_decode($r, ENT_QUOTES, 'UTF-8');
}

var_dump('AFTER');
var_dump(json_decode(fix($tag_info)));

See it in action at https://3v4l.org/cD86J.

Added #10 and #17, and converted MediaWysiwygFilterTest to a unit test.

+++ b/src/Plugin/migrate/process/MediaWysiwygFilter.php
@@ -180,7 +181,12 @@ class MediaWysiwygFilter extends EmbedFilterBase implements ConfigurableInterfac
+          $tag_info = $decoder->decode(Html::decodeEntities($matches['tag_info']), JsonEncoder::FORMAT);

Why doesn't this need the

$r = str_replace("\n", " ", $parsed_potentially_broken_json);

anymore?

Ahhh … this pre-existing line:

      // Replace line breaks with a single space for valid JSON.
      $matches['tag_info'] = preg_replace('/\s+/', ' ', $matches['tag_info']);

👍

I don't see why we would not ship this hardening to media_migration users right away — the test coverage proves that it works. So … 🚢!