When running a standard D7 to D8 node migration with media_wysiwyg tokens that have line breaks inside them, the REGEX pattern matching results in the following warning:
[warning] Message: The following media_wysiwyg token in node NODE_ID does not have valid JSON: ...

CommentFileSizeAuthor
#16 media_migration-parse_tokens_with_linebreaks-3098833-16--test-only.patch1.26 KBhuzooka
#16 media_migration-parse_tokens_with_linebreaks-3098833-16--complete.patch2.51 KBhuzooka
#10 media_migration-invalid_media_wysiwyg_token-3098833-10.patch738 byteshuzooka
#5 media_migration-invalid_media_wysiwyg_token-3098833-5.patch1.23 KBhuzooka
#2 3098833-media_migration-json-line-break-valid-2.patch793 bytesfirewaller

Comments

firewaller created an issue. See original summary.

firewaller’s picture

firewaller commented
Version: 8.x-1.0-alpha1 » 8.x-1.x-dev
Status: Active » Needs review
StatusFileSize
new 3098833-media_migration-json-line-break-valid-2.patch793 bytes

Attached is a quick patch that replaces line breaks within the JSON with a single space to make the JSON valid.

juampynr’s picture

juampynr commented
Status: Needs review » Needs work

Thanks @firewaller!

Looks like the patch needs some work so it applies:

juampy@carboncete:/tmp/media_migration:8.x-1.x$ git apply -v --index 3098833-media_migration-json-line-break-valid-2.patch
Checking patch Plugin/migrate/process/MediaWysiwygFilter.php...
error: Plugin/migrate/process/MediaWysiwygFilter.php: does not exist in index
huzooka’s picture

huzooka
Primary language Hungarian
Location Hungary 🇭🇺🇪🇺
commented
Assigned: Unassigned » huzooka

Another related thing: The 'default' view mode entity embed display is not valid (the plugin derivative does not exist), we have to map default to full.

huzooka’s picture

huzooka
Primary language Hungarian
Location Hungary 🇭🇺🇪🇺
commented
Assigned: huzooka » Unassigned
Status: Needs work » Needs review
StatusFileSize
new media_migration-invalid_media_wysiwyg_token-3098833-5.patch1.23 KB

Addressing #3 and also #4.

huzooka’s picture

huzooka
Primary language Hungarian
Location Hungary 🇭🇺🇪🇺
commented
Related issues: +#3096087: Media WYSIWYG token matching conflicts with other token types
juampynr’s picture

juampynr commented
Status: Needs review » Needs work

Woops, another conflict:

juampy@carboncete:/tmp/media_migration:8.x-1.x$ curl https://www.drupal.org/files/issues/2020-04-16/media_migration-invalid_media_wysiwyg_token-3098833-5.patch | git apply -v --index
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1264  100  1264    0     0   4899      0 --:--:-- --:--:-- --:--:--  4880
Checking patch src/Plugin/migrate/process/MediaWysiwygFilter.php...
error: while searching for:
    $value['value'] = preg_replace_callback($pattern, function ($matches) use ($replacement_template, $messenger, $nid) {
      $decoder = new JsonDecode(TRUE);
      try {
        $tag_info = $decoder->decode($matches['tag_info'], JsonEncoder::FORMAT);
        return sprintf($replacement_template, $tag_info['view_mode'], $tag_info['fid']);
      }
      catch (NotEncodableValueException $e) {
        // There was an error decoding the JSON. Remove code.

error: patch failed: src/Plugin/migrate/process/MediaWysiwygFilter.php:60
error: src/Plugin/migrate/process/MediaWysiwygFilter.php: patch does not apply
huzooka’s picture

huzooka
Primary language Hungarian
Location Hungary 🇭🇺🇪🇺
commented
Assigned: Unassigned » huzooka
huzooka’s picture

huzooka
Primary language Hungarian
Location Hungary 🇭🇺🇪🇺
commented
Issue tags: +Needs reroll
huzooka’s picture

huzooka
Primary language Hungarian
Location Hungary 🇭🇺🇪🇺
commented
Assigned: huzooka » Unassigned
Status: Needs work » Needs review
Issue tags: -Needs reroll
StatusFileSize
new media_migration-invalid_media_wysiwyg_token-3098833-10.patch738 bytes
firewaller’s picture

firewaller commented
Status: Needs review » Reviewed & tested by the community

#10 works for me

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Issue tags: +Needs reroll
huzooka’s picture

huzooka
Primary language Hungarian
Location Hungary 🇭🇺🇪🇺
commented
Status: Reviewed & tested by the community » Postponed (maintainer needs more info)
Issue tags: +Needs steps to reproduce

Hi @firewaller,

Could you please sharewith us an example string that fails?

firewaller’s picture

firewaller commented

@huzooka it looks like the regex picks this up correctly, but the decoded JSON breaks when line breaks are present in the token. I don't know how this would happen, but that's the way it is on some D7 nodes so I'm assuming WYSIWYG formatted it that way.

[[{ "fid":"123456","view_mode":"default","type":"media","attributes":{ "class":"xxxxx-xxxxxxx
xxxx-xxxxxxx xx-xxxxx-xxxxx-xxx" ,"alt":"xxx xxxxxxxx xxxx" ,"title":"xxx xxxxxxxx xxxx" },
"fields":{ "format":"default" ,"alt":"xxx xxxxxxxx
xxxx","field_file_image_alt_text[und][0][value]":"xxx xxxxxxxx xxxx" ,"title":"xxx xxxxxxxx
xxxx","field_file_image_title_text[und][0][value]":"xxx xxxxxxxx xxxx"} }]]
firewaller’s picture

firewaller commented
Status: Postponed (maintainer needs more info) » Active
huzooka’s picture

huzooka
Primary language Hungarian
Location Hungary 🇭🇺🇪🇺
commented
Status: Active » Needs review
Issue tags: -Needs reroll, -Needs steps to reproduce
StatusFileSize
new media_migration-parse_tokens_with_linebreaks-3098833-16--complete.patch2.51 KB
new media_migration-parse_tokens_with_linebreaks-3098833-16--test-only.patch1.26 KB

Status: Needs review » Needs work

The last submitted patch, 16: media_migration-parse_tokens_with_linebreaks-3098833-16--test-only.patch, failed testing. View results

huzooka’s picture

huzooka
Primary language Hungarian
Location Hungary 🇭🇺🇪🇺
commented
Status: Needs work » Needs review
huzooka’s picture

huzooka
Primary language Hungarian
Location Hungary 🇭🇺🇪🇺
commented
Title: Invalid media_wysiwyg token JSON when linebreaks are present » Handle media_wysiwyg token JSON with line breaks
Category: Bug report » Feature request
Status: Needs review » Fixed

  • huzooka committed 795884d on 8.x-1.x 
    Issue #3098833 by huzooka, firewaller: Handle media_wysiwyg token JSON...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.