Migrate Process: Importing Multiple Paragraphs

I got it to work!

And I'm going to document the process here in its entirety so that people who have the same task can copy my work. It's non trivial.

This will take a couple hours to write up, just wanted to post this quick update before someone replied to my message and tried to tackle the problem.

OK, so here's how to attach multiple paragraphs. In my context, I'm trying to use Migrate Source CSV to import multiple paragraphs, then attach those paragraphs to content.

In my example I have 4 paragraphs (paragraph_test_1.csv):

id,number_1,text_1
11,111,Text 11
12,112,Text 12
21,121,Text 21
22,122,Text 22

I have two nodes I want to attach those paragraphs to. Two paragraphs are to be added to each node (content_test_1.csv):

id,title,body,paragraph_test_1
1,Title 1,Body 1,"11,12"
2,Title 2,Body 2,"21,22"

There are a number of issues to address. Firstly it's critical that the EntityReference migrate destination patch be applied against Entity Reference Revisions, so that importing paragraphs will store their target_id and target_revision_id in the map: https://www.drupal.org/node/2809793

Here's my YAML file for importing paragraphs. It's fairly simple:

uuid: 88b8d4b8-0887-4dbf-884e-1d2483c3f2c1
id: import_paragraph_test_1
label: Import Paragraph Test 1

source:
  plugin: csv
  path: 'private://import_csv/paragraph_test_1.csv'
  header_row_count: 1
  keys:
    - id
process:
  field_number_1: number_1
  field_text_1: text_1
  type:
    plugin: default_value
    default_value: paragraph_test_1
destination:
  plugin: entity_reference_revisions:paragraph

After running this against your content, pop into sql and check the map tables for your import. There should be destid1 and destid2 fields in the table, your target entity id and target entity revision id.

So, here's where things get messy. The core problem here is that it's not possible to refer to the target_id and target_revision_id subfields using field_paragraph_test_1/target_id and field_paragraph_test_1/target_revision_id. That works fine for a single paragraph, but process is unable to iterate against the paragraphs properly.

It's possible to chain the process plugins like so: explode > migration > extract . However the values that come out of that are all wrong:

Array
(
    [target_id] => Array
        (
            [0] => 54
            [1] => 56
        )

    [target_revision_id] => Array
        (
            [0] => 55
            [1] => 57
        )

)

It's not possible to iterate through paragraphs when the target_id and target_revision_ids are lacking keys. As I said in my original post, you need to be able to iterate through the paragraphs as a list.

So in theory I'd like to run my process migrations in this chain: explode > iterator (migration > extract)

There's a big problem here. iterator expects an keyed array as input. But expode is just returning a series of strings. We need to somehow wrap these strings in a keyed array so that iterator can work against them.

Behold, the custom key_wrapper plugin:

<?php
namespace Drupal\migrate_process_key_wrapper\Plugin\migrate\process;

use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\Row;
 
/**
 * Determine the most recent entity revision id given an entity id
 *
 * @MigrateProcessPlugin(
 *   id = "key_wrapper"
 * )
 */
class MigrateProcessKeyWrapper extends ProcessPluginBase {
 
  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    $new_value = array(
        'key_wrapper' => $value,
    );
    $value = $new_value;
    return $value;
  }
 
}

This takes the string "11", and changes it to array('key'=>11). And now the iterator can do it's magic. Here's the YAML for my node import:

uuid: 88b8d4b8-0887-4dbf-884e-1d2483c3f2c0
id: import_content_test_1
label: Import Content Test 1

source:
  plugin: csv
  path: 'private://import_csv/content_test_1.csv'
  header_row_count: 1
  keys:
    - id
  column_names:
    -
      id: id
    - 
      title: title
    - 
      body: body
    -
      paragraph_test_1: paragraph_test_1
process:
  title: title
  body: body
  field_paragraph_test_1:
    - 
      plugin: explode
      delimiter: ,
      source: paragraph_test_1
    -
      plugin: key_wrapper
    -
      plugin: iterator
      process:
        target_id:
          -
            plugin: migration
            migration: import_paragraph_test_1
            source: key_wrapper
            no_stub: true
          -
            plugin: extract
            index:
              - 0
        target_revision_id:
          -
            plugin: migration
            migration: import_paragraph_test_1
            source: key_wrapper
            no_stub: true
          -
            plugin: extract
            index:
              - 1
  type:
    plugin: default_value
    default_value: content_test_1
destination:
  plugin: entity:node

And bam! The chain works. (source) "11,12" becomes (explode) 11, becomes array('key_wrapper' => 11), which is chewed up by (iterator), passed to migration to become array('target_id'=>54,'target_revision_id'=>56), and then (extract) to be just 54.

Having written this out, this still doesn't feel quite right. It feels like explode>migration just isn't returning that correct structure, and that if somehow the combo returned

Array
(
    [0] => Array
        (
            [target_id] => 54
            [target_revision_id] => 56
        )

    [1] => Array
        (
            [target_id] => 55
            [target_revision_id] => 57
        )

)

it would work properly.

Anyways, it was important that I documented this. It's likely I'll come back and revise this with a better solution...

I'm making some real progress here. It's possible to do this without added plugins, but it requires several patches.

#1: EntityReference migrate destination Patch #15: https://www.drupal.org/node/2809793
#2: Migrate SQL Map doesn't get array keys for compound keys Patch #14: https://www.drupal.org/node/2810907
#3: Scalar to Array Migration returns Null: https://www.drupal.org/node/2767643 (Either of my #1 or #3 patches in comments #10 and #12).

Then the simple YML file for import will work:

uuid: 88b8d4b8-0887-4dbf-884e-1d2483c3f2c0
id: import_content_test_1
label: Import Content Test 1

source:
  plugin: csv
  path: 'private://import_csv/content_test_1.csv'
  header_row_count: 1
  keys:
    - id
process:
  title: title
  body: body
  field_paragraph_test_1:
    -
      plugin: explode
      source: paragraph_test_1
      delimiter: ,
    -
      plugin: migration
      migration: import_paragraph_test_1
      no_stub: true
    -
      plugin: iterator
      process:
        target_id: id
        target_revision_id: revision_id
  type:
    plugin: default_value
    default_value: content_test_1
destination:
  plugin: entity:node

The plugins are almost perfect, and the migration plugin returns this structure to save into the node:

    [destination:protected] => Array
        (
            [title] => Title 2
            [body] => Body 2
            [field_paragraph_test_1] => Array
                (
                    [0] => Array
                        (
                            [id] => 82
                            [revision_id] => 84
                        )

                    [1] => Array
                        (
                            [id] => 83
                            [revision_id] => 85
                        )

                )

            [type] => content_test_1
        )

There is a problem here though. The SQL Key patch is returning the keys "id" and "target_id", but paragraph fields expect keys of "target_id" and "target_revision_id". Adding in the simple iterator plugin solves the problem.

Still looking into it. There has to be an even cleaner solution.

@ruloweb: I agree in principle, but at the moment this is stream of conciousness documentation. And there are still changes going on over in #drupal-migrate. Once I settle on how this should actually be solved, I'll post back there. (Shortly!)

@heddn is saying in #drupal-migrate that the SQL patch may be a no-go, partly because of the key mismatch between paragraphs and their references.

Here is a variant of my YML that works without the SQL patch. It still requires the scalar=>array is NULL patch and the Entity Reference Relations patch.

uuid: 88b8d4b8-0887-4dbf-884e-1d2483c3f2c0
id: import_content_test_1
label: Import Content Test 1

source:
  plugin: csv
  path: 'private://import_csv/content_test_1.csv'
  header_row_count: 1
  keys:
    - id
process:
  title: title
  body: body
  field_paragraph_test_1:
    -
      plugin: explode
      source: paragraph_test_1
      delimiter: ,
    -
      plugin: migration
      migration: import_paragraph_test_1
      no_stub: true
    -
      plugin: iterator
      process:
        target_id: '0'
        target_revision_id: '1'
  type:
    plugin: default_value
    default_value: content_test_1
destination:
  plugin: entity:node

Not entirely sure why in the iterator plugin, 0 and 1 have to be in quotes. Otherwise the process plugin discards the data.

@heddn: Agreed. This is sufficiently unmessy for a complex import.

It still requires that migrate patch to function (permitting scalar source keys to map to 2 destination keys) which is a bit concerning.

Nodes point to paragraphs. Paragraphs don't point to nodes - they're their own "atomic" entity.

I'm thinking your nodes.csv would be along the lines of:

main_id,title,body,paragraph_test_1
1,Title 1,Body 1,"1,2"
2,Title 2,Body 2,"3,4"

The main_id field in your paragraphs.csv would be unused then. (It's fine to have it in the csv, but your yml files shouldn't make reference to it.)

The migration yml for your nodes would be similar to the one I posted above in #8:

The explode plugin takes the "1,2" and converts it to "1" and "2" and does each paragraph on its own loop.

The migration plugin converts "1" into a target_id and a target_revision_id (internal to drupal entity reference.) "1" would convert to something like: [0=>123, 1=>124] (The target_id and target_revision_id are assigned when the paragraphs are imported). Ditto for what was once "2".

Finally the iterator plugin assigns the target_id and target_revision_id to the appropriate fields in the entity reference. From there, you should have a paragraph attached to your node.

Good luck! It's complicated at first, cranky if you attempt to reference something that isn't imported yet (always import your paragraphs first!), but powerful once it's working.

At the end of the day, the paragraph id's need to be connected at the node level. That's just how paragraphs work in Drupal.

I'm not sure if there's a great way of doing this other than to write your own process plugin. "Find and iterate through all Paragraphs that have this Store ID"

Alternately, if it's a one time import, and you can't get the CSVs in any other format, I might consider writing a pre-processor to "fix" the CSVs so they're in a format Drupal Expects. Read the hours csv and store the data, read the stores csv, and add a new column based on the data.

Alternately, you might consider changing your data model, so that the Store Hours are their own node object, and they externally reference your stores (as your CSVs do). This seems really backward to me though.

This is a closed thread, you might be asking in the wrong place. :)