Problem/Motivation

When we have something like:

process:
  'body/format':
    plugin: explode
    delimiter: "|"
    source: format
  'body/value': 
    plugin: explode
    delimiter: "|"
    source: value
  'body/summary': 
    plugin: explode
    delimiter: "|"
    source: teaser

in the migration yaml this will give a structure like this in the row destination:

[
  body => [
    'value' =>
      [0] => [
        'Body text 1',
      ]
      [1] => [
        'Body text 2',
      ]
    ]
...
  ]
]

When it should be:

[
  body => [
    [0] => [
      'value' =>
        'Body text 1',
      ]
      ...
    ]
    [1] => [
      'value' =>
        'Body text 2',
      ]
      ...
    ]
  ]
]

All the destinations are configured as multiple because of this bug #2639556: InvalidArgumentException: Placeholders must have a trailing [] if they are to be expanded with an array of values.. The explode process plugin is just an example.

When the destination array is wrong it throws an error

InvalidArgumentException: Placeholders must have a trailing [] if they are to be expanded with an array of values.
and the migration fails for that row.

Proposed resolution

Make Drupal\migrate\Row::setDestinationProperty handle this special case by setting the parents correctly for
NestedArray::setValue

https://api.drupal.org/api/drupal/core!modules!migrate!src!Row.php/funct...

Remaining tasks

- Write tests.

User interface changes

API changes

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

rodrigoaguilera created an issue. See original summary.

rodrigoaguilera’s picture

Status: Needs review » Needs work

The last submitted patch, 2: 2789125-subproperties-with-multiple-process-plugin-2.patch, failed testing.

rodrigoaguilera’s picture

rodrigoaguilera’s picture

I use composer and I made the patch for drupal/core instead of drupal/drupal

Status: Needs review » Needs work

The last submitted patch, 5: 2789125-subproperties-with-multiple-process-plugin-5.patch, failed testing.

rodrigoaguilera’s picture

Ok Writing the test first it becomes clear that Row::getDestinationProperty() needs to altered too.

In the meantime here is the patch with the test only.

Status: Needs review » Needs work
youfei.sun’s picture

I think I'm caught by the exact same error, but is this patch working correctly now?

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mikeryan’s picture

Wouldn't the example in https://www.drupal.org/node/2639556#comment-11981287 address the real-world use case behind the constructed example in the issue summary?

gmaxwelled’s picture

How would the solution in #11 work? Are you suggesting something like the following?

process:
  'body':
    - 
        plugin: explode
        delimiter: "|"
        source: format
    -
       plugin: iterator
       process:
           format: [value]

If so, what goes in place of [value]? I can reference a specific item in an array using the extract plugin, but is it possible to access the value of an iteration?

edit: Also, how would we pass the array to the iterator plugin? In the above example, it attempts to run the iterator plugin on each element. I'm confused as to how we get the array to feed to the iterator plugin in the first place. I'm reading data from a CSV.

StryKaizer’s picture

Status: Postponed (maintainer needs more info) » Needs review

Patch from #5 fixes my issue (I was working with multivalue daterange fields, trying to set both value and end_value).

Thanks!

PS: suggestion from #11 did not work for me. It looks like this is a bug to me

mikeryan’s picture

Assigned: Unassigned » mikeryan
mikeryan’s picture

Assigned: mikeryan » Unassigned
Status: Needs review » Needs work
Issue tags: -Needs tests

The patch in #5 fails tests, that needs to be fixed. The patch in #7 is a test only - what we need in the next iteration here is a complete patch (fix and test) that passes the tests, and a test-only patch that fails the tests.

jofitz’s picture

Version: 8.3.x-dev » 8.4.x-dev
Status: Needs work » Needs review
FileSize
2.59 KB
1.81 KB
3.79 KB

I have re-uploaded the test-only patch from #7 and extended the fix from #5.

The last submitted patch, 16: 2789125-16-test_only.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 16: 2789125-16-complete.patch, failed testing.

jofitz’s picture

Status: Needs review » Needs work

The last submitted patch, 19: 2789125-19.patch, failed testing.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

dmurphy1’s picture

In response to the suggestion in comment #11 and the follow-up question in comment #12, I was able to work around this issue by following this example: https://gist.github.com/rocketeerbkw/239219a6493c2adee34fb7ceb5af2072

To summarize here for posterity, that gist suggests using prepare row to process your links into the format:

$links = [
  [
    'title' => 'Link 1',
    'uri' => 'URI 1',
  ],
  [
    'title' => 'Link 2',
    'uri' => 'URI 2',
  ],
];

$row->setSourceProperty('links', $links);

And then use the iterator plugin (now deprecated in favor of the SubProcess plugin that works in the same way) as follows:

field_links:
  plugin: iterator
  source: links
  process:
    title: title
    uri: uri

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

safetypin’s picture

@dmurphy1 How does that format translate into the body field? Using a configuration like this doesn't seem to migrate anything:

  body:
    plugin: iterator
    source: body
    process:
      value: value
      format: format
safetypin’s picture

Turns out I was looking at a node that had an empty body field, so it turns out that the pattern in #23 works for me. For the body field, this works:

  body:
    plugin: iterator
    source: body
    process:
      value: value
      format:
        plugin: default_value
        default_value: advanced_html

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

adevms’s picture

The patch works nicely with migrating multivalue link fields but only for uri and title values. Not for attributes for ex. Any ideas as how to make this work?

In the process plugin is have:

 'field_link/uri': link_url
 'field_link/title': link_text

This works nicely for multiple entries.

The following does not work:

 'field_link/options/attributes/target':
    source: link_target
    plugin: static_map
    map:
      'false': '_self'
      'true': '_blank'
    default_value: '_blank'

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

danflanagan8’s picture

Here's a way to do this that I came up with using Migrate Sandbox. Let's say this is your source.

Array
(
    [format] => format 1|format 2|format 3
    [value] => This is the first value|Here is the second value|Third value now
    [teaser] => Teaser 1|Teaser 2|Teaser 3
)

The process pipeline below uses the transpose plugin from migrate_plus.

format_array:
  plugin: explode
  delimiter: "|"
  source: format
value_array: 
  plugin: explode
  delimiter: "|"
  source: value
teaser_array: 
  plugin: explode
  delimiter: "|"
  source: teaser
body_not_associative:
  plugin: transpose
  source:
    - '@format_array'
    - '@value_array'
    - '@teaser_array'
body:
  plugin: sub_process
  source: '@body_not_associative'
  process:
    format:
      plugin: extract
      source:
        - 0
      index:
        - 0
    value:
      plugin: extract
      source:
        - 1
      index:
        - 0
    teaser:
      plugin: extract
      source:
        - 2
      index:
        - 0

Which results in

Array
(
  [body] => Array
        (
            [0] => Array
                (
                    [format] => format 1
                    [value] => This is the first value
                    [teaser] => Teaser 1
                )

            [1] => Array
                (
                    [format] => format 2
                    [value] => Here is the second value
                    [teaser] => Teaser 2
                )

            [2] => Array
                (
                    [format] => format 3
                    [value] => Third value now
                    [teaser] => Teaser 3
                )

        )
)

I personally think it's not quite right to consider this a bug since the source data is extremely different from any source data that one would expect when migrating versions of Drupal. It's a challenge, not a bug!

benjifisher’s picture

I agree with #34. The transpose process plugin is designed for this problem. Now that it is available, @MikeRyan's initial response (#11 on this issue) is right. We should use the sub_process plugin (previously named iterator) here.

Untested, but I think it can be simplified: something like this:

body:
  - plugin: transpose
    source:
      - '@format_array'
      - '@value_array'
      - '@teaser_array'
  - plugin: sub_process
    process:
      format: '0'
      value: '1'
      teaser: '2'
benjifisher’s picture

The issue summary quotes the error message:

InvalidArgumentException: Placeholders must have a trailing [] if they are to be expanded with an array of values.

That is not very helpful. Maybe we can detect this situation and provide a better message before sending it t the database layer.

If we can do that, I am not sure whether we should do it in a new issue or repurpose this one.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.