Problem/Motivation

The "sub_process" migrate plugin iterates over a set of values, but assumes any value item is an array, trying to use the "+" operator with the source values to merge both values.
When we are using multiple plugins to process a field, this could not be true, for example when we use the explode plugin before the sub_process plugin.

Steps to reproduce

Having a migration with a field, with a comma separated values as field value, we have the following process configuration:

  field_bar:
    -
      plugin: explode
      delimiter: ','
      source: bar
    -
      plugin: sub_process
      include_source: true
      process:
        label:
          plugin: concat
          source:
            - source/foo
            - '0'

When running the migration (in our case executed by drush), we can see the following error:

[error]  Error: Unsupported operand types in Drupal\migrate\Plugin\migrate\process\SubProcess->transform() (line 211 of /var/www/html/web/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php) #0 /var/www/html/web/core/modules/migrate/src/MigrateExecutable.php(394): Drupal\migrate\Plugin\migrate\process\SubProcess->transform(Array, Object(Drupal\migrate_tools\MigrateExecutable), Object(Drupal\migrate\Row), 'field_model_exa...')
#1 /var/www/html/web/core/modules/migrate/src/MigrateExecutable.php(203): Drupal\migrate\MigrateExecutable->processRow(Object(Drupal\migrate\Row))
#2 /var/www/html/vendor/drush/drush/includes/drush.inc(206): Drupal\migrate\MigrateExecutable->import()
#3 /var/www/html/vendor/drush/drush/includes/drush.inc(197): drush_call_user_func_array(Array, Array)
#4 /var/www/html/web/modules/contrib/migrate_tools/src/Commands/MigrateToolsCommands.php(846): drush_op(Array)
#5 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->executeMigration(Object(Drupal\migrate\Plugin\Migration), 'FOO', Array)
#6 /var/www/html/web/modules/contrib/migrate_tools/src/Commands/MigrateToolsCommands.php(319): array_walk(Array, Array, Array)
#7 [internal function]: Drupal\migrate_tools\Commands\MigrateToolsCommands->import('FOO', Array)
#8 /var/www/html/vendor/consolidation/annotated-command/src/CommandProcessor.php(257): call_user_func_array(Array, Array)
#9 /var/www/html/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
#10 /var/www/html/vendor/consolidation/annotated-command/src/CommandProcessor.php(176): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#11 /var/www/html/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(302): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#12 /var/www/html/vendor/symfony/console/Command/Command.php(255): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /var/www/html/vendor/symfony/console/Application.php(1023): Symfony\Component\Console\Command\Command->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /var/www/html/vendor/symfony/console/Application.php(271): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /var/www/html/vendor/symfony/console/Application.php(147): Symfony\Component\Console\Application->doRun(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /var/www/html/vendor/drush/drush/src/Runtime/Runtime.php(118): Symfony\Component\Console\Application->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /var/www/html/vendor/drush/drush/src/Runtime/Runtime.php(49): Drush\Runtime\Runtime->doRun(Array, Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /var/www/html/vendor/drush/drush/drush.php(72): Drush\Runtime\Runtime->run(Array)
#19 /var/www/html/vendor/drush/drush/includes/preflight.inc(18): require('/var/www/html/v...')
#20 phar:///usr/local/bin/drush/bin/drush.php(141): drush_main()
#21 /usr/local/bin/drush(10): require('phar:///usr/loc...')
#22 {main}.

Proposed resolution

Ensure the value is an array before merging it on the sub_process migrate plugin.

Remaining tasks

Review

User interface changes

None

API changes

None

Data model changes

None

Release notes snippet

CommentFileSizeAuthor
#3 core-3175508-3.patch1.17 KBakalam

Comments

akalam created an issue. See original summary.

akalam’s picture

akalam commented
Issue summary: View changes
akalam’s picture

akalam commented
Status: Active » Needs review
StatusFileSize
new core-3175508-3.patch1.17 KB

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

mikelutz’s picture

mikelutz
He/Him
Primary language English
Location Michigan, USA
commented
Category: Bug report » Feature request
Status: Needs review » Needs work
Issue tags: -migrate

This is not the correct way to use sub_process. The plugin documentation is clear that the input must be an array of arrays. An array of scalers is not a valid input to sub_process, and is not something it's designed to handle.

I would be willing to consider an improvement to sub_process, to handle arrays of scalers, but the feature requires documentation, tests, a change record, etc.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

danflanagan8’s picture

danflanagan8
Primary language English
Location St. Louis, US
commented

FWIW, here's a way to handle the situation in the IS using existing process plugins:

Assume the source row is

foo: my_foo
bar: 'one,two,three,four'

And I define a constant equal to 1.

  constants:
    one: 1

Then this process pipeline works

bar_array:
  plugin: explode
  delimiter: ','
  source: bar
chunked_bar_array:
  plugin: callback
  callable: array_chunk
  unpack_source: true
  source:
    - '@bar_array'
    - constants/one

field_bar:
  plugin: sub_process
  source: '@chunked_bar_array'
  include_source: true
  process:
    label:
      plugin: concat
      source:
        - source/foo
        - '0'

The result is:

Array
(
[field_bar] => Array
        (
            [0] => Array
                (
                    [label] => my_fooone
                )

            [1] => Array
                (
                    [label] => my_footwo
                )

            [2] => Array
                (
                    [label] => my_foothree
                )

            [3] => Array
                (
                    [label] => my_foofour
                )

        )
)

I did this in Migrate Sandbox.

mikelutz’s picture

mikelutz
He/Him
Primary language English
Location Michigan, USA
commented
Status: Needs work » Closed (works as designed)

This shows we don't need to add new features to sub_process anyway, so closing this one.

danflanagan8’s picture

danflanagan8
Primary language English
Location St. Louis, US
commented
Related issues: +#3048464: SubProcess migrate process plugin should throw exception on invalid input

I think that's a good call, @mikelutz. There's even another issue where we add an exception when it's not an array of arrays. It would be weird to immediately reverse course on that.

#3048464: SubProcess migrate process plugin should throw exception on invalid input

danflanagan8’s picture

danflanagan8
Primary language English
Location St. Louis, US
commented
Related issues: +#3314502: Add wrapper process plugin to wrap/unwrap values in arrays

Here's an issue for migrate_plus that offers a cleaner approach than the array_chunk trick described in #8

#3314502: Add wrapper process plugin to wrap/unwrap values in arrays