Scenario

Migrating a D7 list text field to a D8 list text field. The field is a select field so only one value can be selected. Some of the D7 list fields values were never set (it was not required). In D8, the field is now required so I want to set the default value during the migration for any of those fields that never had a value on the D7 site.

The list text field has three possible values ("top", "bottom" and "center") and we want "center" to be the default if nothing was ever set on the D7 site. Here is the migration config for the process plugin.

field_hero_image_position:
  plugin: default_value
  default_value: center
  source: field_banner_image_position

Expected Behavior

The expected behavior is that the migration will migrate over the content from the D7 field to the D8 field if it is set. However, in the event that it was not set, the default_value plugin should be run in which case it would determine the value to be empty and therefore use center as the default value.

How to Reproduce

Migrate a D7 list text field with unset values (I am not sure if this applies to any other field type) to a D8 list text field and try to apply a default value during the migration.

If you remove the source line from the migration configuration. The default_value plugin works as expected and the default is set on the field.

Why it's happening (see code snippets below)

1. MigrateExecutable.php calls the processRow method for each row. This loops through each field and plugin. In this case we have one field but it sets the field as having two plugins (Get and DefaultValue).
1. The Get process plugin runs first, and tries to grab the value but there is no value set, so it sets $value to an empty array.
2. The process then sets the $multiple variable to TRUE. This is important because it controls what happens in the next iteration of the loop.
3. On the second iteration of the loop, the code falls into the first if statement since $multiple variable is now TRUE.
4. Since $value is an empty array, the process skips over the foreach and the DefaultValue process plugin is never called.
5. This causes the default value of any fields that were not set on the D7 site to remain null/empty on the D8 site.

Here is the relevant code from the processRow method in MigrateExecutable.php. You should be able to follow the steps above and see what is happening.

foreach ($plugins as $plugin) {
  $definition = $plugin->getPluginDefinition();
  // Many plugins expect a scalar value but the current value of the
  // pipeline might be multiple scalars (this is set by the previous
  // plugin) and in this case the current value needs to be iterated
  // and each scalar separately transformed.
  if ($multiple && !$definition['handle_multiples']) {
    $new_value = [];
    if (!is_array($value)) {
      throw new MigrateException(sprintf('Pipeline failed at %s plugin for destination %s: %s received instead of an array,', $plugin->getPluginId(), $destination, $value));
    }
    $break = FALSE;
    foreach ($value as $scalar_value) {
      try {
        $new_value[] = $plugin->transform($scalar_value, $this, $row, $destination);
      }
      catch (MigrateSkipProcessException $e) {
        $new_value[] = NULL;
        $break = TRUE;
      }
    }
    $value = $new_value;
    if ($break) {
      break;
    }
  }
  else {
    try {
      $value = $plugin->transform($value, $this, $row, $destination);
    }
    catch (MigrateSkipProcessException $e) {
      $value = NULL;
      break;
    }
    $multiple = $plugin->multiple();
  }
}

Here is the relevant code for the Get process plugin. I think pretty much any field will be returned as an array and will therefore set the $multiple variable to TRUE.

public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    $source = $this->configuration['source'];
    $properties = is_string($source) ? [$source] : $source;
    $return = [];
    foreach ($properties as $property) {
      if ($property || (string) $property === '0') {
        $is_source = TRUE;
        if ($is_source) {
          $return[] = $row->getSourceProperty($property);
        }
      }
      else {
        $return[] = $value;
      }
    }

    if (is_string($source)) {
      $this->multiple = is_array($return[0]); //This is where multiple is set to TRUE
      return $return[0];
    }
    return $return;
  }

The Solution

Since I don't completely understand how the Migration process is working, I don't know the best way to fix this without potentially negatively impacting something else.

There may be an alternative solution with a different process plugin. I am also not sure if this is intended behavior but if it is, then I believe this behavior should be documented more clearly.

Comments

smthomas created an issue. See original summary.

codekarate’s picture

I was able to get this to work by updating the annotation in the DefaultValue.php process plugin to add the line handle_multiples = TRUE.

/**
 * @MigrateProcessPlugin(
 *   id = "default_value"
 * )
 */
class DefaultValue extends ProcessPluginBase {

To

/**
 * @MigrateProcessPlugin(
 *   id = "default_value",
 *   handle_multiples = TRUE
 * )
 */
class DefaultValue extends ProcessPluginBase {

This does not seem to effect any of the other migrations I have running. It seems like it shouldn't cause many issues as the Default Value plugin simply checks if a value is empty or isset, both of which should work if an array is passed into the function.

codekarate’s picture

Attaching a patch of the previous change to set the Default Value process plugin to be able to handle multiples.

codekarate’s picture

Version: 8.6.5 » 8.6.7
codekarate’s picture

Status: Active » Needs review
heddn’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests

Needs work because needs tests. This should be fairly simple to test though.

robpowell’s picture

StatusFileSize
new3.78 KB
new3.06 KB

Let's see if this works. I based this off the following plugins that also handle_multiples:

grep -R "*   handle_multiples = TRUE" ./docroot/core/modules/migrate
./docroot/core/modules/migrate/src/Plugin/migrate/process/Extract.php: *   handle_multiples = TRUE
./docroot/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php: *   handle_multiples = TRUE
./docroot/core/modules/migrate/src/Plugin/migrate/process/Concat.php: *   handle_multiples = TRUE
./docroot/core/modules/migrate/src/Plugin/migrate/process/SkipRowIfNotSet.php: *   handle_multiples = TRUE
./docroot/core/modules/migrate/src/Plugin/migrate/process/Flatten.php: *   handle_multiples = TRUE
./docroot/core/modules/migrate/src/Plugin/migrate/process/Iterator.php: *   handle_multiples = TRUE
./docroot/core/modules/migrate/src/Plugin/migrate/process/ArrayBuild.php: *   handle_multiples = TRUE
robpowell’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 7: interdiff-3027998-7-3.patch, failed testing. View results

quietone’s picture

robpowell’s picture

StatusFileSize
new3.88 KB
new848 bytes

Let's try that again, this time we will set the @coversDefaultClass annotation :).

robpowell’s picture

Status: Needs work » Needs review

The last submitted patch, 11: 3027998-11.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

robpowell’s picture

StatusFileSize
new5.22 KB
new4.78 KB

After reading this comment in the doc block,

The plugin returns a default value if the input value is considered
* empty (NULL, FALSE, 0, '0', an empty string, or an empty array).

I decided to make sure we test each of those conditions. Latest update also includes name changes to be more clear on what each test is asserting.

heddn’s picture

Status: Needs review » Needs work
Issue tags: -Needs tests
jofitz’s picture

Assigned: Unassigned » jofitz

I'll have a go at this (although I may be a little rusty!)

jofitz’s picture

Assigned: jofitz » Unassigned
Status: Needs work » Needs review
StatusFileSize
new3.36 KB
new6.38 KB

Reworked tests into a data provider.

I've opted for a simpler test function and a larger data provider array. The data provider could be smaller with the use of a couple of conditions and default values.

heddn’s picture

Status: Needs review » Needs work
  1. +++ b/core/modules/migrate/tests/src/Unit/process/DefaultValueTest.php
    @@ -0,0 +1,103 @@
    +  public function testDefaultValue($configuration, $value, $expected_value) {
    ...
    +    $this->assertSame($expected_value, $value);
    

    Nit: can we swap the expected_value and value variables? So they match up with assertSame's order.

  2. +++ b/core/modules/migrate/tests/src/Unit/process/DefaultValueTest.php
    @@ -0,0 +1,103 @@
    +   * Tests that default value is returned when Strict is true and value is NULL.
    

    This comment seems off given the data provider.

    We also seem to be missing a few strict scenarios. I see strict zero and null, but nothing else.

jofitz’s picture

Version: 8.6.7 » 8.8.x-dev
Status: Needs work » Needs review
StatusFileSize
new4.39 KB
new3.9 KB
  1. Swapped the order of expected_value and value variables.
  2. Corrected the comment.
  3. Added missing strict scenarios.
heddn’s picture

Status: Needs review » Reviewed & tested by the community

Nice work here. This is one of those examples where the tests are more work then the fix. But now we can be sure this won't surface again. Thanks everyone.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 19: 3027998-19.patch, failed testing. View results

heddn’s picture

Status: Needs work » Reviewed & tested by the community
alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed and pushed 11c46fc2ac to 9.0.x and 1f0cd959e3 to 8.9.x and de971ea5f3 to 8.8.x. Thanks!

backported to 8.8.x because the fix is not disruptive and nice for anyone doing migrations.

  • alexpott committed 11c46fc on 9.0.x
    Issue #3027998 by robpowell, jofitz, codekarate, heddn: Default Value...

  • alexpott committed 1f0cd95 on 8.9.x
    Issue #3027998 by robpowell, jofitz, codekarate, heddn: Default Value...

  • alexpott committed de971ea on 8.8.x
    Issue #3027998 by robpowell, jofitz, codekarate, heddn: Default Value...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.