migrate-import fails with field_instances, upgrade_d6_field_instance

Confirmed! another sighting on my end. Doing a d6 to d8 migration. Using recipe https://drupalize.me/blog/201606/custom-drupal-drupal-migrations-migrate...

Something that might be of interest is that the field instances on d7 to d8 migration .. has field instances instances migrating but not all.

Same behaviour when migrating from D6 to 8.1.6
Migration yaml file: https://www.pastery.net/tdjbnx/

Trying to provide readers with additional information about this error.

✝ multi-vlad ✝ /var/www/site/vlad_aux/d7d8-dev $ drush8 mi upgrade_d7_field_instance --limit=1
Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_field_instance'                                [status]

✝ multi-vlad ✝ /var/www/site/vlad_aux/d6d8-dev $ drush8 mi upgrade_d6_field_instance --limit=1
Processed 1 item (0 created, 0 updated, 1 failed, 0 ignored) - done with 'upgrade_d6_field_instance'                                [status]
✝ multi-vlad ✝ /var/www/site/vlad_aux/d6d8-dev $ drush8 mmsg upgrade_d6_field_instance
 source_ids_hash                                                   level   message                   
 32579df3f118e54f51a88b991e5df96bca25b4804320c5e015ee3f771383e9e9  1       Input should be an array

It's currently a big PITA to do debug_backtrace and log the result array for further viewing in a dashboard.
I usually store the results of debug_backtrace() to a $log variable and through d.o/project/object_log I am able to see the code flow in a dashbord. Attempts to store that crucial information is currently crashing to : exception 'LogicException' with message The database connection is not serializable.
I was trying to add debug_backtrace on function drush_migrate_tools_migrate_import

function drush_migrate_tools_migrate_import($migration_names = '') {
  // Take it one group at a time, importing the migrations within each group.
  foreach ($migrations as $group_id => $migration_list) {
    array_walk($migration_list, '_drush_migrate_tools_execute_migration', $options);
  }
  $log = debug_backtrace();
  object_log('migrate-import', $log);
}

It's related to several issues in drupal core already..

Logic Exception with preview button for block-comment
https://www.drupal.org/node/2620346
Serializing the database connection is dangerous and error-prone, make it unserializable again
https://www.drupal.org/node/2463321

Having said that I am sorry If some of my information is wrong..
It looks like :
"upgrade_d6_field_instance" is an instance of /core/modules/field/src/Plugin/migrate/source/d6/FieldInstance.php
"upgrade_d7_field_instance" is an instance of /core/modules/field/src/Plugin/migrate/source/d7/FieldInstance.php
I printed out the $row objects at the prepare method.. the printouts are at the attached files:
d6tod8.error_.txt
d7tod8.success.txt

More information about the process :
Attached files describes the instances :
migrate_plus.migration.upgrade_d6_field_instance.yml
migrate_plus.migration.upgrade_d7_field_instance.yml

Triangulating error message and d6 field instance yml config file, I've determine the process fails at
/core/modules/migrate/src/Plugin/migrate/process/Extract.php
Attached is the log of Extract failure
d6tod8.migrate-plugin-migrate-process-extract.txt
Brought about by code :

class Extract extends ProcessPluginBase {

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    drush_print_r([
      'value' => $value,
      'row' => $row,
    ]);
    if (!is_array($value)) {
      drush_print_r("went to line throw new MigrateException('Input should be an array.');");
      throw new MigrateException('Input should be an array.');
    }
    $new_value = NestedArray::getValue($value, $this->configuration['index'], $key_exists);
    if (!$key_exists) {
      if (isset($this->configuration['default'])) {
        $new_value = $this->configuration['default'];
      }
      else {
        throw new MigrateException('Array index missing, extraction failed.');
      }
    }
    return $new_value;
  }

}

process extract transform method fails because $value is empty.

Here's the problem:

process:
  field_type_exists:
    -
      plugin: migration
      migration: d6_field
      source:
        - field_name

That should be migration: upgrade_d6_field - migrate_upgrade did not properly rewrite the reference there, a bug that was fixed in #2759507: Recursion to apply prefix does not handle arrays.

You'll need to either regenerate your migrations with --configure-only, or edit them to add the upgrade_ prefix to any migration: references lacking the prefix.

Actually, it was #2709349: --configure-only should recurse while adding the upgrade_ prefix in this case...

Confirmed! I've updated the suite of migrate modules to the latest dev snapshots. Re-ran the process and field instance migration for d6 to d8 seems to be in play!

Thanks Mike!

Hi Mike

I have use your solution but it do not work for me. Can you please tell me exactly what i should do.
Thanks in advance.

I believe this bug came back between 8.3.1 and 8.3.2

After having done a migrate-upgrade --configure-only :
$ drush mi --group=migrate_drupal_6

[...]
Processed 23 items (0 created, 0 updated, 23 failed, 0 ignored) - done with 'upgrade_d6_field_instance'
[...]

$ drush mmsg upgrade_d6_field_instance

437b6748d87bccf68011f56ed60846da28c181550426db8debe85b5bcd846bf4 1 Input should be an array.
d4e8ce12c55462a7fe832c1122a7378015c266aa4814699c8cd8cb107c2ba41d 1 Input should be an array.
8b6147334b9592997c0905bbf988405afe18408c5faf6870f1b7f3e231657d9d 1 Input should be an array.
88d6a25b77717209da095c53f9ae198df5613387ada16d4b235e39eb3f9c01c0 1 Input should be an array.
c112b47c7f1586dc3b22f6e9ae46cb3ba093aa7b39ac0714ecc4b316b7cdf63e 1 Input should be an array.
[... x 23]

That was not an issue with 8.3.1

Call Stack:
    0.0002     235728   1. {main}() /usr/local/bin/drush:0
    0.0500     864160   2. require('phar:///usr/local/bin/drush/drush') /usr/local/bin/drush:10
    0.0513     929920   3. drush_startup() phar:///usr/local/bin/drush/drush:114
    0.0520     931544   4. drush_run_main() phar:///usr/local/bin/drush/includes/startup.inc:365
    0.0548    1104952   5. drush_main() phar:///usr/local/bin/drush/includes/startup.inc:458
    1.1360   12234600   6. Drush\Boot\BaseBoot->bootstrap_and_dispatch() phar:///usr/local/bin/drush/includes/preflight.inc:66
    3.1722   28425328   7. drush_dispatch() phar:///usr/local/bin/drush/lib/Drush/Boot/BaseBoot.php:67
    3.8342   31665512   8. call_user_func_array:{phar:///usr/local/bin/drush/includes/command.inc:199}() phar:///usr/local/bin/drush/includes/command.inc:199
    3.8342   31666032   9. drush_command() phar:///usr/local/bin/drush/includes/command.inc:199
    3.8351   31671248  10. _drush_invoke_hooks() phar:///usr/local/bin/drush/includes/command.inc:231
    3.8488   31991960  11. call_user_func_array:{phar:///usr/local/bin/drush/includes/command.inc:422}() phar:///usr/local/bin/drush/includes/command.inc:422
    3.8488   31992400  12. drush_migrate_tools_migrate_import() phar:///usr/local/bin/drush/includes/command.inc:422
    4.3763   36823624  13. array_walk() /modules/migrate/migrate_tools/migrate_tools.drush.inc:248
    4.3763   36824184  14. _drush_migrate_tools_execute_migration() /modules/migrate/migrate_tools/migrate_tools.drush.inc:248
    4.4663   37434584  15. drush_op() /modules/migrate/migrate_tools/migrate_tools.drush.inc:282
    4.4664   37435096  16. drush_call_user_func_array() phar:///usr/local/bin/drush/includes/drush.inc:711
    4.4664   37435144  17. call_user_func_array:{phar:///usr/local/bin/drush/includes/drush.inc:720}() phar:///usr/local/bin/drush/includes/drush.inc:720
    4.4664   37437480  18. Drupal\migrate\MigrateExecutable->import() phar:///usr/local/bin/drush/includes/drush.inc:720
    9.8584   40173632  19. Drupal\migrate\MigrateExecutable->processRow() /core/modules/migrate/src/MigrateExecutable.php:203
    9.8637   40183280  20. Drupal\migrate\Plugin\migrate\process\Extract->transform() /core/modules/migrate/src/MigrateExecutable.php:380
    9.8637   40183328  21. xdebug_print_function_stack() /core/modules/migrate/src/Plugin/migrate/process/Extract.php:71

$destination_property = "field_type_exists"
$value = NULL

Sample of $row:

class Drupal\migrate\Row#658 (7) {
  protected $source =>
  array(25) {
    'field_name' =>
    string(27) "field_link_visual_carrousel"
    'type_name' =>
    string(9) "carrousel"
    'weight' =>
    string(2) "-3"
    'label' =>
    string(3) "URL"
    'widget_type' =>
    string(14) "text_textfield"
    'widget_settings' =>
    array(4) {
      'rows' =>
      int(5)
      'size' =>
      string(3) "200"
      'default_value' =>
      array(1) {
        [0] =>
        array(2) {
          'value' =>
          string(0) ""
          '_error_element' =>
          string(59) "default_value_widget][field_link_visual_carrousel][0][value"
        }
      }
      'default_value_php' =>
      NULL
    }
    'display_settings' =>
    array(5) {
      'label' =>
      array(2) {
        'format' =>
        string(5) "above"
        'exclude' =>
        int(0)
      }
      'teaser' =>
      array(2) {
        'format' =>
        string(7) "default"
        'exclude' =>
        int(0)
      }
      'full' =>
      array(2) {
        'format' =>
        string(7) "default"
        'exclude' =>
        int(0)
      }
      [4] =>
      array(2) {
        'format' =>
        string(7) "default"
        'exclude' =>
        int(0)
      }
      'token' =>
      array(2) {
        'format' =>
        string(7) "default"
        'exclude' =>
        int(0)
      }
    }
    'description' =>
    string(3) "foo"
    'widget_module' =>
    string(4) "text"
    'widget_active' =>
    string(1) "1"
    'type' =>
    string(4) "text"
    'global_settings' =>
    array(4) {
      'text_processing' =>
      string(1) "0"
      'max_length' =>
      string(3) "254"
      'allowed_values' =>
      string(0) ""
      'allowed_values_php' =>
      string(0) ""
    }
    'required' =>
    string(1) "1"
    'multiple' =>
    string(1) "0"
    'db_storage' =>
    string(1) "1"
    'module' =>
    string(4) "text"
    'db_columns' =>
    string(126) "a:1:{s:5:"value";a:5:{s:4:"type";s:7:"varchar";s:6:"length";s:3:"254";s:8:"not null";b:0;s:8:"sortable";b:1;s:5:"views";b:1;}}"
    'active' =>
    string(1) "1"
    'locked' =>
    string(1) "0"
    'migrate_map_sourceid1' =>
    string(27) "field_link_visual_carrousel"
    'migrate_map_sourceid2' =>
    string(9) "carrousel"
    'migrate_map_source_row_status' =>
    string(1) "1"
    'key' =>
    string(8) "drupal_6"
    'plugin' =>
    string(17) "d6_field_instance"
    'constants' =>
    array(1) {
      'entity_type' =>
      string(4) "node"
    }
  }
  protected $sourceIds =>
  array(2) {
    'field_name' =>
    array(2) {
      'type' =>
      string(6) "string"
      'alias' =>
      string(4) "cnfi"
    }
    'type_name' =>
    array(1) {
      'type' =>
      string(6) "string"
    }
  }
  protected $destination =>
  array(0) {
  }
  protected $idMap =>
  array(10) {
    'source_ids_hash' =>
    string(64) "a989af07813b8ecf81501722be3a17138ed2ab68fe8a086266838e5ff4fd4a26"
    'sourceid1' =>
    string(27) "field_link_visual_carrousel"
    'sourceid2' =>
    string(9) "carrousel"
    'destid1' =>
    NULL
    'destid2' =>
    NULL
    'destid3' =>
    NULL
    'source_row_status' =>
    string(1) "1"
    'rollback_action' =>
    string(1) "0"
    'last_imported' =>
    string(1) "0"
    'hash' =>
    string(0) ""
  }
  protected $frozen =>
  bool(true)
  protected $rawDestination =>
  array(0) {
  }
  protected $isStub =>
  array(0) {
  }
}

process:
  field_type_exists:
    -
      plugin: migration_lookup
      migration: d6_field
      source:
        - field_name

The same kind of buggy config is generated. Why?

[edit] Using 8.3.1, I get the (correct) migration: upgrade_d6_field

Any other reports of this? I've not see this is a problem with the latest versions of things.

heddn,

I was testing #2853872: Migration for forum and article comments: duplicate comment types and incorrect comment_entity_statistics using 3.0-rc1 version of Migrate Upgrade and experienced this weird behavior:
https://www.drupal.org/node/2853872#comment-12244723

I was debugging this for several hours and banging my head against the wall. Eventually maxocub woke up in Canada and we started to debug this together on #drupal-migrate.

The reason why my drush9 mi upgrade_d7_comment_field skipped the rows was that it was trying to do the lookup from migrate_map_d7_comment_type database table which was empty. The mapping values were populated to migrate_map_upgrade_d7_comment_type.

When I took the latest dev version of Migrate Upgrade, then drush9 mi upgrade_d7_comment_field worked as expected.

The conclusion I came to with maxocub was that Migrate Upgrade 3.0-rc1 does not work with migration_lookup plugin but only with the deprecated migration plugin.

Would it be possible to cut a new release candidate of Migrate Upgrade?

Cheers,
Markus

Done/fixed: https://www.drupal.org/project/migrate_upgrade/releases/8.x-3.0-rc2