Problem/Motivation

When using the migration_lookup process plugin and providing multiple migrations to search in, the plugin may exit early due to NULL destination ids from skipped rows being returned by the SqlMap plugin.

Example setup:

  • A migration lookup using multiple migrations

    -
      plugin: migration_lookup
      migration:
        - node_by_x
        - node_by_y
        - node_by_z
      no_stub: true
      source: nid
    
  • Source ids are valid for multiple of these migrations
  • Each migration is only responsible for a subset of its source rows, so these migrations have some ignored rows in the map
    _valid:
      -
        plugin: static_map
        map:
          12: true
          123: true 
          1234: true
        default_value: ''
        source: uid
      -
        plugin: skip_on_empty
        method: row
    

This code in MigrationLookup::transform is perfectly fine:

      // Break out of the loop as soon as a destination ID is found.
      if ($destination_ids = $migration
        ->getIdMap()
        ->lookupDestinationId($source_id_values[$migration_id])) {
        break;
      }

However, in a setup where a migration may skip rows, the Sql::lookupDestinationIds method will treat skipped rows as if they were imported, but with NULL destination id(s), breaking any further lookups for migrations that may have actually imported the row.

Proposed resolution

Only return rows from the migrate map that were handled by the migration (= are not ignored) in Sql::lookupDestinationIds.

As a workaround, we've been chaining lookups for each migration individually, using a custom processor to check for empty destination ids.

Remaining tasks

Review, add tests.
Optionally decide whether other map row statuses should be excluded as well (e.g. failed rows).

Comments

ckaotik created an issue. See original summary.

ckaotik’s picture

Assigned: ckaotik » Unassigned
Status: Active » Needs review
StatusFileSize
new852 bytes

Status: Needs review » Needs work

The last submitted patch, 2: drupal-migration_lookup_ignored-2954192-2.patch, failed testing. View results

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

jcnventura’s picture

Status: Needs work » Closed (duplicate)