When a migration defines an entity reference and therefor depends upon another 2nd migration which gets run at a later time (thus not enforced via "migration_dependencies: required: ...") the migration will fail if the 2nd migration has Substr as one of the used plugins.

When the migration runs the entity references are created as stubs. The stubs are created creating a Row with only the target_id filled out, the rest of the values are empty. This stub row is then processed by the 2nd migration. When the 2nd migration defines Substr as the plugin for one of the values the Substr plugin will have to process a NULL value and throw MigrateException('The input value must be a string.');

This background-run of the 2nd migration on a stub row to create the entity-reference in the 1st migration effectively fails the whole migration.

The fix is to check for null on $value. If it is NULL then just return $value (NULL) because there won't be anything for Substr to do on NULL.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Jeroen created an issue. See original summary.

Jeroen’s picture

Jeroen’s picture

Jeroen’s picture

Jeroen’s picture

I've added a test too.

benjy’s picture

This stub row is then processed by the 2nd migration. When the 2nd migration defines Substr as the plugin for one of the values the Substr plugin will have to process a NULL value and throw MigrateException('The input value must be a string.');

Is this because the values are null in the source data? I'm not opposed to adding your check to the substr plugin but I think it could also be solved with:

myfield:
  -
    plugin: skip_on_empty
    method: process
    source: mysource_field
  -
    plugin: substr
mikeryan’s picture

Status: Needs review » Postponed (maintainer needs more info)

Sorry Jeroen, now that I look closely at other string-manipulating process plugins such as explode and machine_name, I'm not sure the special NULL handling is appropriate here. The throwing of MigrateException is our established pattern in the !is_string() case - I agree it's harsh to hard-code throwing out the whole row for a missing value in one field (I think it would be better to return NULL and anyone who wanted to throw it out could chain to skip_on_empty with method: row), but at this point changing that behavior would be a backwards-compatibility break. And even if we thought it was worth a BC break, it's something that should be applied to all the process plugins, not just singling out this one. As benjy points out, you do have the option to skip_on_empty upstream.

Your stub problem is more general than substr - basically, in any scenario where stubbing may occur, the migration creating the stub must be prepared for empty values in all fields. I could have sworn we had at some point discussed adding a mechanism for providing default source values for stubs via the migration process plugin, but can't find such an issue for it at the moment. Unless/until we did something like that, I think about all we can do is document that you need to bullet-proof any migrations which are used for stubbing.

mikeryan’s picture

Title: Migrate's Substr plugin makes a stub-migration fail. » Document that migrations used for stubbing need to deal with empty source values
Assigned: Jeroen » Unassigned
Category: Bug report » Task
Status: Postponed (maintainer needs more info) » Active
Issue tags: +Documentation

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

agileadam’s picture

I just ran into this issue related to substr. Unfortunately it cannot be solved by simply adding a skip_if_empty plugin before substr in the chain. If you use skip_if_empty and you are re-running a migration (--update and/or track_changes: true) and a field that previous had values now no longer has values, the field will be ignored during the migration process and the old field data will be kept in the entity. I can confirm that adding the if null check solves this issue.

It happens with the StaticMap process plugin as well. I'm about to go see if an issues exists for that one.

agileadam’s picture

Here's a patch (against 8.3.7) that adds NULL handling and associated tests to Concat, Substr, and StaticMap. I've tested on about 30 migrations (100k records) without issues (FWIW). I will create a patch against 8.4.x-dev if/when time allows.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

mpp’s picture

I can confirm that the patch provides a workaround.
Otoh adding a skip process plugin also works.

@mikeryan, could you comment on #11?

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev
iarla’s picture

Re-rolling #12 to account for Drupal 9.3.x code changes.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

danflanagan8’s picture

The Migrate Conditions module provides some tricks to help guard against problems during stubbing, specifically using the is_stub condition.

See https://www.drupal.org/docs/contributed-modules/migrate-conditions/migra...

Version: 9.5.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.