Problem/Motivation

\Drupal\taxonomy\Plugin\migrate\D7TaxonomyTermDeriver derives the d7_taxonomy_term migration base plugin definition into one migration plugin definition per vocabulary.

This is great! It makes it easy to see how many terms are migrated per vocabulary, which is valuable. 👍

What's less good, is that every such derived migration also declares an optional dependency on every other one. Even though there is absolutely no interdependency between terms in different vocabularies. For example, when migrating wimleers.com, I see this:

I suspect this was done for historical reasons, because prior to #2977882: Term entity type's "parent" field does not specify "target_bundles" setting: a term from *any* vocabulary can be chosen, it was in principle possible to link to any Term entity, regardless of its bundle. But that was fixed >1 year ago.

Proposed resolution

Remove this optional dependency!

(This allows for greater freedom in which to run migrations. It also is less confusing.)

Remaining tasks

Figure out what is adding this optional dependency. I spent 15 minutes grepping and searching and could not figure it out yet 😑

User interface changes

API changes

Data model changes

Release notes snippet

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Wim Leers created an issue. See original summary.

Wim Leers’s picture

Title: Derived per-vocabulary d7_taxonomy_term migration has optional dependency on all otherper-vocabulary d7_taxonomy_term derived migrations » Derived per-vocabulary d7_taxonomy_term migration has optional dependency on all other per-vocabulary d7_taxonomy_term derived migrations

Typo 🤦‍♂️

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

quietone’s picture

Took a while but this is what I see.

First I noticed that after the migrations are created the migration dependencies on the d7_taxonomy_term is changed from

migration_dependencies:
  required:
    - d7_taxonomy_vocabulary
  optional:
    - d7_field_instance

to

migration_dependencies:
  required:
    - d7_taxonomy_vocabulary
  optional:
    - d7_field_instance
    - d7_taxonomy_term

A look at the process pipeline and there is a use of migration_lookup with a migration of d7_taxonomy_term. So, somewhere there is code that thoughtfully looks for dependent migrations in the pipeline and adds it to the optional dependencies. Where? In \Drupal\migrate\Plugin\Migration::findMigrationDependencies.

And looks like we need to make a new issue to remove the use of 'iterator' and 'migration' from findMigrationDependencies because they are deprecated.

quietone’s picture

And a patch.

Status: Needs review » Needs work

The last submitted patch, 5: 3097312-5.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
FileSize
743 bytes
2.29 KB

Funny, I meant to add the isset().

quietone’s picture

The change to the test was using the removed process plugin, 'iterator'. This patch removes that usage. Since the test has change there is a new fail patch.

The last submitted patch, 8: 3097312-8-fail.patch, failed testing. View results

mikelutz’s picture

Status: Needs review » Reviewed & tested by the community

I like it. This makes sense. There's no point in adding a dependency on yourself, but this will add the dependencies in still if you are looking up against a different migration, which is good to help order them.

Wim Leers’s picture

🥳

larowlan’s picture

I reviewed this with a view to commit it, but I honestly are not qualified to do so, its outside my area of comfort/expertise

I get what is happening in principle, but I'll leave this to someone more familiar with this area of migrate

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/core/modules/migrate/src/Plugin/Migration.php
@@ -624,6 +624,17 @@ protected function findMigrationDependencies($process) {
+        // If the migration uses a deriver and has a migration_lookup with
+        // itself as the source migration then skip adding dependencies
+        // otherwise the migration will depend on all the variations of itself.
+        // See d7_taxonomy_term for an example.
+        $migration = $this->getBaseId();
+        if (isset($this->deriver)) {
+          if (in_array($plugin_configuration['plugin'], ['migration_lookup'], TRUE)
+            && ($plugin_configuration['migration'] == $migration)) {
+            continue;
+          }
+        }

This can be written slightly more concisely as

        // If the migration uses a deriver and has a migration_lookup with
        // itself as the source migration then skip adding dependencies
        // otherwise the migration will depend on all the variations of itself.
        // See d7_taxonomy_term for an example.
        if (isset($this->deriver)
            && $plugin_configuration['plugin'] === 'migration_lookup'
            && $plugin_configuration['migration'] == $this->getBaseId()) {
          continue;
        }

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

raman.b’s picture

Status: Needs work » Needs review
FileSize
2.1 KB
1.08 KB

Addressing #13

benjifisher’s picture

Status: Needs review » Needs work

@raman.b:

Thanks for updating the patch!

According to Line length and wrapping in the coding standards,

  • Control structure conditions may exceed 80 characters, if they are simple to read and understand: ...
  • Conditions should not be wrapped into multiple lines.

But the patch in #15 does exactly what #13 requested, and I am not going to argue with @alexpott.

I will ask for a couple of changes to the comment:

  1. Add a comma before "then skip adding dependencies".
  2. Add a period at the end of that phrase, and start a new sentence with "Otherwise".
Wim Leers’s picture

Status: Needs work » Reviewed & tested by the community
FileSize
1003 bytes
2.1 KB

#16++

Done.

Wim Leers’s picture

Title: Derived per-vocabulary d7_taxonomy_term migration has optional dependency on all other per-vocabulary d7_taxonomy_term derived migrations » Never generate dependencies on derivatives of itself when there's a self-referencing lookup

Generalizing title.

Wim Leers’s picture

Title: Never generate dependencies on derivatives of itself when there's a self-referencing lookup » Never generate migration dependencies on derivatives of itself when there's a self-referencing lookup
quietone’s picture

Title: Never generate migration dependencies on derivatives of itself when there's a self-referencing lookup » Never generate migration dependencies on derivatives of itself is a self_referencing migration_lookuplookup

Want to emphasize the migration_lookup.

quietone’s picture

Title: Never generate migration dependencies on derivatives of itself is a self_referencing migration_lookuplookup » Never generate migration dependencies on derivatives of itself is a self_referencing migration_lookup
alexpott’s picture

Version: 9.2.x-dev » 9.1.x-dev
Status: Reviewed & tested by the community » Fixed

Committed and pushed 0e13819245 to 9.2.x and bf40e13e24 to 9.1.x. Thanks!

  • alexpott committed 0e13819 on 9.2.x
    Issue #3097312 by quietone, Wim Leers, raman.b, benjifisher: Never...

  • alexpott committed bf40e13 on 9.1.x
    Issue #3097312 by quietone, Wim Leers, raman.b, benjifisher: Never...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.