Problem/Motivation

Drupal migration source plugins should use SourcePluginBase::getSourceModule() to check their requirements instead of simply checking their plugin definition. (This method was unused until #2936365: Migrate UI - allow modules to declare the state of their migrations aka https://git.drupalcode.org/project/drupal/-/commit/c1734c61852faecf3f358... was commited).

Current checkRequirements only checks the plugin definition for the source module whereas getSourceModule will check configuration first, for an overridden source module, and then the plugin definition.

Proposed resolution

  • Use getSourceModule in DrupalSqlBase
  • Modify tests that need the source module enabled in the database so that requirements are met.
  • Remove two lines from core/modules/field/migrations/d6_field.yml, which duplicate two lines in core/modules/field/migrations/state/field.migrate_drupal.yml, since they break several of the improved tests.

Remaining tasks

Identify and address BC issues.

User interface changes

Nothing.

API changes

Data model changes

Release notes snippet

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

huzooka created an issue. See original summary.

huzooka’s picture

Before adding tests or doing anything more, let's see what this patch breaks.

Status: Needs review » Needs work
quietone’s picture

Just some thoughts.

  1. +++ b/core/modules/migrate/src/Plugin/MigrateSourceInterface.php
    @@ -116,4 +116,13 @@ public function getIds();
    +  public function getMinimumSchemaVersion();
    

    I think that since this is specific to a Drupal database it should be in DrupslSqlBase. I know that this interface also has getSourceModule which is also specific to Drupal database, it shouldn't be here either. I'd rather not compound the error.

  2. +++ b/core/modules/migrate_drupal/src/Plugin/migrate/source/DrupalSqlBase.php
    @@ -103,14 +103,16 @@ public static function create(ContainerInterface $container, array $configuratio
    +          $minimum_schema_version = $this->getMinimumSchemaVersion();
    +          $module_schema_version = (int) $this->getModuleSchemaVersion($source_module);
    +          if ($minimum_schema_version && !($module_schema_version < $minimum_schema_version)) {
    

    It is nice to have the comparison done on integers.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Wim Leers’s picture

Issue summary: View changes
+++ b/core/modules/migrate_drupal/src/Plugin/migrate/source/DrupalSqlBase.php
@@ -103,14 +103,16 @@ public static function create(ContainerInterface $container, array $configuratio
-          if (isset($this->pluginDefinition['minimum_schema_version']) && !$this->getModuleSchemaVersion($this->pluginDefinition['source_module']) < $this->pluginDefinition['minimum_schema_version']) {
-            throw new RequirementsException('Required minimum schema version ' . $this->pluginDefinition['minimum_schema_version'], ['minimum_schema_version' => $this->pluginDefinition['minimum_schema_version']]);

I didn't even know the migration system supported this! Very good to know 🤓

quietone’s picture

Status: Needs work » Needs review
FileSize
1.08 KB
2.37 KB

Reroll and remove the source_module declaration from d6_field.yml and let's see what breaks.

Status: Needs review » Needs work

The last submitted patch, 7: 3176393-7.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
Issue tags: -migrate, -migration
FileSize
1.39 KB
3.75 KB

Ah, those tests are nice proof that this is working as intended.

Do we need the tags since this in the migration system component?

quietone’s picture

Meant to change the name of the dblog test method. It looked like an old copy/paste error.

quietone’s picture

Title: SourcePluginBase::getSourceModule() is used only by MigrationState, but not e.g. in DrupalSqlBase::checkRequirements() » Use SourcePluginBase::getSourceModule() in DrupalSqlBase::checkRequirements()
Matroskeen’s picture

Status: Needs review » Needs work

I wanted to review this issue, but I noticed that last patch doesn't match the issue summary.

Introduce a new ::getMinimumSchemaVersion() method in MigrateSourceInterface, implement the new method in SourcePluginBase that applies the same logic as SourcePluginBase::getSourceModule() (but for minimum_schema_version) and refactor DrupalSqlBase::checkRequirements() to use these methods.

I don't see these changes in the patch. If the proposed resolution was changed, we should probably reflect this in the issue summary. Or it still needs more work. In any case, I'm moving this to NR.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

Version: 9.5.x-dev » 10.0.x-dev
Issue summary: View changes
Status: Needs work » Needs review
FileSize
1.18 KB
4.26 KB

IS updated and new patch with just comment changes.

Wim Leers’s picture

Status: Needs review » Reviewed & tested by the community

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 16: 3176393-16.patch, failed testing. View results

Wim Leers’s picture

Status: Needs work » Reviewed & tested by the community
alexpott’s picture

Category: Feature request » Task
Status: Reviewed & tested by the community » Needs review
+++ b/core/modules/field/migrations/d6_field.yml
@@ -11,8 +11,6 @@ source:
-  # Phone is here since it does not use a migrate field plugin.
-  source_module: phone

Why are we removing this? I can't see how this change makes it not necessary more. I checked the issue comments and I couldn't see an explanation. It was added in #2936365-203: Migrate UI - allow modules to declare the state of their migrations maybe the comment #203 on that issue will shed some light as to whether this will break something.

quietone’s picture

Issue summary: View changes

I looked back at the issue that added the 'source_module: phone' and I am convinced it was an error. The correct entry is in core/modules/field/migrations/state/field.migrate_drupal.ymlcheckRequirementscheckRequirements.

benjifisher’s picture

Issue summary: View changes
Status: Needs review » Reviewed & tested by the community

Regarding #20, #21:

I checked the issue comments and I couldn't see an explanation.

The change is mentioned in #7. You will not see it in the interdiff attached to that comment, maybe because that patch was also a reroll.

Looking at the final patch from #2936365-309: Migrate UI - allow modules to declare the state of their migrations, I see this duplication:

    diff --git a/core/modules/field/migrations/d6_field.yml b/core/modules/field/migrations/d6_field.yml
    index e25d7ae0c2..2c64f21ea0 100644
    --- a/core/modules/field/migrations/d6_field.yml
    +++ b/core/modules/field/migrations/d6_field.yml
    @@ -10,6 +10,8 @@ source:
       constants:
         entity_type: node
         langcode: en
    +  # Phone is here since it does not use a migrate field plugin.
    +  source_module: phone
     process:
       entity_type: 'constants/entity_type'
       status: active
    diff --git a/core/modules/field/migrations/state/field.migrate_drupal.yml b/core/modules/field/migrations/state/field.migrate_drupal.yml
    new file mode 100644
    index 0000000000..53c6eb61e2
    --- /dev/null
    +++ b/core/modules/field/migrations/state/field.migrate_drupal.yml
    @@ -0,0 +1,12 @@
    +finished:
    +  6:
    +    content: field
    +    email: core
    +    # Phone does not use a migrate field plugin so it is added here.
    +    phone: field
    +  7:
    +    email: core
    +    entityreference: core
    +    field: field
    +    field_sql_storage: field
    +    number: core

I will mention the disputed change in the issue summary.

I also cannot think of any reason why the field module’s plugin, which includes handling for several other modules (nodereference, number_integer, email, …) in the process section, should declare one of them as its source module.

I was not sure whether this change was related to this issue or an unrelated fix, like the fix mentioned in #10. I tried restoring the two lines to d6_field.yml, and several of the kernel tests in the field module failed. This change is in scope.

In short, I do not see any reason to disagree with #21. Of course, @quietone is far more familiar with both issues than I am.

Back to RTBC.

alexpott’s picture

Version: 10.0.x-dev » 9.5.x-dev
Status: Reviewed & tested by the community » Fixed

Committed and pushed 12df2086f0 to 10.1.x and 397b8d5959 to 10.0.x and 8ca4362795 to 9.5.x. Thanks!

Backported this to 9.5.x since this task is low risk and improves the migration system.

  • alexpott committed 12df208 on 10.1.x
    Issue #3176393 by quietone, huzooka, benjifisher: Use SourcePluginBase::...

  • alexpott committed 397b8d5 on 10.0.x
    Issue #3176393 by quietone, huzooka, benjifisher: Use SourcePluginBase::...

  • alexpott committed 8ca4362 on 9.5.x
    Issue #3176393 by quietone, huzooka, benjifisher: Use SourcePluginBase::...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.