This issue is spun off from #2908282: Throw exception for source plugins without a source_module property.

Problem/Motivation

There are several implicitily required properties to destination, source and process plugins. Rather than waiting for someone to WSOD when they miss one of these implicit requirements, we should throw a loud exception.

The MigrateDestination annotation defines a destination_module property which needs to identify the system which will contain the processed, migrated data. In Migrate Drupal's case, this property is used to identify the D8 module which will "own" the migrated data.

When providing a strong API, it's important to tell consumers when they are doing something wrong. For Migrate Drupal's purposes (our main use case), this property is required in order to inform users whether their D6/D7 data will be migrated into Drupal 8. Therefore, this blocks the completion of the core migration path to Drupal 8, and is Migrate-critical.

Proposed resolution

CommentFileSizeAuthor
#53 2923810-53.patch20.17 KBquietone
#53 interdiff-47-53.txt1.37 KBquietone
#47 interdiff-45-47.txt562 bytesquietone
#47 2923810-47.patch20.76 KBquietone
#45 2923810-45.patch20.44 KBquietone
#45 diff-39-45.txt1.54 KBquietone
#39 2923810-39.patch20.28 KBrakesh.gectcr
#39 interdiff-2923810-37-39.txt495 bytesrakesh.gectcr
#37 interdiff-2923810-34-37.txt867 bytesrakesh.gectcr
#37 2923810-37.patch20.28 KBrakesh.gectcr
#34 interdiff-2923810-33-34.txt3.05 KBphenaproxima
#34 2923810-34.patch18.85 KBphenaproxima
#33 interdiff-2923810-31-33.txt2.29 KBrakesh.gectcr
#33 2923810-33.patch19.24 KBrakesh.gectcr
#31 2923810-31.patch16.78 KBphenaproxima
#27 interdiff-2923810-20-23.txt4.29 KBrakesh.gectcr
#27 2923810-23.patch25.14 KBrakesh.gectcr
#23 interdiff-2923810-20-23.txt4.29 KBrakesh.gectcr
#23 2951715-23.patch1.7 KBrakesh.gectcr
#20 2923810-20.patch20.52 KBheddn
#20 interdiff_12-20.patch9.81 KBheddn
#18 2923810-18.patch16.25 KBheddn
#16 interdiff-2923810-12-15.txt6.78 KBrakesh.gectcr
#15 2923810-15.patch16.94 KBheddn
#12 interdiff-2923810-10-12.txt9.14 KBphenaproxima
#12 2923810-12.patch20.88 KBphenaproxima
#10 interdiff-2923810-8-10.txt450 bytesphenaproxima
#10 2923810-10.patch8.2 KBphenaproxima
#8 2923810-8.patch7.64 KBheddn
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

phenaproxima created an issue. See original summary.

phenaproxima’s picture

Priority: Normal » Critical

By committer consensus, all Migrate criticals are now straight-up critical.

phenaproxima’s picture

This will need a change record, as well as updates to this one.

dipakmdhrm’s picture

Issue summary: View changes

After spending some time creating patch by pulling code from #2908282-53: Throw exception for source plugins without a source_module property, I realized that it's not the only thing needed for this patch.
We need to rebuild the patch keeping the fact in mind that 'destination_module' property is only valid for migration into Drupal destinations. See #2908282-64: Throw exception for source plugins without a source_module property.

Adding links to related comments from #2908282: Throw exception for source plugins without a source_module property in Issue Summary.

xjm’s picture

Priority: Critical » Major

@catch, @effulgentsia, @Cottser, @lauriii and I discussed this and decided that in this case it's probably actually major. This is an important issue for making Migrate more developer- and user-friendly, but we wouldn't block marking Migrate stable on it. (We allow throwing new exceptions in minor releases especially if the code would break later anyway.)

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Gábor Hojtsy’s picture

Issue tags: -Migrate critical

If its not critical, it cannot be Migrate critical either. I believe we decided to harmonize these in Vienna last autumn.

heddn’s picture

Status: Active » Needs review
FileSize
7.64 KB

Let's get this kicked off. I'm sure there will be several failures.

Status: Needs review » Needs work

The last submitted patch, 8: 2923810-8.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

phenaproxima’s picture

Status: Needs work » Needs review
FileSize
8.2 KB
450 bytes

Let us fix those config schema errors. This will probably still fail. But hopefully not as hard.

Status: Needs review » Needs work

The last submitted patch, 10: 2923810-10.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

phenaproxima’s picture

Status: Needs work » Needs review
FileSize
20.88 KB
9.14 KB

A little bit softer now...

Status: Needs review » Needs work

The last submitted patch, 12: 2923810-12.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

heddn’s picture

I think we are going to have to override the destination plugin manager similar to what we did for the base migration plugin manger and provide our own createInstance special snowflake that calls getDestinationModule during the constructor. It is a little later than discovery and processDefinition(), but at least its still pretty early in the bootstrap.

heddn’s picture

Here's a quick test to see if moving this down into createInstance() even makes sense. No interdiff, as I'm still playing around.

rakesh.gectcr’s picture

FileSize
6.78 KB

I have just added the interdiff, If anyone wants to look in b/w

Status: Needs review » Needs work

The last submitted patch, 15: 2923810-15.patch, failed testing. View results

heddn’s picture

Status: Needs work » Needs review
FileSize
16.25 KB

Status: Needs review » Needs work

The last submitted patch, 18: 2923810-18.patch, failed testing. View results

heddn’s picture

Status: Needs work » Needs review
FileSize
9.81 KB
20.52 KB

Here's a more real patch. Let's see how many errors we have.

The last submitted patch, 20: interdiff_12-20.patch, failed testing. View results

heddn’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests

We should now load all destination modules and test they have a destination module, similar to how we are doing that for source_module. I think, given the nature of how we are gathering things, all modules /will/ have a destination module. But we should test for that.

rakesh.gectcr’s picture

Status: Needs work » Needs review
FileSize
1.7 KB
4.29 KB

Tried adding a test, Please review and let me know, is this the right direction?

Status: Needs review » Needs work

The last submitted patch, 23: 2951715-23.patch, failed testing. View results

heddn’s picture

Status: Needs work » Postponed
Related issues: +#2937045: source plugin source_module testing seems incomplete

Take a look at #2937045: source plugin source_module testing seems incomplete and that's what we'll want to do, but for destinations. But since that one is already RTBC let's just postpone this on it.

heddn’s picture

Status: Postponed » Needs work

Hit the wrong status on that last save. Resetting status.

rakesh.gectcr’s picture

FileSize
25.14 KB
4.29 KB

Apologies #23 was totally wrong patch

rakesh.gectcr’s picture

Status: Needs work » Needs review
phenaproxima’s picture

Status: Needs review » Needs work

I think we are going to have to override the destination plugin manager similar to what we did for the base migration plugin manger and provide our own createInstance special snowflake that calls getDestinationModule during the constructor. It is a little later than discovery and processDefinition(), but at least its still pretty early in the bootstrap.

I'm unclear on why we should need to override the destination plugin manager for this functionality. Don't we only want to enforce the destination_module in the context of the migration (given a specific set of tags), not the destination itself? Isn't that why we put the source_module enforcement into the overridden migration manager, not the source plugin manager?

phenaproxima’s picture

Assigned: Unassigned » phenaproxima

Self-assigning, because I want to figure this one out.

phenaproxima’s picture

Status: Needs work » Needs review
FileSize
16.78 KB

Okay, let's try this on for size. It passes tests on my machine. Still needs a dedicated test with all providers enabled, but let's see if there are any failures to deal with first.

The hasDefinition() checks are needed in order to get the tests to pass, because in many tests, certain destinations (like entity:node) won't exist unless certain modules are enabled, thus leading to PluginNotFoundException being thrown. This leads me to agree with @heddn that we should do the source check in processDefinition(), as before, and the destination check in createInstance(). It's weird, but it might be the most appropriate path forward. We can discuss that when we have a green patch with proper tests, though.

No interdiff because I rolled this straight from a clean 8.6.x.

rakesh.gectcr’s picture

Discussed this on migration Maintainers meeting, assigning myself writing the test

rakesh.gectcr’s picture

I have added the test according to the direction of #25

Looks like its going to be some code duplication. so made some changes according to that.

Do we need to add testdestinationProvider like testSourceProvider?

phenaproxima’s picture

Looks good! Thanks for writing the test. I did a little bit of refactoring to slim it down a bit, and fixed some whitespace issues.

The last submitted patch, 33: 2923810-33.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Status: Needs review » Needs work

The last submitted patch, 34: 2923810-34.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

rakesh.gectcr’s picture

Status: Needs work » Needs review
FileSize
20.28 KB
867 bytes
phenaproxima’s picture

Status: Needs review » Needs work
+++ b/core/modules/migrate/src/Plugin/migrate/destination/NullDestination.php
@@ -10,6 +10,7 @@
  * @MigrateDestination(
  *   id = "null",
+ *   destination_module = "system",
  *   requirements_met = false
  * )

Thanks for those fixes -- one change, though. NullDestination's destination_module should probably be migrate, not system.

rakesh.gectcr’s picture

Status: Needs work » Needs review
FileSize
495 bytes
20.28 KB

Thanks Adam :)

quietone’s picture

Status: Needs review » Needs work

I just reviewed the comments and found 1 main issues, that is the comments state that the source or destination module property must be on the annotation but isn't it really a requirement of the plugin definition?

  1. +++ b/core/modules/migrate/tests/src/Kernel/Plugin/MigrationProvidersExistTest.php
    @@ -26,20 +26,36 @@ public function testSourceProvider() {
    +    // All source plugins must define a source_module annotation property.
    

    The source_module property can be defined in the migration yml as well.

  2. +++ b/core/modules/migrate/tests/src/Kernel/Plugin/MigrationProvidersExistTest.php
    @@ -26,20 +26,36 @@ public function testSourceProvider() {
    +    // All destinations must define a destination_module annotation property.
    

    Same here, it can be defined in the migration yml.

  3. +++ b/core/modules/migrate/tests/src/Kernel/Plugin/MigrationProvidersExistTest.php
    @@ -26,20 +26,36 @@ public function testSourceProvider() {
    +  ¶
    

    trailing whitespace

  4. +++ b/core/modules/migrate/tests/src/Kernel/Plugin/MigrationProvidersExistTest.php
    @@ -26,20 +26,36 @@ public function testSourceProvider() {
    +  ¶
    

    trailing whitespace

  5. +++ b/core/modules/migrate_drupal/config/install/migrate_drupal.settings.yml
    @@ -3,3 +3,9 @@
    +# Migrations with any of these tags will raise an exception if their
    

    Is it raise or throw for an exception? Just using dreditor but the 80 column wrapping looks wrong.

  6. +++ b/core/modules/migrate_drupal/src/MigrationPluginManager.php
    @@ -7,21 +7,19 @@
    + * their respective plugin annotations. This is done in order to support the
    

    Hmm, 'plugin annotations' again. Are the definitions in the migration yml considered annotations?

  7. +++ b/core/modules/migrate_drupal/src/MigrationPluginManager.php
    @@ -87,21 +116,42 @@ public function processDefinition(&$definition, $plugin_id) {
    +      // Throw an exception if the destination plugin definition does not define
    +      // a destination_module.
    

    Here there is no mention of 'annotation'. It just referring to the plugin definition which is better and accurate.

rakesh.gectcr’s picture

Issue tags: +Nwdug_may18
rakesh.gectcr’s picture

Assigned: rakesh.gectcr » Unassigned

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

quietone’s picture

Status: Needs work » Needs review
FileSize
1.54 KB
20.44 KB

Just a reroll but includes fix whitespace errors

Status: Needs review » Needs work

The last submitted patch, 45: 2923810-45.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
FileSize
20.76 KB
562 bytes

Add destination_module to the color destination plugin.

quietone’s picture

Issue tags: -Needs tests

Tests were added in #33

Status: Needs review » Needs work

The last submitted patch, 47: 2923810-47.patch, failed testing. View results

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

quietone’s picture

Status: Needs work » Needs review
FileSize
1.37 KB
20.17 KB

Add destination_module to MigrateEntityComplete.

Status: Needs review » Needs work

The last submitted patch, 53: 2923810-53.patch, failed testing. View results

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.