Problem/Motivation

Calling MigrationPluginManager::getDefinitions() when the migrate_drupal module is enabled and no connection is defined for the drupal migrations, we get ConnectionNotDefinedException from the node derivers, and thus get no migrations back (even the ones that are fine). This happens because the derivers execute the node_type source plugin - with no explicit database connection configured, SqlBase attempts to use a connection named 'migrate' and if no such connection exists ConnectionNotDefinedException is thrown.

Proposed resolution

When SqlBase falls back to attempting the 'migrate' connection, the lack of such a connection should be treated as a RequirementsException rather than allowing ConnectionNotDefinedException to propagate. SqlBase should also implement RequirementsInterface, which should throw RequirementsException in this instance. And, the node derivers should check the requirements on the node_type migration and return without deriving any node migrations when the requirements are not met.

Remaining tasks

Extract the relevant parts of the patch in #2700693: [meta] Make MigratePluginManager::getDefinitions() work cleanly with migrate_drupal enabled.

User interface changes

N/A

API changes

getDatabase() will now throw RequirementsException in the absence of a configured database connection.

SqlBase and all derived source plugins will implement RequirementsInterface, and thus checkRequirements() will now be available on all such source plugins.

Data model changes

N/A

Comments

mikeryan created an issue. See original summary.

mikeryan’s picture

Assigned: Unassigned » mikeryan
Issue tags: +Migrate critical
mikeryan’s picture

Status: Active » Needs review
Issue tags: +Needs tests
StatusFileSize
new6.3 KB

I need to add a fail test where the node derivers blow up.

mikeryan’s picture

Issue tags: -Needs tests
StatusFileSize
new1.99 KB
new8.29 KB

Here we go.

The last submitted patch, 3: 2830036-3.patch, failed testing.

The last submitted patch, 4: 2830036-4-FAIL.patch, failed testing.

The last submitted patch, 4: 2830036-4-FAIL.patch, failed testing.

The last submitted patch, 4: 2830036-4-FAIL.patch, failed testing.

mikeryan’s picture

Results as expected - ready for human review.

mikeryan’s picture

Draft change record added.

heddn’s picture

Status: Needs review » Needs work
  1. +++ b/core/modules/migrate/src/Plugin/migrate/source/SqlBase.php
    @@ -127,7 +135,29 @@ protected function setUpDatabase(array $database_info) {
    +      if ($key == 'migrate') {
    

    What if someone for legacy reasons defined their connection 'migrate'? Why does it matter if the key exists or not? Just add the key to the requirements exception and throw it anyway.

  2. +++ b/core/modules/node/src/Plugin/migrate/D6NodeDeriver.php
    @@ -107,7 +117,7 @@ public function getDerivativeDefinitions($base_plugin_definition) {
    -      foreach (static::getSourcePlugin('d6_node_type') as $row) {
    +      foreach ($node_types as $row) {
    

    Is this necessary for fixing this error?

  3. +++ b/core/modules/node/src/Plugin/migrate/D7NodeDeriver.php
    @@ -84,7 +94,7 @@ public function getDerivativeDefinitions($base_plugin_definition) {
    -      foreach (static::getSourcePlugin('d7_node_type') as $row) {
    +      foreach ($node_types as $row) {
    

    Same here. This seems like a stray code cleanup.

  4. +++ b/core/modules/node/tests/src/Kernel/Migrate/MigratePluginListTest.php
    @@ -0,0 +1,57 @@
    +    unset($migration_plugins['user_picture_field']);
    

    Can we check migration plugins that use a source plugin of embed_data? That would make this more extensible for the future.

  5. +++ b/core/modules/node/tests/src/Kernel/Migrate/MigratePluginListTest.php
    @@ -0,0 +1,57 @@
    +    unset($migration_plugins['user_picture_field']);
    

    Can we check migration plugins that use a source plugin of embed_data?

mikeryan’s picture

Status: Needs work » Needs review
StatusFileSize
new8.34 KB
new774 bytes

What if someone for legacy reasons defined their connection 'migrate'? Why does it matter if the key exists or not? Just add the key to the requirements exception and throw it anyway.

The distinction being made here is that 'migrate' is an established (if undocumented) fallback in the absence of any explicit key. Thus, if no explicit key has been provided, the requirement is that there be a 'migrate' key, so if that does not exist then we've failed to meet requirements. If, however, you have provided an explicit key but neglected to actually define the connection, I think it is appropriate to allow the ConnectionNotDefinedException to propagate.

Is this necessary for fixing this error?

Same here. This seems like a stray code cleanup.

It is necessary to separate out the static::getSourcePlugin('d*_node_type') calls to check the requirements up front - having done that, it makes sense to use that result in the for loop rather than duplicate the calls.

Can we check migration plugins that use a source plugin of embed_data?

Even better, we can check that we have none left that use a SQL-based source plugin (patch attached).

(Edit: inserted missing 'not')

heddn’s picture

Status: Needs review » Needs work
+++ b/core/modules/node/tests/src/Kernel/Migrate/MigratePluginListTest.php
@@ -49,9 +49,11 @@
+      $this->assertNotInstanceOf('Drupal\migrate\Plugin\migrate\source\SqlBase',

Nit:
\Drupal\migrate\Plugin\migrate\source\SqlBase::class is more resilient. Or even import that thing and SqlBase::class

mikeryan’s picture

Status: Needs work » Needs review
StatusFileSize
new8.35 KB
new905 bytes

Done, thanks!

heddn’s picture

Status: Needs review » Reviewed & tested by the community

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 14: 2830036-14.patch, failed testing.

mikeryan’s picture

Assigned: mikeryan » Unassigned
Status: Needs work » Needs review
StatusFileSize
new8.39 KB

Simple reroll.

phenaproxima’s picture

Assigned: Unassigned » phenaproxima

Self-assigning for review.

phenaproxima’s picture

Assigned: phenaproxima » Unassigned
Status: Needs review » Needs work
  1. +++ b/core/modules/migrate/src/Plugin/migrate/source/SqlBase.php
    @@ -127,12 +130,17 @@ public function getDatabase() {
    +   *   Thrown if there is no properly-configured database.
    

    Can this be changed to "Thrown if no source database connection is configured"?

  2. +++ b/core/modules/migrate/src/Plugin/migrate/source/SqlBase.php
    @@ -127,12 +130,17 @@ public function getDatabase() {
    +      // If there is no explicit database configuration at all, fallback to a
    

    Nit: Should be "fall back".

  3. +++ b/core/modules/migrate/src/Plugin/migrate/source/SqlBase.php
    @@ -144,7 +152,29 @@ protected function setUpDatabase(array $database_info) {
    +        throw new RequirementsException("No database connection configured for source plugin " . $this->pluginId);
    

    Let's pass the original ConnectionNotDefinedException to the RequirementsException as the previous exception (the fourth constructor argument).

  4. +++ b/core/modules/migrate/src/Plugin/migrate/source/SqlBase.php
    @@ -144,7 +152,29 @@ protected function setUpDatabase(array $database_info) {
    +    if ($this->pluginDefinition['requirements_met'] === TRUE) {
    

    What does requirements_met do? Is it documented in the plugin annotation? If not, can this be commented?

  5. +++ b/core/modules/node/tests/src/Kernel/Migrate/MigratePluginListTest.php
    @@ -0,0 +1,59 @@
    +namespace Drupal\Tests\node\Kernel\Migrate;
    

    Why is this test part of Node? Seems like it should be part of Migrate.

  6. +++ b/core/modules/node/tests/src/Kernel/Migrate/MigratePluginListTest.php
    @@ -0,0 +1,59 @@
    +    // Any database-based source plugins should fail a requirements test in the
    +    // absence of a database connection.
    

    Let's clarify this a bit: "...in the absence of a source database connection (i.e., a connection with the 'migrate' key)."

  7. +++ b/core/modules/node/tests/src/Kernel/Migrate/MigratePluginListTest.php
    @@ -0,0 +1,59 @@
    +      if ($migration->getSourcePlugin() instanceof RequirementsInterface) {
    

    This is kind of a nit, but I don't really like that we're calling $migration->getSourcePlugin() three times, especially since we have no need for the rest of the migration. Can we do something like array_map(function ($migration_plugin) { return $migration_plugin->getSourcePlugin(); }, $migration_plugins) to avoid this?

mikeryan’s picture

mikeryan’s picture

Status: Needs work » Needs review
StatusFileSize
new8.42 KB
new9.54 KB

What does requirements_met do? Is it documented in the plugin annotation? If not, can this be commented?

It is documented in plugin annotation - it defaults to TRUE and gets set FALSE when requirements are not met, thus in this case if there's already a requirements failure we don't check.

Why is this test part of Node? Seems like it should be part of Migrate.

It was part of the node module since we were focused on the errors generated by the node derivers. However, since we're now also dealing with taxonomy derivers as well, I've moved the tests into the existing MigrationPluginListTest in migrate.

All other @phenaproxima issues are addressed, as well as the taxonomy deriver bug (a patch without the taxonomy deriver fix is included to demonstrate that failure).

mikeryan’s picture

StatusFileSize
new6.12 KB

Status: Needs review » Needs work

The last submitted patch, 21: 2830036-21.patch, failed testing.

mikeryan’s picture

Status: Needs work » Needs review

Fail patch failed as expected.

phenaproxima’s picture

Assigned: Unassigned » phenaproxima

Self-assigning for review...again.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community
+++ b/core/modules/migrate/src/Plugin/migrate/source/SqlBase.php
@@ -144,7 +152,29 @@ protected function setUpDatabase(array $database_info) {
+        throw new RequirementsException("No database connection configured for source plugin " . $this->pluginId,[], 0, $e);

Supernit: there should be a space after $this->pluginId. Fixable on commit.

Otherwise, I love it.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 21: 2830036-21.patch, failed testing.

denutkarsh’s picture

Status: Needs work » Needs review
StatusFileSize
new9.54 KB
new760 bytes

Setting this issue to Needs Review. @phenaproxima I am uploading the new patch which follows the suggestion in #26.

denutkarsh’s picture

Status: Needs review » Reviewed & tested by the community

I am setting this to RTBC as set by @phenaproxima.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 28: 2830036-28.patch, failed testing.

mikeryan’s picture

Status: Needs work » Reviewed & tested by the community

Looks like one of those random fails, retesting...

hongpong’s picture

Applying this patch improved my luck in attempting to import via wordpress_migrate , see #2839482: Wordpress Migration UI dies with a ConnectionNotDefinedException when import starts for the gory details. I got my posts into the site correctly, if not the categories! RTBC please.

catch’s picture

The fix looks fine but is there somewhere we can add some docs for contrib doing similar with derivers?

heddn’s picture

Issue tags: +Needs change record

working on that.

heddn’s picture

Issue tags: -Needs change record

Add a CR.

  • catch committed a7ab06c on 8.3.x
    Issue #2830036 by mikeryan, denutkarsh: MigrationPluginManager::...
catch’s picture

Status: Reviewed & tested by the community » Fixed

Committed/pushed to 8.3.x and cherry-picked to 8.2.x, thanks!

hongpong’s picture

Thank you catch, mikeryan and everyone - I appreciate the prompt attention to this issue. Best regards.

  • catch committed a7ab06c on 8.4.x
    Issue #2830036 by mikeryan, denutkarsh: MigrationPluginManager::...

  • catch committed a7ab06c on 8.4.x
    Issue #2830036 by mikeryan, denutkarsh: MigrationPluginManager::...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

mikeryan’s picture

Version: 8.2.x-dev » 8.3.x-dev

Committed/pushed to 8.3.x and cherry-picked to 8.2.x, thanks!

FYI - this actually was not committed to 8.2.x, look for it in 8.3.x forward only...

denutkarsh’s picture

quietone’s picture

Published change record.

quietone’s picture

Publish change record.