In #2752335: Properly integrate configuration-entity-based migrations with the core plugin manager, we should be removing (or deprecating) the configuration-entity-specific plugin manager - migrate_tools should then use the core plugin manager directly.

Comments

mikeryan created an issue. See original summary.

mikeryan’s picture

Version: 8.x-2.x-dev » 8.x-3.x-dev
mikeryan’s picture

joelpittet’s picture

I gave this a try with those other patches well not the meta but the fallback #2830031: Fix SqlBase fallback priorities and document them.

It through a lot of errors on drush ms and showed a lot of migrations that may have been from tests.

Example output:



SQLSTATE[42S02]: Base table or view not found: 1146 Table 'cs6.node_revision' doesn't exist: SELECT COUNT(*) AS expression
FROM
(SELECT 1 AS expression
FROM
{node_revision} nr
INNER JOIN {node} n ON n.nid = nr.nid AND n.vid <> nr.vid
WHERE ((n.tnid = 0 OR n.tnid = n.nid)) AND (n.type = :db_condition_placeholder_0)) subquery; Array
(
    [:db_condition_placeholder_0] => support
)

SQLSTATE[42S02]: Base table or view not found: 1146 Table 'cs6.node_revision' doesn't exist: SELECT COUNT(*) AS expression
FROM
(SELECT 1 AS expression
FROM
{node_revision} nr
INNER JOIN {node} n ON n.nid = nr.nid AND n.vid <> nr.vid
WHERE ((n.tnid = 0 OR n.tnid = n.nid)) AND (n.type = :db_condition_placeholder_0)) subquery; Array
(
    [:db_condition_placeholder_0] => webform
)

 Group: Default (default)                Status     Total  Imported  Unprocessed  Last imported
 d6_date_formats                         Idle       3      3         0
 d6_dblog_settings                       Idle       0      1         -1
 d6_filter_format                        Idle       5      5         0
 d6_imagecache_presets                   Idle       11     4         0
 d6_menu                                 Idle       8      8         0
 d6_path_redirect                        Importing  688    0         688
 d6_search_settings                      Idle       1      0         1
 d6_system_cron                          Idle       1      0         1
 d6_system_date                          Idle       1      0         1
 d6_system_file                          Idle       1      0         1
 d6_system_performance                   Idle       1      0         1
...
joelpittet’s picture

Oh but without this, I'm getting "Migration upgrade_d6_field_formatter_settings does not exist" which is weird because it's in the status list... so this seems like the right thing to do. How can I help make this work?

mikeryan’s picture

Version: 8.x-3.x-dev » 8.x-4.x-dev
Status: Needs review » Needs work

The necessary core support to do this is in Drupal 8.3.x - so, I've opened an 8.x-4.x branch which will be compatible with core 8.3.x and above, and the 8.x-3.x will be only compatible with 8.2.x.

mikeryan’s picture

Issue tags: +Baltimore2017
mikeryan’s picture

Status: Needs work » Needs review
StatusFileSize
new2.29 KB
new373 bytes

Ready to go!

joelpittet’s picture

StatusFileSize
new49.01 KB

Still having those issues I mentioned in #5:
test

mikeryan’s picture

@joelpettit: How/where are you defining the "cs6" database connection? Are you sure it's pointing at a full Drupal 6 database?

How did you get your migrations set up? The drush ms output is showing the raw migration IDs, but in #5 you had a message referencing upgrade_d6_field_formatter_settings, which would have been created by a drush migrate-upgrade --configure-only. I would expect all of your D6 migrations to be prefixed with 'upgrade_', or none of them.

I guess, in other words... steps to reproduce?

joelpittet’s picture

@mikerayn;) I'm sure it's pointing to the full database, cs6 is the database name I chose as I work for Computer Science and it's D6, I defined it in the settings.local.php. I've been using it for a while now to migrate. I built the migration using migrate-upgrade config only like this tutorial plus some custom class overrides and tweaks: https://drupalize.me/blog/201604/custom-drupal-drupal-migrations-migrate...

All the migrations are in a custom module's config/install folder. If I don't use this patch then I don't get that kind of drush ms output.

mikeryan’s picture

@joelpettit: I see that the message is for "cs6.node_revision", but in D6 the revision table name was "node_revisions" - it seems like somehow you've picked up at least one D7 migration. I also note that your migrations are in the 'default' group, where it should be 'migrate_drupal_6'.

I'm not able to reproduce this problem - here are the steps I took to try:

drush si -y standard
drush en -y migrate_upgrade
drush migrate-upgrade --legacy-db-url=mysql://root@127.0.0.1:33067/mryan1_dev --legacy-root=http://mikeryan.name --configure-only
drush cex --destination=~/tmp/export -y
cp ~/tmp/export/migrate_plus.migration.* ~/tmp/export/migrate_plus.migration_group.migrate_drupal_6.yml ../../custom/migrate_test/config/install/
drush si -y standard
drush en -y migrate_test,migrate_tools
drush ms

and it comes out fine:

 Group: Import from Drupal 6 (migrate_drupal_6)  Status  Total  Imported  Unprocessed  Last imported
 upgrade_block_content_type                      Idle    1      0         1
 upgrade_d6_date_formats                         Idle    3      0         3
 upgrade_d6_dblog_settings                       Idle    0      0         0
 upgrade_d6_imagecache_presets                   Idle    2      0         2
 upgrade_d6_menu                                 Idle    4      0         4
...
mikeryan’s picture

@joelpettit: You're not setting a 'migrate' connection in settings.php, are you? That's going to cause migrations not relevant to your use case to get instantiated. That 'migrate' fallback is very problematic and I strongly recommend against using it.

joelpittet’s picture

Some possible relevant config:

$databases['default']['default'] = [
  'database' => 'cs8',
  'username' => 'root',
  'password' => 'mikerayn->mikeryan',
  'prefix' => '',
  'host' => 'localhost',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
];
$databases['migrate']['default'] = $databases['default']['default'];
$databases['migrate']['default']['database'] = 'cs6';

$settings['cache']['bins']['discovery_migration'] = 'cache.backend.memory';

Am I doing what you said in #14? I don't totally understand, but kinda sounds like what I'm doing...

mikeryan’s picture

$databases['migrate']['default'] = $databases['default']['default'];
$databases['migrate']['default']['database'] = 'cs6';

Don't do that!

As documented in the patch at #2830031: Fix SqlBase fallback priorities and document them:

It is strongly recommended that database connections be explicitly defined via 'database_state_key' or in the source plugin configuration. Defining migrate.fallback_state_key or a 'migrate' connection affects not only any migrations intended to use that particular connection, but all SqlBase-derived source plugins which do not have explicit database configuration.

Having a hard-coded global fallback was always a bad idea, and it leads into situations like this. If that global 'migrate' connection is defined, it is present for *all* the migrations in core, both those that are relevant to your specific D6 migration, and those that are not, so all those migrations will appear (and get in the way) This wasn't seen in migrate_plus/migrate_tools before, because it was only using configuration entities to define plugins, thus only the migrations that had been configured with a database connection. It's important, however, for the tools to support "regular" migration plugins (defined in the migrations or migration_templates directories of your module), and also having the two plugin managers has caused any number of subtle buggy interactions. So, while it was never a good idea to use the default 'migrate' connection, doing that becomes more painful with this patch.

It's understandable why people did it before - if they ended up trying to use migrations which were missing a configured database connection, they'd get an error along the lines of "Missing 'migrate' database connection" and naturally think they should define such a connection. With the recent changes to SqlBase in core, though, that misleading message should be much less likely to occur, and hopefully we can wean people off using the 'migrate' connection.

joelpittet’s picture

The database connection made sense to me without documentation. Though I appreciate that I may not be doing it the way intended, it's a bit too convenient to make that mistake and likely documented like the above tutorial to do it that way.

I'm not sure what a database_state_key is, so I'll have to go read and understand what it's to do.

joelpittet’s picture

Ok I rebuilt the migrations as you indicated without the migrate database key and it looks better but it's still showing unexpected groups. Is it ok to be showing the 'Default' group below?

 Group: Import from Drupal 6 (migrate_drupal_6)  Status  Total  Imported  Unprocessed  Last imported
 upgrade_block_content_type                      Idle    1      0         1
 upgrade_d6_date_formats                         Idle    3      0         3
....
 upgrade_d6_term_node_revision_5                 Idle    1528   0         1528
 upgrade_d6_term_node_revision_7                 Idle    858    0         858
 upgrade_d6_upload                               Idle    217    0         217
 upgrade_d6_upload_entity_display                Idle    23     0         23
 upgrade_d6_upload_entity_form_display           Idle    23     0         23
 upgrade_d6_user_contact_settings                Idle    4863   0         4863
 Group: Default (default)                        Status  Total  Imported  Unprocessed  Last imported
 block_content_type                              Idle    1      0         1
 block_content_body_field                        Idle    1      0         1
 block_content_entity_display                    Idle    1      0         1
 block_content_entity_form_display               Idle    1      0         1
 user_picture_field                              Idle    1      0         1
 d6_upload_field                                 Idle    1      0         1
mikeryan’s picture

Unfortunately, I haven't figured out a way to keep those extra migrations under "default" from showing up: #2700693-37: [meta] Make MigratePluginManager::getDefinitions() work cleanly with migrate_drupal enabled. Please ignore them for now.

joelpittet’s picture

Thanks, glad to know I'm not the only one:)

  • mikeryan committed de135b1 on 8.x-4.x
    Issue #2795447 by mikeryan, joelpittet: Use the core plugin manager
    
mikeryan’s picture

Status: Needs review » Fixed

At long last, committed!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.