Migration migrate_update_8009 for source hash

Working

I think this $this->messageTableName() causes a loop:

messageTableName()
$this->init();
$this->ensureTables();

+++ b/core/modules/migrate/src/Plugin/migrate/id_map/Sql.php
@@ -439,6 +439,18 @@ protected function ensureTables() {
+      // Add any missing columns to the message table.
+      if ($this->getDatabase()->schema()->tableExists($this->messageTableName)) {
+        if (!$this->getDatabase()->schema()->fieldExists($this->messageTableName, static::SOURCE_IDS_HASH)) {

Could this be in an update hook rather than in the map itself?

I remember asking this in irc at one point and someone mentioned that since migrate was an experimental module, an update wasn't necessary. If we think about it, it isn't just the creation of the column we need to worry about. We also need to worry about the change of the PK and the fact that all lookups are now done by a hash. So, all previous messages (and mappings) would need to go into that hook update and do a one-time hash into the hash column, right?

OK, we should write an update test here. And we'll need a test where the db table was created using a version of drupal < 8.0.5. Not sure how to do that. I guess we could create the table and fabricate the data in the test. Is there a standard way to do this?

I'm guessing there is a way to stub out the db table in phpunit.

+++ b/core/modules/migrate/migrate.install
@@ -23,5 +23,144 @@ function migrate_update_8001() {
+  $storage = \Drupal::service('entity_type.manager')->getStorage('migration');

What happens if the migrate module isn't even enabled, will this fatal?

ah yeah of course. Patch looks good, just needs a test.

@davidwbarratt it was possible in D7 and earlier, if a module was installed but disabled... (though module updates in this state often fail).

Does this patch require the tip of 8.0.x or 8.1.x?

I just tried applying to 8.0.5 and it fails to apply changes to the tests. Ignored those and attempted to run the update and I got:

PHP Fatal error: Class 'Unicode' not found in /var/www/html/asap/d8/core/modules/migrate/migrate.install on line 44

Indeed, there's a missing Unicode class.. This patch adds it.

Hi,

I applied patch 32. After that i uninstalled and reinstalled migrate, migrate_upgrade and migrate_drupal. When trying to "Rerun" import via UI (/upgrade) I get the following errors:
Warning: mt_rand(): max(-1) is smaller than min(0) in Drupal\Core\Database\Database::parseConnectionInfo() (line 217 of core/lib/Drupal/Core/Database/Database.php).
Notice: Undefined offset: 0 in Drupal\Core\Database\Database::parseConnectionInfo() (line 217 of core/lib/Drupal/Core/Database/Database.php).
Notice: Undefined index: driver in Drupal\Core\Database\Database::openConnection() (line 369 of core/lib/Drupal/Core/Database/Database.php).
Notice: Undefined index: driver in Drupal\migrate_upgrade\Form\MigrateUpgradeForm->validateCredentialForm() (line 986 of modules/migrate_upgrade/src/Form/MigrateUpgradeForm.php).

Resolve the issue below to continue the upgrade. Driver not specified for this database connection: upgrade

I have not specified a database called "upgrade". This is what's in my settings.php:

$databases['drupal6_kupf']['default'] = array (
  'database' => 'drupal6_kupf',
  'username' => 'drupal6_kupf',
  'password' => 'secret',
  'prefix' => '',
  'host' => 'localhost',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
);
$databases['default']['default'] = array (
  'database' => 'drupal8_kupf',
  'username' => 'kupf8',
  'password' => 'secret',
  'prefix' => '',
  'host' => 'localhost',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
);

Do you have any idea how I can fix this?

#33 is unrelated to this issue. Please open as a separate thing.

Thanks! I posted the issue here.

Applied patch #32. Importing new content give me Integrity constraint violation, on 2 migrations tested.

$ drush mi blog
exception 'PDOException' with message 'SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '78' for key 'PRIMARY'' in          [error]
/var/www/bb/html/core/lib/Drupal/Core/Database/Statement.php:64
Stack trace:
#0 /var/www/bb/html/core/lib/Drupal/Core/Database/Statement.php(64): PDOStatement->execute(Array)
#1 /var/www/bb/html/core/lib/Drupal/Core/Database/Connection.php(615): Drupal\Core\Database\Statement->execute(Array, Array)
#2 /var/www/bb/html/core/lib/Drupal/Core/Database/Driver/mysql/Connection.php(86): Drupal\Core\Database\Connection->query('INSERT INTO {mi...',
Array, Array)
#3 /var/www/bb/html/core/lib/Drupal/Core/Database/Driver/mysql/Insert.php(37): Drupal\Core\Database\Driver\mysql\Connection->query('INSERT INTO
{mi...', Array, Array)
#4 /var/www/bb/html/core/lib/Drupal/Core/Database/Query/Merge.php(376): Drupal\Core\Database\Driver\mysql\Insert->execute()
#5 /var/www/bb/html/core/modules/migrate/src/Plugin/migrate/id_map/Sql.php(548): Drupal\Core\Database\Query\Merge->execute()
#6 /var/www/bb/html/core/modules/migrate/src/MigrateExecutable.php(260):
Drupal\migrate\Plugin\migrate\id_map\Sql->saveIdMapping(Object(Drupal\migrate\Row), Array, 3)
#7 [internal function]: Drupal\migrate\MigrateExecutable->import()
#8 phar:///usr/local/bin/drush/includes/drush.inc(715): call_user_func_array(Array, Array)
#9 phar:///usr/local/bin/drush/includes/drush.inc(706): drush_call_user_func_array(Array, Array)
#10 /var/www/bb/html/sites/bb-dev/modules/contrib/migrate_tools/migrate_tools.drush.inc(235): drush_op(Array)
#11 [internal function]: drush_migrate_tools_migrate_import('branches_blog')
#12 phar:///usr/local/bin/drush/includes/command.inc(364): call_user_func_array('drush_migrate_t...', Array)
#13 phar:///usr/local/bin/drush/includes/command.inc(215): _drush_invoke_hooks(Array, Array)
#14 [internal function]: drush_command('branches_blog')
#15 phar:///usr/local/bin/drush/includes/command.inc(183): call_user_func_array('drush_command', Array)
#16 phar:///usr/local/bin/drush/lib/Drush/Boot/BaseBoot.php(65): drush_dispatch(Array)
#17 phar:///usr/local/bin/drush/includes/preflight.inc(64): Drush\Boot\BaseBoot->bootstrap_and_dispatch()
#18 phar:///usr/local/bin/drush/includes/startup.inc(289): drush_main()
#19 phar:///usr/local/bin/drush/drush(114): drush_startup(Array)
#20 /usr/local/bin/drush(10): require('phar:///usr/loc...')
#21 {main}

Next exception 'Drupal\Core\Database\IntegrityConstraintViolationException' with message 'SQLSTATE[23000]: Integrity constraint violation: 1062
Duplicate entry '78' for key 'PRIMARY': INSERT INTO {migrate_map_blog} (source_ids_hash, sourceid1, source_row_status, rollback_action,
hash) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4);
Array
(
    [:db_insert_placeholder_0] => 34bf071f7083187c532015c732883ef4cd27d0351b55d4a5c6233fefa9a5d0a5
    [:db_insert_placeholder_1] => 78
    [:db_insert_placeholder_2] => 3
    [:db_insert_placeholder_3] => 0
    [:db_insert_placeholder_4] =>
)
' in /var/www/bb/html/core/lib/Drupal/Core/Database/Connection.php:673
Stack trace:
#0 /var/www/bb/html/core/lib/Drupal/Core/Database/Connection.php(640):
Drupal\Core\Database\Connection->handleQueryException(Object(PDOException), 'INSERT INTO {mi...', Array, Array)
#1 /var/www/bb/html/core/lib/Drupal/Core/Database/Driver/mysql/Connection.php(86): Drupal\Core\Database\Connection->query('INSERT INTO {mi...',
Array, Array)
#2 /var/www/bb/html/core/lib/Drupal/Core/Database/Driver/mysql/Insert.php(37): Drupal\Core\Database\Driver\mysql\Connection->query('INSERT INTO
{mi...', Array, Array)
#3 /var/www/bb/html/core/lib/Drupal/Core/Database/Query/Merge.php(376): Drupal\Core\Database\Driver\mysql\Insert->execute()
#4 /var/www/bb/html/core/modules/migrate/src/Plugin/migrate/id_map/Sql.php(548): Drupal\Core\Database\Query\Merge->execute()
#5 /var/www/bb/html/core/modules/migrate/src/MigrateExecutable.php(260):
Drupal\migrate\Plugin\migrate\id_map\Sql->saveIdMapping(Object(Drupal\migrate\Row), Array, 3)
#6 [internal function]: Drupal\migrate\MigrateExecutable->import()
#7 phar:///usr/local/bin/drush/includes/drush.inc(715): call_user_func_array(Array, Array)
#8 phar:///usr/local/bin/drush/includes/drush.inc(706): drush_call_user_func_array(Array, Array)
#9 /var/www/bb/html/sites/bb-dev/modules/contrib/migrate_tools/migrate_tools.drush.inc(235): drush_op(Array)
#10 [internal function]: drush_migrate_tools_migrate_import('branches_blog')
#11 phar:///usr/local/bin/drush/includes/command.inc(364): call_user_func_array('drush_migrate_t...', Array)
#12 phar:///usr/local/bin/drush/includes/command.inc(215): _drush_invoke_hooks(Array, Array)
#13 [internal function]: drush_command('branches_blog')
#14 phar:///usr/local/bin/drush/includes/command.inc(183): call_user_func_array('drush_command', Array)
#15 phar:///usr/local/bin/drush/lib/Drush/Boot/BaseBoot.php(65): drush_dispatch(Array)
#16 phar:///usr/local/bin/drush/includes/preflight.inc(64): Drush\Boot\BaseBoot->bootstrap_and_dispatch()
#17 phar:///usr/local/bin/drush/includes/startup.inc(289): drush_main()
#18 phar:///usr/local/bin/drush/drush(114): drush_startup(Array)
#19 /usr/local/bin/drush(10): require('phar:///usr/loc...')
#20 {main}

Re-tested my case in #36 after apply only the db update code in the patch without the unit tests. It works for new migration, but still fail for migrate-import --update

Are there specific step required? Uninstall migrate? Rollback all migrations?

Needed patch for 8.1.0

Re-uploading correct version

1. +++ b/core/modules/migrate/migrate.install
   @@ -23,5 +25,144 @@ function migrate_update_8001() {
   +      $map_table_name = 'migrate_map_' . Unicode::strtolower($machine_name);
   +      $map_table_name = Unicode::substr($map_table_name, 0, 63 - $prefix_length);
   +      $message_table_name = 'migrate_message_' . Unicode::strtolower($machine_name);
   +      $message_table_name = Unicode::substr($message_table_name, 0, 63 - $prefix_length);
   

   The idmap plugin has this code already, you can just call $map->mapTableName() and $map->messageTableName(). Note that in theory one could swap in their own idmap plugin and use a different scheme for naming tables.

2. +++ b/core/modules/migrate_drupal/tests/fixtures/drupal6.php
   @@ -30532,21 +30532,6 @@
    ->values(array(
   -  'fid' => '13',
   -  'title' => 'Blog',
   -  'name' => 'profile_blog',
   -  'explanation' => 'Paste the full URL, including http://, of your personal blog.',
   -  'category' => 'Personal information',
   -  'page' => '',
   -  'type' => 'url',
   -  'weight' => '3',
   -  'required' => '0',
   -  'register' => '0',
   -  'visibility' => '3',
   -  'autocomplete' => '0',
   -  'options' => '',
   -))
   -->values(array(
   

   What does this have to do with the issue?

I've had this issue after upgrading from Drupal 8.0.3 -> 8.0.6.

I applied the patch in #40 (had to remove core/modules/user/tests/src/Kernel/Migrate/d6/MigrateUserProfileValuesTest.php part as that file doesn't exist in 8.0.6), and it's worked fine for me - thanks all.

I have no idea of why there's something in the patch and I don't have the technical know-how. Just adjusted the patch for last Drupal :)

Anybody can answer mikeryan?

The version from #40, just without MigrateUserProfileValuesTest.php part.

I've had a case where there are source_ids_hash fields, but their empty.
So I changed the code to always regenerate source_ids_hash field values.

Also removed stuff as per #43.

Also a patch for 8.0.x is included.

I've had a case where there are source_ids_hash fields, but their empty.
So I changed the code to always regenerate source_ids_hash field values.

I had the same - the patch added the new column, but all values were empty so any migrations that used the migration process plugin failed to lookup the value from the other migrations.

I've applied the 8.0.x patch from #47 & it applies cleanly, adds the new column + generates the source_ids_hash value.

Cheers

Hello

We are having some real trouble upgrading from 8.0.x to 8.1.x

I've added an 8.1.8 core, connected to a 8.0.x database.

I've successfully applied patch #47 migration_failed_with-2679797-47.patch

And received the following error.

/var/core/www$ drush updb -y
The following updates are pending:

migrate module :
  8009 -   Update existing database tables if necessary.

Do you wish to run all pending updates? (y/n): y
PHP Fatal error:  Call to undefined method Drupal\migrate_plus\Entity\Migration::getSourcePlugin() in /var/core/www/core/modules/migrate/migrate.install on line 37
PHP Stack trace:
PHP   1. {main}() /home/vagrant/.config/composer/vendor/drush/drush/drush.php:0
PHP   2. drush_main() /home/vagrant/.config/composer/vendor/drush/drush/drush.php:12
PHP   3. Drush\Boot\BaseBoot->bootstrap_and_dispatch() /home/vagrant/.config/composer/vendor/drush/drush/includes/preflight.inc:66
PHP   4. drush_dispatch() /home/vagrant/.config/composer/vendor/drush/drush/lib/Drush/Boot/BaseBoot.php:67
PHP   5. call_user_func_array:{/home/vagrant/.config/composer/vendor/drush/drush/includes/command.inc:185}() /home/vagrant/.config/composer/vendor/drush/drush/includes/command.inc:185
PHP   6. drush_command() /home/vagrant/.config/composer/vendor/drush/drush/includes/command.inc:185
PHP   7. _drush_invoke_hooks() /home/vagrant/.config/composer/vendor/drush/drush/includes/command.inc:217
PHP   8. call_user_func_array:{/home/vagrant/.config/composer/vendor/drush/drush/includes/command.inc:366}() /home/vagrant/.config/composer/vendor/drush/drush/includes/command.inc:366
PHP   9. drush_core_updatedb_batch_process() /home/vagrant/.config/composer/vendor/drush/drush/includes/command.inc:366
PHP  10. _update_batch_command() /home/vagrant/.config/composer/vendor/drush/drush/commands/core/core.drush.inc:1221
PHP  11. drush_batch_command() /home/vagrant/.config/composer/vendor/drush/drush/commands/core/drupal/update.inc:174
PHP  12. _drush_batch_command() /home/vagrant/.config/composer/vendor/drush/drush/includes/batch.inc:98
Drush command terminated abnormally due to an unrecoverable error.                                                                                         [error]
Error: Call to undefined method Drupal\migrate_plus\Entity\Migration::getSourcePlugin() in /var/core/www/core/modules/migrate/migrate.install, line 37
PHP  13. _drush_batch_worker() /home/vagrant/.config/composer/vendor/drush/drush/commands/core/drupal/batch.inc:111
PHP  14. call_user_func_array:{/home/vagrant/.config/composer/vendor/drush/drush/commands/core/drupal/batch.inc:163}() /home/vagrant/.config/composer/vendor/drush/drush/commands/core/drupal/batch.inc:163
PHP  15. drush_update_do_one() /home/vagrant/.config/composer/vendor/drush/drush/commands/core/drupal/batch.inc:163
PHP  16. migrate_update_8009() /home/vagrant/.config/composer/vendor/drush/drush/commands/core/drupal/update.inc:60
The external command could not be executed due to an application error.                                                                                    [error]
Cache rebuild complete.                                                                                                                                    [ok]
Finished performing updates.

The line that appears to be wrong is
migrate.install

foreach ($migration->getSourcePlugin()->getIds() as $field => $schema) { 

Has anyone else got this issue, and if possible how do I resolve it. I believe this is a critical issue as its preventing users from reliably upgrading to 8.1.x leaving them running out of date software.

Thanks

I've elevated this issue to critical from major.
I've done some further work with this issue, and found the migrate_plus is a particular source of the issue. It appears to be the lack the getSourcePlugin which is loaded within

$storage = \Drupal::service('entity_type.manager')->getStorage('migration');

This issue blocks us from being able to move to Drupal 8.1.x.

I believe the issue is a widespread thing, and to de-escalate the issue a patch which reliably works is provided, and preferably merged into the migrate module. If we are to use a hash it should be that the upgrade path to 8.1.x is possible.

This issue hasn't moved in months, and needs to be addressed if people are to want to upgrade to the later versions of Drupal.

I will be following this issue closely, and any ways to get this turned round into production ready code I'm happy to help.

As previously mentioned, experimental modules are not required to provide update paths (although it may be nice to do so in some cases, like this) - moving back to major.

The problem you're seeing is that the last patch was made before #2625696: Make migrations themselves plugins instead of config entities went in (see the change record). That patch removed migration configuration entities from core - since migrate_plus added its own implementation of migration configuration entities on the contrib side, the old update function here is picking those up, but too much has changed since 8.0.x for that to work cleanly. If you didn't have migrate_plus, the update function would have done nothing at all.

So, the update function here would need to use the core migration plugin manager instead of trying to load config entities. That probably won't work until #2560795: Source plugins have a hidden dependency on migrate_drupal is fixed - well, maybe it would work if you have migrate_drupal enabled, but not in the general case.

I'm sorry, but about all I can suggest in the short term is a custom script to update the tables.

As for the patch itself:

1. Why is it numbered 8009 rather than 8002?
2. Somehow some irrelevent user profile stuff crept in.

Discussed in weekly migrate meeting. Considering this is for functionality added in 8.0.x, and we are now in 8.2.x we are going to mark this "not fix". There were no promises that updates would be provided for experimental modules and no one has stepped forward to provide such an update.