After uninstall migration modules the tables created not removed and still exist and can not be reasonable to deploy without remove that tables.

Problem:
The 3 modules Migrate, Migrate_drupal, Migrate_drupal_ui didn't implement Hook_uninstall() to remove these tables or other data related to migration and have no use after complete migration.

Motivation:
List all tables prefixed with migrate_ in hook_install.
Implement hook_uninstall to remove all tables prefixed with migrate_ except those listed when install the same prefix.

Solution
The migrate core module must list all tables prefixed with migrate_ during installl.
The migrate module must remove all tables prefixed with migrate_ during uninstall.

Kindly reveal the wrong suppositions I made here and the best practice to achieve.

Members fund testing for the Drupal project. Drupal Association Learn more

Comments

mhmd created an issue. See original summary.

catch’s picture

Title: Remove migration tables during uninstall » Migration tables (ID map etc.) don't get removed
Priority: Critical » Major
Issue tags: +Migrate critical

Migrate is still experimental, so moving to major and tagging 'migrate critical' instead.

The ID map table will be removed in MigrateIdMapInterface::destroy() - however I don't see that called anywhere outside tests so think you're right that at the moment there's no straightforward way to remove the tables.

mhmd’s picture

I have implemented the hook_install && uninstall hooks in migrate.install to drop all tables prefixed with migrate except those where found during module installation (may be other contributed or custom modules defines tables prefixed with migrate) which is not a good decision.

Please review and test.

swentel’s picture

I don't see why this is a bug report, having those tables actually still around is interesting for figuring out what was migrated, what status they have and messages (if any). This allows you to even write custom post migration scripts to fix or do additional things that aren't in a plugin at the moment or aren't straightfoward to handle.

So to me this is a task, not a bug.

- edit - oh wait, on uninstall .. that's different, I need to read everything :)
I'd say this is a normal bug report though :)

cilefen’s picture

Priority: Major » Normal

Agreed–I do not see that having extra tables in a database qualifies as Major.

catch’s picture

Issue tags: -Migrate critical

Which means not migrate critical either. If there was a problem re-installing it'd be different, but because the tables are created when needed that shouldn't be an issue.

ckaotik’s picture

Personally, when I uninstall a module and read

The following modules will be completely uninstalled from your site, and all data from these modules will be lost!

I kind of expect the migration map tables to be part of "all data". Drupal\migrate\Plugin\migrate\id_map\Sql::destroy() implements exactly this functionality (and is currently the only id_map core knows).

I see two main options we could implement:

  1. Implement hook_uninstall and run destroy() on all migrations' id_map plugins, as long as the migrations requirements are met. This ensures that only tables stemming from our migrations are affected.
  2. Provide some other means to run destroy(), e.g. via a button on a (not yet existing) migrate configuration page or a drush command in Contrib
JoshOrndorff’s picture

Just to confirm:
In the meantime, Is it safe for me to manually drop these tables?

chegor’s picture

Version: 8.1.0 » 8.2.4

Confirming that problem still available for 8.2.4
Decision with manual removing tables works for me

joelpittet’s picture

Version: 8.2.4 » 8.4.x-dev
claudiu.cristea’s picture

+1 for #4. Those tables should not be removed. The behavior it's not a bug, it's intentional. The same behavior was in the Migrate module.

rodrigoaguilera’s picture

I feel that if you need to keep the references from the maps you can just keep the migrate module installed.

chegor’s picture

Let's think about next use case:
I enabled modules for migration (incl. "Migrate_drupal_ui").
I migrated content by using UI.
Now have drupal8 and want to live with it.

What is the point of having an interface for migration and not be able to remove everything (by using UI/uninstalling modules) that I do not need after the migration has been completed?

rocketeerbkw’s picture

Here's another use case for removing them:

I have migrations running on cron on Drupal 8.0.x. I upgraded to 8.2.x, and all the migrations are broken. I can fix the migration and custom source plugins easily (simple migrations, not migrate_drupal), but I'm still stuck because there is no upgrade path between migrate module versions, see for example https://www.drupal.org/node/2679797.

I'd like to just rollback all content, uninstall all migrate related modules, upgrade drupal, then re-install migrate and import the new content. That fails because there are still all the old map and message tables with invalid schema.

When I uninstall a module, I want to start from scratch. I believe that was the reasoning for removing the "disabled" status for modules.

Tytest’s picture

So the solution is to drop any table with migrate_map_* or migrate_message_* ?

mErilainen’s picture

Dropping tables sound reasonable, at least it would offer a way to start from scratch without deleting tables from database manually.

ckaotik’s picture

Dropping the tables is fine and safe as long as you delete both map and message tables. To delete all tables of currently present migrations, you can run something like this before uninstalling the module:

  // Specify migration ids. An empty array means "all migrations".
  $migrations = [];
  $plugin_manager = \Drupal::service('plugin.manager.migration');
  $instances = $plugin_manager->createInstances($migrations);

  foreach ($instances as $id => $instance) {
    $instance->getIdMap()->destroy();
  }
eneko1907’s picture

Per conversation with Mike, the process described in #3 and #17 present challenges.

It would be safer to check and delete the tables in the state, rather than scan the tables prefixed with migrate. Why? it could happen that other contrib or custom module creates tables whose name would begin with migrate -- therefore, the uninstall process suggeted here would result in bad consequences (break other modules)

mhmd’s picture

@eneko1907, I wrote the code to scan the DB before starting the migration process and store all tables starting with migrate in a variable.

When uninstall drop the tables not present in that variable.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Andrew Answer’s picture

I found the problem with #3. When you patch 'migrate' module with this patch and try to uninstall module, you get

in_array() expects parameter 2 to be array, null given migrate.install:57

because you haven't prefixed_migrate_tables variable in DB.
I fixed the patch to check this situation. Minor styling added also.

quietone’s picture

Thanks everyone for working on this.

I am concerned that the tables will be removed prematurely. That is, before the site is thoroughly tested or without the user fully understanding the risk. Whatever solution is taken to remove the tables it really should include a warning (in big red scary letters) that those tables are needed to fix problems with the migration. They need to be left until they are 100% sure that the site has been thoroughly tested. And that once removed, it may not be possible to fix a problem.

mhmd’s picture

I totally agree with you @quietone, we shall display the list of tables and let the user take the action either delete or keep the tables.

sopranos’s picture

yes that is what I am talking about......I installed just a core drupal 8 site...used migrate to migrade D6 site, it was just a standard site, nothing fancy.....few blog posts...few pages......nothng major, no extra field, custome stuff.....it has like 70 tables in drupal 6.....but after using coremigrate module.....the number of tables jumped to over 220.....which is a disaster..because blue host limits me with 1000 tables per hostig acccount.....so I want to save on tables.....

somebody needs to fix this...this module is just a disaster.....

quietone’s picture

@sopranos, thanks for reporting your problem with the extra tables. As pointed in #11 and #22 the tables are left after the upgrade so that it is possible to fix any problem. It is up to the user to delete them, knowing that it will make fixing problems with the upgrade difficult. But once you are completely sure the new site is functioning as expected it is perfectly OK to drop the tables by whatever means available to you.

To help fix this, you could try the patch in #21 in a test environment and report the results here.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

hudri’s picture

I've got the same issue as @rocketeerbkw in #14.

Uninstall module => reinstall module => migration failed because of invalid old table schema. For me this is a bug, and the tables should be removed on uninstall.

I can't follow the reasoning behind "leaving the tables so user can fix problems with migration"... when the user uninstalls the module, his intentions are obvious. IMHO the current handling is counter-intuitive.