Export migration configuration as code

This would be a general Migrate feature - ideally integrating with Ctools/features.

Reminder that in Drupal 8 you get this for free when you use CMI for configuration.

Yes, this will be much easier with D8. I should note for anyone looking for this on D7, once I get 2.6 out I will be focusing on D8 - I do not expect to implement this in D7, any such effort would have to come from the community (i.e., if someone submits a patch I will review it).

Can't you just define the database-stored class to add a prepareRow() method for example?

Any update on this?

My workaround was to export the settings into SQL file as follows:

#/bin/sh # File: up_migrate_sets.sh
mysqldump -u root -p --extended-insert=FALSE --complete-insert=TRUE --add-drop-table foo7 migrate_field_mapping migrate_group migrate_status > migrate.sql

alternatively with drush:

drush --ordered-dump sql-dump | head -n20 | grep -w SET > migrate.sql
drush --data-only --ordered-dump sql-dump | grep '`migrate' >> migrate.sql

Then to import the settings, either import the SQL file manually, or by the following script at: #2153897-2: Ability to add database credentials in settings.php.
This can be extended to the drush shell-alias, e.g.:

  'foo-migrate-refresh' => '!drush eval "module_load_include(\'install\', \'foo_migrate\'); foo_migrate_install();" && drush migrate-register',

Note: It could have some problem with older version of PHP (5.3).

So if you're working on multiple environments, all the migrate classes can use the following db credential format in settings file:

$databases['migrate']['default'] = array (
  'driver' => 'mysql',
  'database' => 'drupal6',
  'username' => 'root',
  'password' => 'root',
  'host' => '127.0.0.1',
  'port' => '3306',
  'prefix' => '',
);

In your migrate classes you load the query by:

$query = Database::getConnection('default', 'migrate');

Then to import the configuration on different environment, you can just run: `drush foo-migrate-refresh` which will load the following code (import sql file and update db credentials based on the settings file):

/**
 * Implements hook_install().
 */
function foo_migrate_install() {
  $sql = file_get_contents('scripts/foo/migrate/migrate.sql');

  preg_match_all('/(.*?);[\r\n]+/sm', $sql, $matches);

  $connection6 = Database::getConnection('default', 'migrate');
  $connection7 = Database::getConnection('default', 'default');
  $info6       = $connection6->getConnectionOptions();

  // Executing migrate SQL queries.
  foreach ($matches[0] as $query) {
    $stmt = $connection7->prepare($query);
    $stmt->execute();
  }

  // Updating migrate group source connection settings.
  $row = $connection7->select('migrate_group', 'mg')
    ->fields('mg')
    ->condition('name', 'FooGroup')
    ->execute()
    ->fetchObject();

  $arguments = unserialize($row->arguments);
  $arguments = MigrationBase::decryptArguments($arguments);

  $arguments['source_database']['driver']    = $info6['driver'];
  $arguments['source_database']['database']  = $info6['database'];
  $arguments['source_database']['username']  = $info6['username'];
  $arguments['source_database']['password']  = $info6['password'];
  $arguments['source_database']['db_prefix'] = $info6['prefix']['default'];
  $arguments['source_database']['port']      = $info6['port'];
  $arguments['source_database']['host']      = $info6['host'];

  $connection7->update('migrate_group')
    ->fields(array(
      'arguments' => serialize(MigrationBase::encryptArguments($arguments)),
    ))
    ->execute();
}

In D8 you would get export and import for free. What you wouldn't get is anyway to build this in the UI and then implement a plugin (or something) that would do additional processing. What this really needs is views style alter hooks, that would probably be the easiest.

Ideally one would want to build the initial mappings and settings with the UI and then be able to export that class as a module to add custom php to prepare row or complete or where ever this is needed. Or alternatively have the UI offer the ability to make a wizard defined class as an extension point for a new migrate class. Then that class could be registered and run instead of the UI defined class.

This would be a huge benefit to both D7 and D8 versions of migrate.

This is a particular issue for my project as well. I put together a ctools/features implementation for migrate that is minimally invasive & should work for the use-case of UI configured migration patterns needing to be exported/imported/managed by features.

Attached is a first pass at it. Once applied, you should be able to export individual migration plans, and import/revert them into new environments.

A nice benefit to this is that if you need to add arguments/options that aren't supplied as configurable in the UI (such as batch_limit), you can edit the mymodule.migrate.inc file add them to the migration config, and revert the feature/component.

This needs some testing, but should be pretty solid.

minor bugfix to add to this

sorry for the update stream, but one final update to get rid of some php errors if there are no fields configured for a migration.

I tested the patch - it applies correctly, but I had to use Features to export it. Would it be possible to allow also exporting with Ctools Bulk Exporter? I didn't have the time to test actual export and import. Also, isn't it required to reinstall Migrate to be able to use this patch?

I'm not sure about the ctools bulk exporter. My specific use-case was around making this exportable with features. I don't see a reason why it wouldn't work though, but as they say, buyer beware :)

You should not have to do any re-installing in order to get this to work. Just apply the patch, and clear the cache. After that, all the migration plans should show up in features under "Migrate".

thanks for your patch, netw3rker

After applying the patch I saw all migrations from features' menu (inside migrate fieldgroup), just as you said. Unfortunately I haven't been able to make it work, as no migration group is exported.

Even when I manually start the drupal2drupal migration using the ui so the migration group is created, the migration mappings seem to be ignored.

Maybe I'm doing something wrong?

EDIT: I have edited the information as it was poorly written.

Hi c-c-m,

make sure that on the destination environment you revert the feature after enabling it. The one bug I can't seem to solve here is that features doesn't do anything with the export during module_enable().

That should put everything in the migrate_status and migrate_field_mappings tables for you.

I caught a bug with my first patch. The highwater mark was being exported, and reverted to on features_revert. This causes successive migrations to backtrack over data that should have been ignored. On very large migrations it can add lots of time to the migration, and if you ran the migration before exporting the settings, it can cause destination environments to not import anything.

This new patch fixes this and preserves the highwater mark where applicable. Hope this helps!

I have the same issue than c-c-m, I don't see any groups: "No migration groups defined."

I create my migrate feature, by selecting all my Migrate tasks inside the component Migrate. Features added automatically the dependencies Ctools and Migrate.

After installing my new features, I could see all my migration tasks inside the component migrate of my Feature.

As explained, I revert the migrate component which was in overridden status.
But still in /admin/content/migrate I don't see any migration group.

By applying the Reset option, it shows my Migration group not by name but by its ID.
But it also diplays lots of errors identicall for each migration task:

Migration d1e4e5c53File could not be constructed.
No source_version provided in migrate_d2d migration arguments.
Migration d1e4e5c53Nodeapproval_icons could not be constructed.
No source_version provided in migrate_d2d migration arguments.

I tested the patch #16 but no luck. I am having the same issue as #17 where I don't see any migration group under /admin/content/migrate.

Same issue than #17 and #18
Export is ok but i don't see any migration group :(

I dont see any migration group either, and I get this message when I try to run it with drush : "No source_version provided in migrate_d2d migration arguments. "

It's working, the concept is that it'll never consider the group(as this contains the db connection information which can be changed environment wise) so you should create the group in each environment from migrate_ui along with the source db information, then you revert the feature so that manual mappings(from previous environment) are all kept as it's in the new environment.

RTBC

yes It's working #21,please follow those steps to achieve the migration group settings.

After a year, I finally got patch in #16 to work, thanks to #21. I am posting exact steps below for those who are still stuck. I hope noob developers like me would find the steps below useful.

Steps below were tested using Migrate 7.x-2.x-dev and Migrate D2D 7.x-2.x-dev (2015-Nov-28).

1) Install and enable Features, Migrate , and Migrate D2D modules on both the source (site containing the manually mapped field configurations) and destination (site where you want to import the field mapping configurations).

2) Apply patch in comment #16 to the Migrate module on both source and destination site. Then clear cache How to apply patch? https://www.drupal.org/patch/apply. Note: You need to apply patch to BOTH source and destination.

3) On your source site, create feature (admin/structure/features/create) with all your migrate configurations. Select all your migrate task which will be listed under Components >> Migrate. The dependencies (Chaos tools and Migrate) will automatically be selected. Download the feature (.tar) and install it on your destination site. Enable the feature on your destination site. Note: Those that are not familiar with Features module, you might want to look at the tutorial: https://www.youtube.com/playlist?list=PL-Ve2ZZ1kZNTykHOLO0jzRADq8iZxvzkU

4) On your destination site, create a new migration group (admin/content/migrate/new/migrated2dwizard). Input all your database credentials and go through the wizard. The migration group will now be visible in Migrate (admin/content/migrate). Go to this group and delete all the task that were created with the wizard.

5) On your destination site, go to manage Features (admin/structure/features) and you will see your feature from step 3 in Overridden state. Select all your migration task and "Revert components".

6) After step 5, all your field mapping configurations from the source will be duplicated to destination.

In addition to my post in #23, I wanted to update everyone on new my new findings.

Using patch #16 also enables importing/exporting the migration configurations using the Configuration Management module. Thus no need to use Features module. However, using Configuration Management module is not fully tested. Below are my testing steps and results.

source site: site1
destination site: site2

Due to the Features module, both site1 and site2 had the same migration configurations. I made changes to one of the migration configurations on site1 (ex: f53297286Nodenews). I used Configuration Management to export the migration configurations of f53297286Nodenews and import it into site2 using the .tar file. On my first attempt I got the error blow on site2 upon importing. In addition, the migration configuration, f53297286Nodenews, was no longer present inside my migration group (admin/content/migrate/groups/f53297286). After seeing the error below, I tried to re-import on site2 which resulted in success without errors, and migration configuration, f53297286Nodenews, reappeared under my migration group with the changes I made in site1. I am unsure why the errors below are occurring. I repeated these steps three times, and it seems that on the first attempt of importing changes from site1 to site2 the error below is give, but second attempt fixes everything.

Everything works smoothly without errors if I first delete all the migration configurations from site2 under the migration group and then import all the migration configurations from site1.

Warning: Illegal string offset 'default hook' in _ctools_export_get_defaults() (line 666 of C:\xampp\htdocs\www\site2\sites\all\modules\ctools\includes\export.inc).
Warning: Illegal string offset 'default hook' in _ctools_export_get_defaults() (line 674 of C:\xampp\htdocs\www\site2\sites\all\modules\ctools\includes\export.inc).
Warning: Illegal string offset 'default hook' in _ctools_export_get_defaults() (line 699 of C:\xampp\htdocs\www\site2\sites\all\modules\ctools\includes\export.inc).
Notice: Trying to get property of non-object in migrate_ctools_save() (line 636 of C:\xampp\htdocs\www\site2\sites\all\modules\migrate\migrate.module).
Warning: Creating default object from empty value in migrate_ctools_save() (line 639 of C:\xampp\htdocs\www\site2\sites\all\modules\migrate\migrate.module).
Notice: Undefined property: stdClass::$machine_name in migrate_ctools_save() (line 642 of C:\xampp\htdocs\www\site2\sites\all\modules\migrate\migrate.module).
Notice: Undefined property: stdClass::$fields in migrate_ctools_save() (line 644 of C:\xampp\htdocs\www\site2\sites\all\modules\migrate\migrate.module).
Notice: Undefined property: stdClass::$machine_name in migrate_ctools_save() (line 654 of C:\xampp\htdocs\www\site2\sites\all\modules\migrate\migrate.module).
PDOException: SQLSTATE[HY000]: General error: 1364 Field 'machine_name' doesn't have a default value: INSERT INTO {migrate_status} (highwater) VALUES (:db_insert_placeholder_0); Array ( [:db_insert_placeholder_0] => ) in drupal_write_record() (line 7361 of C:\xampp\htdocs\www\site2\includes\common.inc).