Problem/Motivation

Right now, it's difficult with the 8.1.x migration system to identify migrations that are "ready-to-use". For example, if you have custom migrations setup, and also happen to have migrate_drupal enabled but have not configured any Drupal migrations, you want drush migrate-status to list only your custom migrations. Unfortunately, excluding those unwanted Drupal migrations in a general way is... problematic. This is a major hurdle to development of general-purpose tools.

Proposed resolution

Introduce an explicit "registration" capability. Looking at what we've done in the past, two options are:

  1. In D7, there was a registerMigration() API which would add a row to the migrate_status table representing the migration being registered - you could call it explicitly at runtime, or provide migration configuration in hook_migrate_api() which would be fed into the registration process. The tools then would manage only migrations found in that table - you might have migration classes in your environment (like migration plugins in 8.1.x), but they would be invisible to the tools until registered. A D8 version of this approach would, of course, use state rather than a table to record registrations.
  2. In 8.0.x, runnable migrations were configuration entities and tools could identify them simply with an entity query. A module could either provide these entities directly in config/install, thus "registering" them at install time, or implement them as migration templates - these templates could then be configured with any additional configuration they needed (such as a database connection) and saved as configuration entities. So, in 8.1.x we could treat the raw migration plugins as templates providing default configuration, and save fully-configured migrations as configuration entities. The issue here will be executing the plugins with the overridden configuration - we'll need to address #2681869: Provide clean way to merge configuration into migration plugins.

I'm leaning towards the configuration entity approach - this is something we'll want anyway to support fully configuring migrations through a UI.

Remaining tasks

Start with a POC.

User interface changes

API changes

Data model changes

Members fund testing for the Drupal project. Drupal Association Learn more

Comments

mikeryan created an issue. See original summary.

mikeryan’s picture

So, pursuing the configuration entity angle... What if we have a migration plugin manager that, rather than using YamlDirectoryDiscovery to define migrations from YAML in migrations and migration_templates directories, instead had a discovery class which did an entity query on our new(/old) Migration configuration entities? The tools could then use this plugin manager instead of the default one in core. Custom migrations could be defined in config/install just like in 8.0.x, or we could generate the configuration entities from scratch in a UI tool. The trickiest part would then be how to leverage existing migration plugins - pull in their default configuration, allow it to be extended/overridden, and save them to configuration entities. Perhaps not too hard, although we may need a public getConfiguration() on the Migration plugin class...

mikeryan’s picture

That was not nearly as hard as I imagined... You'll want to patch migrate_tools as well (see the tail end of #2667368: Make migrate_tools work with Drupal 8.1.x to make use of this.

agoradesign’s picture

Ok, I won't have time to try this out before next week, but just for clarification:

That means: my custom migration files stay in config directory like in 8.0.x, the only thing I have to do is rename them from "migrate.migration.xxx.yml" to "migrate_plus.migration.xxx.yml", and everything should work as expected? Cool :)

mikeryan’s picture

Yep, that should be all you need to do if your migration .yml files were in config.

  • mikeryan committed 52d4ba8 on 8.x-2.x
    Issue #2701277 by mikeryan: Use configuration entities as a registration...
mikeryan’s picture

Status: Needs review » Fixed

I've gone ahead and committed this - the contrib tools were broken anyway, and this will get the new approach more visibility and review.

floydm’s picture

agoradesign’s picture

Great work so far! I've today moved my current project to 8.1.x. My custom migrations still work with latest dev of migrate_plus and migrate_tools :)

Status: Fixed » Needs work

The last submitted patch, 3: provide_a_means_of-2701277-3.patch, failed testing.

The last submitted patch, 3: provide_a_means_of-2701277-3.patch, failed testing.

mikeryan’s picture

Component: Code » API
Status: Needs work » Fixed

Of course it fails to apply you ninny...

Anonymous’s picture

So awesome. Thank you!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.