Problem/Motivation
Right now, it's difficult with the 8.1.x migration system to identify migrations that are "ready-to-use". For example, if you have custom migrations setup, and also happen to have migrate_drupal enabled but have not configured any Drupal migrations, you want drush migrate-status to list only your custom migrations. Unfortunately, excluding those unwanted Drupal migrations in a general way is... problematic. This is a major hurdle to development of general-purpose tools.
Proposed resolution
Introduce an explicit "registration" capability. Looking at what we've done in the past, two options are:
- In D7, there was a registerMigration() API which would add a row to the migrate_status table representing the migration being registered - you could call it explicitly at runtime, or provide migration configuration in hook_migrate_api() which would be fed into the registration process. The tools then would manage only migrations found in that table - you might have migration classes in your environment (like migration plugins in 8.1.x), but they would be invisible to the tools until registered. A D8 version of this approach would, of course, use state rather than a table to record registrations.
- In 8.0.x, runnable migrations were configuration entities and tools could identify them simply with an entity query. A module could either provide these entities directly in config/install, thus "registering" them at install time, or implement them as migration templates - these templates could then be configured with any additional configuration they needed (such as a database connection) and saved as configuration entities. So, in 8.1.x we could treat the raw migration plugins as templates providing default configuration, and save fully-configured migrations as configuration entities. The issue here will be executing the plugins with the overridden configuration - we'll need to address #2681869: Provide clean way to merge configuration into migration plugins.
I'm leaning towards the configuration entity approach - this is something we'll want anyway to support fully configuring migrations through a UI.
Remaining tasks
Start with a POC.
User interface changes
API changes
Data model changes
Comment | File | Size | Author |
---|---|---|---|
#3 | provide_a_means_of-2701277-3.patch | 14.56 KB | mikeryan |
Comments
Comment #2
mikeryanSo, pursuing the configuration entity angle... What if we have a migration plugin manager that, rather than using YamlDirectoryDiscovery to define migrations from YAML in migrations and migration_templates directories, instead had a discovery class which did an entity query on our new(/old) Migration configuration entities? The tools could then use this plugin manager instead of the default one in core. Custom migrations could be defined in config/install just like in 8.0.x, or we could generate the configuration entities from scratch in a UI tool. The trickiest part would then be how to leverage existing migration plugins - pull in their default configuration, allow it to be extended/overridden, and save them to configuration entities. Perhaps not too hard, although we may need a public getConfiguration() on the Migration plugin class...
Comment #3
mikeryanThat was not nearly as hard as I imagined... You'll want to patch migrate_tools as well (see the tail end of #2667368: Make migrate_tools work with Drupal 8.1.x to make use of this.
Comment #4
agoradesign CreditAttribution: agoradesign commentedOk, I won't have time to try this out before next week, but just for clarification:
That means: my custom migration files stay in config directory like in 8.0.x, the only thing I have to do is rename them from "migrate.migration.xxx.yml" to "migrate_plus.migration.xxx.yml", and everything should work as expected? Cool :)
Comment #5
mikeryanYep, that should be all you need to do if your migration .yml files were in config.
Comment #7
mikeryanI've gone ahead and committed this - the contrib tools were broken anyway, and this will get the new approach more visibility and review.
Comment #8
floydm CreditAttribution: floydm at Affinity Bridge commentedComment #9
agoradesign CreditAttribution: agoradesign commentedGreat work so far! I've today moved my current project to 8.1.x. My custom migrations still work with latest dev of migrate_plus and migrate_tools :)
Comment #12
mikeryanOf course it fails to apply you ninny...
Comment #13
Anonymous (not verified) CreditAttribution: Anonymous for The Linux Foundation commentedSo awesome. Thank you!