The migrate_plus sandbox is basically serving a place to do POCs and initial development of various migration-related contrib functionality - the replacements for any D7 migrate and migrate_d2d features not going into core in 8.0 (keeping in mind that the most generally-useful pieces might get into core in a point release). It's convenient for now to have them in a single project so as we chase HEAD they can all be updated in one fell git pull, but once the API stabilizes (see the open API Issues at #2456261: [META] Finalize the Migration system) I think we'll be able to start splitting out pieces, which different people could take ownership of. Let's work out how to break it down, and start splitting them into submodules within this sandbox, so they're ready to go out on their own.

Tools for running and monitoring migrations

This would include the drush commands (migrate-status, migrate-import, etc.) now being developed here, as well as a UI equivalent to D7's dashboard. I would actually hope the drush commands get folded into core drush at some point and replace the migrate-manifest command, which is far more limited. Until then, I would say the drush commands and the UI would be in one module - call it... migrate_tools?

Source plugins

We've got a patch now for a CSV source plugin, and XML is a major requirement, plus D7 has support for JSON, SQL Server, Oracle, DB2, Excel, and raw HTML files. Should there be a single module/project holding a grabbag of plugins (migrate_source_plugins), or should each be on its own (migrate_source_xml etc.)?

Examples

The basics of the old migrate_example are here now - should they be in their own project? Folded into Examples? Perhaps part of the afore-mentioned migrate_tools?

Note there's an updated migrate_example_baseball in the works as well - I think that would go along with the CSV source plugin, wherever that lands.

Drupal-to-Drupal migration support

Core has basic configurations and plugins for D6 and D7 sources, oriented towards complete site upgrades with no modifications. In contrib we'll want to provide tools to support customized Drupal-to-Drupal migrations as migrate_d2d has done in the past. We'll also want to support migration from D5 and D8. My inclination is to make the tools (UI, plus any underlying support for coded custom migrations) the D8 version of migrate_d2d, and migration from other versions should be separate projects/modules.

Thoughts?

Comments

mikeryan’s picture

As far as source plugins go, @phenaproxima has voted for one migrate_source_plugins module.

brantwynn’s picture

I don't understand the benefit of having modules like migrate_source_csv - it just seems like needless deck-chair arrangement. The src/plugins folder is already nested and things will line up cleanly as they fit into the plugin architecture as it is in core.

heddn’s picture

I think I'd vote to keep a single module for all the plugins.

  • mikeryan committed 981fc06 on 8.x-1.x
    Issue #2491643 by mikeryan: Move drush commands to migrate_tools...
mikeryan’s picture

I've moved the drush commands into the new migrate_tools submodule, in anticipation of starting to sketch out the UI which will also go into migrate_tools.

As for the plugins - haven't quite made up my mind, but need to so we know where the MigrateSourceCSV patch will go. I do like the idea of breaking things down into smaller pieces (after dealing with the monolithic Migrate module over the past few years), not least because the people most invested in specific plugins can take full ownership of them. E.g., heddn and/or brantwynn could take ownership of migrate_source_csv and move it at their pace, and then it would be on me to submit patches for their approval for it to work with use cases like migrate_example_baseball...

heddn’s picture

Since we are dealing with plugins and specific classes set aside for specific purposes, rather than one large monolithic .module file, I don't think we have to split things into a separate module to achieve separation of duties. I'd like to think of the UX of the end-users here, who just want to enable a module and get all the functionality. They don't really care about who maintains what components of the various sub-systems.

mikeryan’s picture

Version: » 8.x-1.x-dev

OK, I think now's the time to break things up, with new XML and JSON source plugins in the works - specifically, moving migrate_source_csv and migrate_tools to their own projects. We want to maintain the same names, which could be painful during transition if both the new project and migrate_plus modules are present in your installation, so it probably needs to be a rip-off-the-bandage sort of thing - when the new projects are ready, commit the removal to migrate_plus, which will break anyone not jumping right on the new projects. Probably the best way to handle this will be to create the new projects but not initially create releases for them, and I'll announce on my blog and the migrate_plus project page a date for the cutover. On that date, we would cut official releases for migrate_source_csv, migrate_tools, and migrate_plus (the latter with the two submodules removed).

Thoughts? I'll create the migrate_tools project, @heddn - do you want to create migrate_source_csv?

heddn’s picture

@mikeryan, sure I can create the csv today. I'll add you as a co-maintainer. I won't create any releases and see if we can get those tests working again as well.

Anyone else want to jump in there as a maintainer?

heddn’s picture

mikeryan: I've added you as a co-maintainer on migrate_source_csv and pushed the code over there.

mikeryan’s picture

Thanks! So, my current plan is:

  1. Today: Create a migrate_plus branch that has migrate_tools and migrate_source_csv removed.
  2. Today: Put a blog post up on Planet Drupal explaining what's going on, setting D-day as next Monday November 16, suggesting those who want to get ahead of it can install the new projects and run the removal branch temporarily.
  3. November 16: Make true releases of migrate_tools (me) and migrate_source_csv (you) (although now that I think about it, there isn't really a compelling reason those have to wait till Monday...). I'll probably make migrate_tools a beta, since the UI side is still very incomplete - what's your confidence in migrate_source_csv being beta- vs. rc-quality?
  4. Merge the removal branch into 8.x-1.x for migrate_plus, and make an official release.

Thoughts?

heddn’s picture

I'm fine with an RC. It has good test coverage and I've done some manual testing. While I doubt that it has had any really good, real-life testing, the best way to get that is to roll an RC.

mikeryan’s picture

KarenS’s picture

It would be nice to add links on the project page to all these new projects. Nobody who was not following these discussions is going to find them :)

mikeryan’s picture

Status: Active » Fixed

Released!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

mpp’s picture

Nice work! Waiting for a "webservices" plugin that supports migrations from SOAP.
See https://www.drupal.org/node/2726443