[META] Stabilize the older Drupal to newer Drupal migration system

Now that I'm not driving around the Americas, I might have more time to get back into this (once I finish stabilizing). However, I will first have to re-learn everything that's going on here, because it's been about 1.5 years since I looked at it.

Updated the meeting schedule to reflect the new alternate-week times.

Fixed meeting times.

Updated meeting dates.

Updating meeting times - note that the U.S. has ended daylight savings time, UTC times are now 1 hour later.

s/CDT/CST/

Confirming; Migrate is (of course) an active initiative. :) And, as decided in #2810347: [policy, no patch] Mark migrate.module as beta stability and announced in https://groups.drupal.org/node/515916, Migrate API is now in beta!

@catch, @cilefen, @alexpott, @effulgentsia, @Cottser, and I discussed the status of the Migrate experimental module suite. We agree that the current top priority is resolving critical issues with the D6 and D7 migration paths, and in turn getting Migrate Drupal to beta as well once those migration paths are feature-complete.

It's really impressive that the Migrate team is in this for the long haul. This initiative is really going to make a long-term difference for Drupal as a whole in addition to the current technical benefit. Thank you to all contributors and especially the maintainers for your ongoing work.

I just had a look over https://www.drupal.org/project/issues/search?projects=&project_issue_fol...

#2830036: MigrationPluginManager::getDefinitions() blows up in node derivers is RTBC.

#2669964: Migrate Drupal 7 core node translations to Drupal 8 is 7.x only.

#2783423: Taxonomy terms not displayed after migration affects only already-upgrade sites since the root cause was fixed.

This makes me think that the remaining blocker to a stable 6-8 migration path is the following:

#2640496: Revision ID in {node} and {node_revision} can get out of sync / #2748609: [meta] Preserving auto-increment IDs on migration is fragile / #2826047: 6-8 user picture migration must create new fids, which can conflict with fids used by other migrations and sub-issues which are all related.

And finally #2505283: Handle import of private files.

With private files, since that's only on/off for 6.x, I don't think the migration path for private files should block the migration path for 6.x sites using public files (it should however block the 7.x migration path since sites can and do use both, and we'd need to document clearly that it's missing).

So unless there are critical 6.x migration issues in the queue that aren't tagged, what do people think about marking the 6.x migration path as beta/RC after the ID-preservation issues are fixed, then stable a while later (probably for 8.4.0 given 8.3.x is coming up soon).

@Gabor at a minimum we can do a big announcement and document it on the experimental modules page in the handbook. Could look at some messaging in the migrate UI depending on the source version too?

Freshening meetings times, and bringing the maintainers' list up-to-date.

Discussed this in the weekly migrate maintainers meeting. We feel that the D6 => D8 is very stable at this point and *could* be marked beta or RC. However, there isn't any good way to mark half a module as beta, since the D7 migrate path is not 100% complete.

Escalating, based on impact.

Wouldn't we need to sort out #2748609: [meta] Preserving auto-increment IDs on migration is fragile and related issues before it can go to RC? It might be OK for beta with a note that incremental migrations should be avoided.

Also not sure about #2746527: [META] Handle data related to Drupal 6 and 7 node translations with different IDs - obviously doesn't affect sites that aren't using 6.x content translation, but would think it's a blocker for those that do?

How complete/up-to-date should the documentation be to call this "stable"? See #2561243: [meta] Migration system documentation

I've updated the issue summary to reflect our discussion with webchick at last night's meeting (in particular adding the "Remaining Work" section) - Angie and the migrate team in particular should review and make sure I've captured things appropriately.

Updated meeting times.

Had a very productive meeting today with Migrate team members @heddn, @quietone (for a bit), @maxocub, and @phenaproxima, as well as @larowlan! Here were the key takeaways:

• This initiative has several accomplishments to toot about in 8.4; Dates and Node References now have migration paths; lots of multilingual improvements; migrations converted to use new Field Plugins. At this point, migrations are possible (with a bit of Jr. Dev-level massaging) in Drush (though not yet in the UI). D6 => D8 should be considered “beta” level stability, although no way to mark that as such, since both D6 -> D8 and D7 -> D8 are under "Migrate Drupal" module.
• The team set out a list of 3 goals for 8.5: namely to get all the migrate experimental modules to stable, so we can announce to the general public that they can confidently move to D8 (this is one of the top D8 adoption blockers atm):
  • Migrate API is predominantly blocked on #2876085: Before upgrading, audit for potential ID conflicts. We need to confirm that revisions & i18n can be fixed with the current solution. If so, can be follow-ups. Also needs a fix for Drush as well as the UI.
  • Migrate Drupal is predominantly blocked on #2842222: D7 Plain text fields incorrectly migrated to D8 as Text (formatted) (which in turn is blocked by #2893061: Create a ProcessField plugin to process the field types. This is critical to sort out before site builders can widely use the migration path, because it can unwittingly cause Filtered HTML to get incorrectly mapped as Full HTML, causing a security issue.
  • Migrate UI is actually fairly close. The team needs a decision at #2679739: [Policy, no patch] Make the migrate_drupal_ui module part of the migrate_drupal module and to implement a last bit of usability feedback at #2678638-42: [META] Usability refinements for Migrate UI.
• For 8.6, most of the focus will be around getting the Drush integration in https://www.drupal.org/project/migrate_tools into "upstream" Drush proper, so a separate module is not needed. No more core work, at least in the short-term.

Next steps:

• Work with employers to secure more contributors' time to get these key blocker issues banged out. [@webchick/@Acquia]
• Hold a sprint in Vienna around remaining migrate issues with @quietone and (hopefully) @maxocub in attendance, @heddn participating remotely.
• @larowlan has helpfully volunteered to get pings from folks on this initiative when things are awaiting reviews. :)
• Mark Migrate UI "beta" stability before 8.4.0 (@heddn) to indicate its completion level.

Whew! GO TEAM! :D

Apologies for my short attendance at the meeting. The cause was (well, still is) a power outage scheduled from 10am to 4pm. Somehow, we missed the notification from the electricity provider. My neighbor told me it was an unremarkable, small postcard. When it happened I opted to find out what was the problem was and, of course, that took some time with automated answering services to deal with.

Thanks for the summary, this matches the impression I've had from the issue queue.

My general feeling is that once #2842222: D7 Plain text fields incorrectly migrated to D8 as Text (formatted) is fixed we could move migrate as a whole to beta - since the 6.x and 7.x migrations should both be reliable at that point.

Then #2679739: [Policy, no patch] Make the migrate_drupal_ui module part of the migrate_drupal module and #2748609: [meta] Preserving auto-increment IDs on migration is fragile probably need to block an rc since they significantly impact the most and least complex migration paths respectively, but I don't see a reason they should block a beta unless I've missed something (migrate_drupal_ui might need to stay as an empty hidden module for bc maybe).

Does that sounds about right?

I've linked in plan issues for each of the 3 sub-module's plans with linked issues from them on what are considered blocking issues. We're down to ~10 issues in all to call the entire migrate sub-system stable. This is after several years, 1000+opened issues (800+ are already closed) and much effort by too many people to name.

However, let's not start celebrating yet. Let's dive into these child issues and see if we can get Migrate as an ecosystem stable before DrupalCon Vienna.

I added #2906470: Orphan comments and entries in comment_entity_statistics after comment field instance has been deleted to the list of todos elsewhere in core + added myself as documentation maintainer.

In today's maintainer meeting attended by Gábor Hojtsy, maxocub, jofitz, phenaproxima and rakeshjames we discussed as always the state of migrate. Our hope is to resolve the 3 remaining migrate_drupal_ui.module and 1 migrate.module stable blocking issues before Christmas. This is pretty do-able we feel. After these issues get resolved, there is nothing stopping those two modules going stable. That would be a super nice present to ourselves.

Then, to get migrate_drupal over the finish line we are tentatively planning a virtual sprint day on January 11-12, 2018 (Thursday/Friday). We had verbal commitment from many on the call that they think they can attend. Next up is for all of us to line up support from our employers and get a committer to be available to review and commit patches.

Awesome news, yay! :D Please let us know how we can help!

Sprint attendance confirmed so far:

heddn
quieteone
alexpott (committer)

Thanks to AcroMedia for the sponsored time!!!

I'm sure others will also confirm as the data approaches.

Is this virtual sprint something you'd like some promotion around? (E.g. on Twitter, Drupal Planet, g.d.o/core, etc.) or do you want to keep it to a relatively small group of people to really focus on?

Also, can funding be helpful in any way? (Not promising anything, but always have to check! :))

I will also attend the sprint. I'll confirm with my employer, but I'm sure it'll be OK.

So, for the 38 or so followers on this issue, here's some details on the virtual sprint planned for next week.

We'll meet in IRC in #drupal-migrate. Take a look at https://contribkanban.com/board/migrate and filter by Critical for the most important things to work on. If things are already being worked on, filter for needs work or needs review in the issue queue and start plugging away at tasks. You can always ask for assistance or bounce questions in IRC.

We'll have our normal weekly call at 2PM GMT on Thursday January 11, 2018. A link to the hangout will be posted in IRC. The goal of the whole sprint is to see serious progress across the board on issues that are blocking stability of Drupal 8's migrate system. With 4-5 remaining issues... we are very close.

If you still want to help but don't think you can take on anything super time consuming or complicated, please ask. There are 290 (as of this moment) issues in the migrate issue queue that are in some type of open status. We'll hook you up.

Here's a tweet to share: https://twitter.com/lucashedding/status/949041403555872768

Update meeting times and add notice at the top of the issue regarding the sprint.

Drum roll please!!!! Of the 3 modules in the migrate subsystem... one went stable in 8.5.x as of a few hours ago!

See #2931165: Mark Migrate (API) as stable

If you want to help finish this off, visit https://contribkanban.com/board/migrate and filter for Criticals. There's only a few left to finish up the remaining 2 core modules, namely Migrate Drupal and Migrate Drupal UI.

Also note, if you are not part of the 20% Drupal market share that is running an i18n Drupal website. And if you plan to use drush instead of the Migrate Drupal UI, the API is /already stable/. For that +/- 80% market share... happy upgrading.

And the final issue blocker for migrate_drupal_ui just got committed against 8.5.x. So the only thing blocking it from going stable at this point is that it depends on an unstable module. Meaning, migrate_drupal. Otherwise, the UI is now as stable should be considered Un-Officially stable!!! Go out and migrate using it. Especially if you aren't migrating any i18n data.

Remove notice about virtual sprint

Update the communicating with the team section.

Things are winding down (finally) for migrate. Drupal 8.6 sees migrate drupal and migrate drupal UI now are stable. We still have some small gaps in D7 core upgrade path (non-multilingual). See #2456259: [META] Drupal 7 to Drupal 8 Migration path for those details. We also have multilingual issues over in #2208401: [META] Remaining multilingual migration paths. But life is looking good.

Calendar link update.

(fixing my calendar link fix)

I'm inclined to say this is RTBC and can be closed/fixed. Migrate is now marked stable across the board. There are still things to do w/ migrate. Yes. Absolutely. But considering migrate is one of the oldest initiatives for Drupal, maybe it is time to call it done.

+1 for RTBC/closure. The migration system has had a long and turbulent journey, but now all the core migration paths are considered stable. So the scope of this issue is fulfilled.

Related TBD

I've just opened #3120103: Remove migrate initiative from drupal.org as an active initiative to close down the 6+ year initiative. That doesn't mean we don't still have more to do or that everything is fixed and beautiful/wonderful. There's still a few problems. (Some might argue there's a lot of problems) But the huge drive to create a stable upgrade path at long last is now finished (or at the very least it is debatable that it is finished).

HUZZAH! Congrats!

+1 to closing this one. We have #2456259: [META] Drupal 7 to Drupal 8 Migration path and #2208401: [META] Remaining multilingual migration paths open and between those two issues, most of what's in the issue summary is covered already.