Virtual sprint

Come join us on Thursday January 11 and Friday January 12, 2018 to help get migrate stable. Details in #41.

Background

Starting with Drupal 8, major version upgrades are done with a migration path, rather than an upgrade path. Both 6 -> 8 and 7 -> 8 migrations are included in core. While migrations apply to moving from 3rd party systems and other use cases, this meta issue deals with older Drupal to newer Drupal migrations only. Per #2858592: *DRAFT* Proposed product goals for Drupal 8.4/8.5(+) core, this is the top product goal for Drupal 8.4.

Goals

  1. Provide a stable Migration API in core. (Core Migrate module marked stable.)
  2. Resolve all "Migrate critical" issues. These include a working Drupal 6 to Drupal 8 migration, a working Drupal 7 to Drupal 8 migration, and a shippable Drupal Migration UI. (Core Migrate Drupal and Migrate Drupal UI modules marked stable.)

Proposal roadmap

#2456259: [META] Drupal 7 to Drupal 8 Migration path to fix and introduce migrations from earlier Drupal versions.

Triaged migration issues are listed at https://docs.google.com/spreadsheets/d/16SwQsp7nF1UKrZQCNdPYML_wi53ZjM1e... (migrate triage? in IRC). This breaks things down in more detail.

Remaining work

These are roughly the priorities of remaining work:

  1. #2748609: [meta] Preserving auto-increment IDs on migration is fragile - An inherent consequence of Drupal's reliance on serial IDs is that attempting to preserve the "old" IDs in migration means that any attempt to create manual content in the destination system before migration is complete can cause collisions and data loss. It is critical to figure out how best to address this problem. A meeting is being scheduled for the week of March 20-24 to discuss this among the migration team and key core personnel.
  2. #2683435: CCK does not exist in Drupal 7, rename Migrate's cckfield plugins to field plugins - Because of the volume of code touched, and the desire to have other patches immediately start using the new APIs, this blocks a bunch of work due to fear of reroll hell.
  3. #2447727: Add base class for migrating reference fields - Blocked by the above patch, this blocks several reference field migrations.
  4. #2820490: FormatDate process plugin, #2566779: Migration D6 > D8 of CCK date fields - Date fields, along with the afore-mentioned reference fields, are the most important gap in Drupal-to-Drupal migration functionality.
  5. #2569805: For Drupal migration, identify the source module - Replaces the hardcoded list of migration source and destination modules with an automated method of determing them - getting this in means further backend issues no longer have to touch frontend code.
  6. UI issues
  7. Completing D6/D7 upgrade funcionality
  8. Fixing critical and major bugs
  9. #2561243: [meta] Migration system documentation

Communicating with the team #

Members of the team can be found in #drupal-migrate in Freenode IRC.

Weekly migrate meetings are held in google hangouts every week alternating between 2PM and 9PM UTC on Thursdays, focusing on triaging core 'migration system' issues. Please ping heddn in IRC to be added to the invite if interested in attending.

Thus, upcoming meetings for January are:

  • 2PM UTC on Thursday January 11, 2018
  • 9PM UTC on Thursday January 18, 2018
  • 2PM UTC on Thursday January 25, 2018

The migration system component is used to file all issues.

Not in scope

Issues not related to older Drupal migrations to newer Drupal versions.

Related work

Existing core features

Core contains three experimental modules related to migration:

  • Migrate: (beta) The underlying API for the migration system.
  • Migrate Drupal: (alpha) Underlying API for Drupal-to-Drupal migrations, currently including migration paths from Drupal 6 and Drupal 7 to Drupal 8
  • Migrate Drupal UI: (alpha) Provides a very simple user interface for running migrations from older versions of Drupal (similar to update.php).

It should also be noted that migration support for each individual core module is contained within that module, not within the experimental modules, although some of that support may still be considered experimental.

Open core issues

@todo (This would be things NOT in Migrate queues, but affected/related. E.g. the URL case sensitivity upgrade path issue.)

Contributed projects

The following are contributed projects in the migration system ecosystem, none slated for core inclusion at this time:

Team and resources

@todo: Define team members' roles here (e.g. initiative coordinator, etc.; see https://www.drupal.org/contribute/core/maintainers#initiative; compare https://www.drupal.org/node/2721129). List additional team members; there are many active Migrate contributors in addition to the maintainers that might be part of the team.

Signoffs

Signoffs needed

For any significant change: There needs to be a majority of Migration subsystem maintainers who agree to a given patch.

For most significant changes, including API changes and new migrations (?): Migration patches also need sign-off from framework managers.

For user-facing changes to Drupal Migrate UI, additionally the usability and accessibility topic maintainers, and product managers should be involved, since this is a critical site builder UI.

For any changes that are disruptive, that increase the technical debt/followups needed, or that make existing issues more significant: Release managers should sign off.

Signoffs given

The core committers already signed off on replacing the old update.php-based upgrade system with a migration system at DrupalCon Prague in 2013, and prioritizing 6->8 migrations first, during the Drupal 8 Leadership Q&A session.

Framework managers signed off on a major API change for 8.1.x to make migrations plugins instead of configuration entities (now committed with a few remaining followups).

Comments

Gábor Hojtsy created an issue. See original summary.

mikeryan’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Adding #2456259: [META] Drupal 7 to Drupal 8 Migration path which tracks all the issue for the migrations themselves.

Anonymous’s picture

Added the meetup time to my calendar. I should be able to contribute more as my schedule is less busy these days.

oadaeh’s picture

Now that I'm not driving around the Americas, I might have more time to get back into this (once I finish stabilizing). However, I will first have to re-learn everything that's going on here, because it's been about 1.5 years since I looked at it.

mikeryan’s picture

Issue summary: View changes
mikeryan’s picture

Issue summary: View changes

Updated the meeting schedule to reflect the new alternate-week times.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mikeryan’s picture

Issue summary: View changes
mikeryan’s picture

Issue summary: View changes

Fixed meeting times.

iMiksu’s picture

Issue summary: View changes

Updated meeting dates.

mikeryan’s picture

Issue summary: View changes

Updating meeting times - note that the U.S. has ended daylight savings time, UTC times are now 1 hour later.

mikeryan’s picture

Issue summary: View changes

s/CDT/CST/

Gábor Hojtsy’s picture

Project: Drupal core » Drupal core ideas
Version: 8.3.x-dev »
Component: migration system » Active Initiative

Moving to active initiative.

xjm’s picture

Issue summary: View changes

Confirming; Migrate is (of course) an active initiative. :) And, as decided in #2810347: [policy, no patch] Mark migrate.module as beta stability and announced in https://groups.drupal.org/node/515916, Migrate API is now in beta!

@catch, @cilefen, @alexpott, @effulgentsia, @Cottser, and I discussed the status of the Migrate experimental module suite. We agree that the current top priority is resolving critical issues with the D6 and D7 migration paths, and in turn getting Migrate Drupal to beta as well once those migration paths are feature-complete.

It's really impressive that the Migrate team is in this for the long haul. This initiative is really going to make a long-term difference for Drupal as a whole in addition to the current technical benefit. Thank you to all contributors and especially the maintainers for your ongoing work.

catch’s picture

I just had a look over https://www.drupal.org/project/issues/search?projects=&project_issue_fol...

#2830036: MigrationPluginManager::getDefinitions() blows up in node derivers is RTBC.

#2669964: Migrate Drupal 7 core node translations to Drupal 8 is 7.x only.

#2783423: Taxonomy terms not displayed after migration affects only already-upgrade sites since the root cause was fixed.

This makes me think that the remaining blocker to a stable 6-8 migration path is the following:

#2640496: Revision ID in {node} and {node_revision} can get out of sync / #2748609: [meta] Preserving auto-increment IDs on migration is fragile / #2826047: 6-8 user picture migration must create new fids, which can conflict with fids used by other migrations and sub-issues which are all related.

And finally #2505283: Handle import of private files.

With private files, since that's only on/off for 6.x, I don't think the migration path for private files should block the migration path for 6.x sites using public files (it should however block the 7.x migration path since sites can and do use both, and we'd need to document clearly that it's missing).

So unless there are critical 6.x migration issues in the queue that aren't tagged, what do people think about marking the 6.x migration path as beta/RC after the ID-preservation issues are fixed, then stable a while later (probably for 8.4.0 given 8.3.x is coming up soon).

Gábor Hojtsy’s picture

How do we mark specific version upgrade paths beta/stable when multiple versions are contained in the same module?

catch’s picture

@Gabor at a minimum we can do a big announcement and document it on the experimental modules page in the handbook. Could look at some messaging in the migrate UI depending on the source version too?

mikeryan’s picture

Issue summary: View changes

Freshening meetings times, and bringing the maintainers' list up-to-date.

mikeryan’s picture

Issue summary: View changes
heddn’s picture

Discussed this in the weekly migrate maintainers meeting. We feel that the D6 => D8 is very stable at this point and *could* be marked beta or RC. However, there isn't any good way to mark half a module as beta, since the D7 migrate path is not 100% complete.

mikeryan’s picture

Issue summary: View changes
webchick’s picture

Priority: Normal » Major

Escalating, based on impact.

catch’s picture

Wouldn't we need to sort out #2748609: [meta] Preserving auto-increment IDs on migration is fragile and related issues before it can go to RC? It might be OK for beta with a note that incremental migrations should be avoided.

Also not sure about #2746527: [META] Handle data related to Drupal 6 and 7 node translations with different IDs - obviously doesn't affect sites that aren't using 6.x content translation, but would think it's a blocker for those that do?

mikeryan’s picture

How complete/up-to-date should the documentation be to call this "stable"? See #2561243: [meta] Migration system documentation

mikeryan’s picture

Issue summary: View changes

I've updated the issue summary to reflect our discussion with webchick at last night's meeting (in particular adding the "Remaining Work" section) - Angie and the migrate team in particular should review and make sure I've captured things appropriately.

heddn’s picture

Issue summary: View changes
mikeryan’s picture

Issue summary: View changes

Updated meeting times.

Gábor Hojtsy’s picture

Issue summary: View changes

Updated meeting times and added anchor.

webchick’s picture

Had a very productive meeting today with Migrate team members @heddn, @quietone (for a bit), @maxocub, and @phenaproxima, as well as @larowlan! Here were the key takeaways:

Next steps:

  • Work with employers to secure more contributors' time to get these key blocker issues banged out. [@webchick/@Acquia]
  • Hold a sprint in Vienna around remaining migrate issues with @quietone and (hopefully) @maxocub in attendance, @heddn participating remotely.
  • @larowlan has helpfully volunteered to get pings from folks on this initiative when things are awaiting reviews. :)
  • Mark Migrate UI "beta" stability before 8.4.0 (@heddn) to indicate its completion level.

Whew! GO TEAM! :D

quietone’s picture

Apologies for my short attendance at the meeting. The cause was (well, still is) a power outage scheduled from 10am to 4pm. Somehow, we missed the notification from the electricity provider. My neighbor told me it was an unremarkable, small postcard. When it happened I opted to find out what was the problem was and, of course, that took some time with automated answering services to deal with.

catch’s picture

Thanks for the summary, this matches the impression I've had from the issue queue.

My general feeling is that once #2842222: D7 Plain text fields incorrectly migrated to D8 as Text (formatted) is fixed we could move migrate as a whole to beta - since the 6.x and 7.x migrations should both be reliable at that point.

Then #2679739: [Policy, no patch] Make the migrate_drupal_ui module part of the migrate_drupal module and #2748609: [meta] Preserving auto-increment IDs on migration is fragile probably need to block an rc since they significantly impact the most and least complex migration paths respectively, but I don't see a reason they should block a beta unless I've missed something (migrate_drupal_ui might need to stay as an empty hidden module for bc maybe).

Does that sounds about right?

heddn’s picture

I've linked in plan issues for each of the 3 sub-module's plans with linked issues from them on what are considered blocking issues. We're down to ~10 issues in all to call the entire migrate sub-system stable. This is after several years, 1000+opened issues (800+ are already closed) and much effort by too many people to name.

However, let's not start celebrating yet. Let's dive into these child issues and see if we can get Migrate as an ecosystem stable before DrupalCon Vienna.

heddn’s picture

Issue summary: View changes
masipila’s picture

Issue summary: View changes

I added #2906470: Orphan comments and entries in comment_entity_statistics after comment field instance has been deleted to the list of todos elsewhere in core + added myself as documentation maintainer.

heddn’s picture

In today's maintainer meeting attended by Gábor Hojtsy, maxocub, jofitz, phenaproxima and rakeshjames we discussed as always the state of migrate. Our hope is to resolve the 3 remaining migrate_drupal_ui.module and 1 migrate.module stable blocking issues before Christmas. This is pretty do-able we feel. After these issues get resolved, there is nothing stopping those two modules going stable. That would be a super nice present to ourselves.

Then, to get migrate_drupal over the finish line we are tentatively planning a virtual sprint day on January 11-12, 2018 (Thursday/Friday). We had verbal commitment from many on the call that they think they can attend. Next up is for all of us to line up support from our employers and get a committer to be available to review and commit patches.

webchick’s picture

Awesome news, yay! :D Please let us know how we can help!

heddn’s picture

Sprint attendance confirmed so far:

heddn
quieteone
alexpott (committer)

Thanks to AcroMedia for the sponsored time!!!

I'm sure others will also confirm as the data approaches.

webchick’s picture

Is this virtual sprint something you'd like some promotion around? (E.g. on Twitter, Drupal Planet, g.d.o/core, etc.) or do you want to keep it to a relatively small group of people to really focus on?

Also, can funding be helpful in any way? (Not promising anything, but always have to check! :))

maxocub’s picture

I will also attend the sprint. I'll confirm with my employer, but I'm sure it'll be OK.

heddn’s picture

So, for the 38 or so followers on this issue, here's some details on the virtual sprint planned for next week.

We'll meet in IRC in #drupal-migrate. Take a look at https://contribkanban.com/board/migrate and filter by Critical for the most important things to work on. If things are already being worked on, filter for needs work or needs review in the issue queue and start plugging away at tasks. You can always ask for assistance or bounce questions in IRC.

We'll have our normal weekly call at 2PM GMT on Thursday January 11, 2018. A link to the hangout will be posted in IRC. The goal of the whole sprint is to see serious progress across the board on issues that are blocking stability of Drupal 8's migrate system. With 4-5 remaining issues... we are very close.

If you still want to help but don't think you can take on anything super time consuming or complicated, please ask. There are 290 (as of this moment) issues in the migrate issue queue that are in some type of open status. We'll hook you up.

heddn’s picture

quietone’s picture

Issue summary: View changes

Update meeting times and add notice at the top of the issue regarding the sprint.

heddn’s picture

Drum roll please!!!! Of the 3 modules in the migrate subsystem... one went stable in 8.5.x as of a few hours ago!

See #2931165: Mark Migrate (API) as stable

If you want to help finish this off, visit https://contribkanban.com/board/migrate and filter for Criticals. There's only a few left to finish up the remaining 2 core modules, namely Migrate Drupal and Migrate Drupal UI.

heddn’s picture

Also note, if you are not part of the 20% Drupal market share that is running an i18n Drupal website. And if you plan to use drush instead of the Migrate Drupal UI, the API is /already stable/. For that +/- 80% market share... happy upgrading.

heddn’s picture

And the final issue blocker for migrate_drupal_ui just got committed against 8.5.x. So the only thing blocking it from going stable at this point is that it depends on an unstable module. Meaning, migrate_drupal. Otherwise, the UI is now as stable should be considered Un-Officially stable!!! Go out and migrate using it. Especially if you aren't migrating any i18n data.

heddn’s picture

Issue summary: View changes