Background

Starting with Drupal 8, major version upgrades are done with a migration path, rather than an upgrade path. Both 6 -> 8 and 7 -> 8 migrations are included in core. While migrations apply to moving from 3rd party systems and other use cases, this meta issue deals with older Drupal to newer Drupal migrations only. Per #2858592: *DRAFT* Proposed product goals for Drupal 8.4/8.5(+) core, this is the top product goal for Drupal 8.4.

Goals

  1. Provide a stable Migration API in core. (Core Migrate module marked stable.)
  2. Resolve all "Migrate critical" issues. These include a working Drupal 6 to Drupal 8 migration, a working Drupal 7 to Drupal 8 migration, and a shippable Drupal Migration UI. (Core Migrate Drupal and Migrate Drupal UI modules marked stable.)

Proposal roadmap

#2456259: [META] Drupal 7 to Drupal 8 Migration path to fix and introduce migrations from earlier Drupal versions.

Triaged migration issues are listed at https://docs.google.com/spreadsheets/d/16SwQsp7nF1UKrZQCNdPYML_wi53ZjM1e... (migrate triage? in IRC). This breaks things down in more detail.

Remaining work

These are roughly the priorities of remaining work:

  1. #2748609: [meta] Preserving auto-increment IDs on migration is fragile - An inherent consequence of Drupal's reliance on serial IDs is that attempting to preserve the "old" IDs in migration means that any attempt to create manual content in the destination system before migration is complete can cause collisions and data loss. It is critical to figure out how best to address this problem. A meeting is being scheduled for the week of March 20-24 to discuss this among the migration team and key core personnel.
  2. #2683435: CCK does not exist in Drupal 7, rename Migrate's cckfield plugins to field plugins - Because of the volume of code touched, and the desire to have other patches immediately start using the new APIs, this blocks a bunch of work due to fear of reroll hell.
  3. #2447727: Add base class for migrating reference fields - Blocked by the above patch, this blocks several reference field migrations.
  4. #2820490: FormatDate process plugin, #2566779: Migration D6 > D8 of CCK date fields - Date fields, along with the afore-mentioned reference fields, are the most important gap in Drupal-to-Drupal migration functionality.
  5. #2569805: For Drupal migration, identify the source module - Replaces the hardcoded list of migration source and destination modules with an automated method of determing them - getting this in means further backend issues no longer have to touch frontend code.
  6. UI issues
  7. Completing D6/D7 upgrade funcionality
  8. Fixing critical and major bugs
  9. #2561243: [meta] Migration system documentation

Communicating with the team #

Members of the team can be found in #drupal-migrate in Freenode IRC. Meetings are every week alternating between 8am and 8pm US Central time on Thursdays, focusing on triaging core 'migration system' issues. Thus, upcoming meetings as of March 17 2017 are:

  • 8pm CDT on Thursday June 22, 2017 (01:00 UTC Friday June 23 2017)
  • 8am CDT on Thursday June 29, 2017 (13:00 UTC)
  • 8pm CDT on Thursday July 6, 2017 (01:00 UTC Friday July 7 2017)
  • 8am CDT on Thursday July 13, 2017 (13:00 UTC)
  • 8pm CDT on Thursday July 20, 2017 (01:00 UTC Friday July 21 2017)

The migration system component is used to file all issues.

Not in scope

Issues not related to older Drupal migrations to newer Drupal versions.

Related work

Existing core features

Core contains three experimental modules related to migration:

  • Migrate: (beta) The underlying API for the migration system.
  • Migrate Drupal: (alpha) Underlying API for Drupal-to-Drupal migrations, currently including migration paths from Drupal 6 and Drupal 7 to Drupal 8
  • Migrate Drupal UI: (alpha) Provides a very simple user interface for running migrations from older versions of Drupal (similar to update.php).

It should also be noted that migration support for each individual core module is contained within that module, not within the experimental modules, although some of that support may still be considered experimental.

Open core issues

@todo (This would be things NOT in Migrate queues, but affected/related. E.g. the URL case sensitivity upgrade path issue.)

Contributed projects

The following are contributed projects in the migration system ecosystem, none slated for core inclusion at this time:

Team and resources

@todo: Define team members' roles here (e.g. initiative coordinator, etc.; see https://www.drupal.org/contribute/core/maintainers#initiative; compare https://www.drupal.org/node/2721129). List additional team members; there are many active Migrate contributors in addition to the maintainers that might be part of the team.

Signoffs

Signoffs needed

For any significant change: There needs to be a majority of Migration subsystem maintainers who agree to a given patch.

For most significant changes, including API changes and new migrations (?): Migration patches also need sign-off from framework managers.

For user-facing changes to Drupal Migrate UI, additionally the usability and accessibility topic maintainers, and product managers should be involved, since this is a critical site builder UI.

For any changes that are disruptive, that increase the technical debt/followups needed, or that make existing issues more significant: Release managers should sign off.

Signoffs given

The core committers already signed off on replacing the old update.php-based upgrade system with a migration system at DrupalCon Prague in 2013, and prioritizing 6->8 migrations first, during the Drupal 8 Leadership Q&A session.

Framework managers signed off on a major API change for 8.1.x to make migrations plugins instead of configuration entities (now committed with a few remaining followups).

Comments

Gábor Hojtsy created an issue. See original summary.

mikeryan’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Adding #2456259: [META] Drupal 7 to Drupal 8 Migration path which tracks all the issue for the migrations themselves.

Anonymous’s picture

Added the meetup time to my calendar. I should be able to contribute more as my schedule is less busy these days.

oadaeh’s picture

Now that I'm not driving around the Americas, I might have more time to get back into this (once I finish stabilizing). However, I will first have to re-learn everything that's going on here, because it's been about 1.5 years since I looked at it.

mikeryan’s picture

Issue summary: View changes
mikeryan’s picture

Issue summary: View changes

Updated the meeting schedule to reflect the new alternate-week times.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mikeryan’s picture

Issue summary: View changes
mikeryan’s picture

Issue summary: View changes

Fixed meeting times.

iMiksu’s picture

Issue summary: View changes

Updated meeting dates.

mikeryan’s picture

Issue summary: View changes

Updating meeting times - note that the U.S. has ended daylight savings time, UTC times are now 1 hour later.

mikeryan’s picture

Issue summary: View changes

s/CDT/CST/

Gábor Hojtsy’s picture

Project: Drupal core » Drupal core ideas
Version: 8.3.x-dev »
Component: migration system » Active Initiative

Moving to active initiative.

xjm’s picture

Issue summary: View changes

Confirming; Migrate is (of course) an active initiative. :) And, as decided in #2810347: [policy, no patch] Mark migrate.module as beta stability and announced in https://groups.drupal.org/node/515916, Migrate API is now in beta!

@catch, @cilefen, @alexpott, @effulgentsia, @Cottser, and I discussed the status of the Migrate experimental module suite. We agree that the current top priority is resolving critical issues with the D6 and D7 migration paths, and in turn getting Migrate Drupal to beta as well once those migration paths are feature-complete.

It's really impressive that the Migrate team is in this for the long haul. This initiative is really going to make a long-term difference for Drupal as a whole in addition to the current technical benefit. Thank you to all contributors and especially the maintainers for your ongoing work.

catch’s picture

I just had a look over https://www.drupal.org/project/issues/search?projects=&project_issue_fol...

#2830036: MigrationPluginManager::getDefinitions() blows up in node derivers is RTBC.

#2669964: Migrate Drupal 7 core node translations to Drupal 8 is 7.x only.

#2783423: Taxonomy terms not displayed after migration affects only already-upgrade sites since the root cause was fixed.

This makes me think that the remaining blocker to a stable 6-8 migration path is the following:

#2640496: Revision ID in {node} and {node_revision} can get out of sync / #2748609: [meta] Preserving auto-increment IDs on migration is fragile / #2826047: 6-8 user picture migration must create new fids, which can conflict with fids used by other migrations and sub-issues which are all related.

And finally #2505283: Handle import of private files.

With private files, since that's only on/off for 6.x, I don't think the migration path for private files should block the migration path for 6.x sites using public files (it should however block the 7.x migration path since sites can and do use both, and we'd need to document clearly that it's missing).

So unless there are critical 6.x migration issues in the queue that aren't tagged, what do people think about marking the 6.x migration path as beta/RC after the ID-preservation issues are fixed, then stable a while later (probably for 8.4.0 given 8.3.x is coming up soon).

Gábor Hojtsy’s picture

How do we mark specific version upgrade paths beta/stable when multiple versions are contained in the same module?

catch’s picture

@Gabor at a minimum we can do a big announcement and document it on the experimental modules page in the handbook. Could look at some messaging in the migrate UI depending on the source version too?

mikeryan’s picture

Issue summary: View changes

Freshening meetings times, and bringing the maintainers' list up-to-date.

mikeryan’s picture

Issue summary: View changes
heddn’s picture

Discussed this in the weekly migrate maintainers meeting. We feel that the D6 => D8 is very stable at this point and *could* be marked beta or RC. However, there isn't any good way to mark half a module as beta, since the D7 migrate path is not 100% complete.

mikeryan’s picture

Issue summary: View changes
webchick’s picture

Priority: Normal » Major

Escalating, based on impact.

catch’s picture

Wouldn't we need to sort out #2748609: [meta] Preserving auto-increment IDs on migration is fragile and related issues before it can go to RC? It might be OK for beta with a note that incremental migrations should be avoided.

Also not sure about #2746527: [META] Handle data related to Drupal 6 and 7 node translations with different IDs - obviously doesn't affect sites that aren't using 6.x content translation, but would think it's a blocker for those that do?

mikeryan’s picture

How complete/up-to-date should the documentation be to call this "stable"? See #2561243: [meta] Migration system documentation

mikeryan’s picture

Issue summary: View changes

I've updated the issue summary to reflect our discussion with webchick at last night's meeting (in particular adding the "Remaining Work" section) - Angie and the migrate team in particular should review and make sure I've captured things appropriately.

heddn’s picture

Issue summary: View changes
mikeryan’s picture

Issue summary: View changes

Updated meeting times.

Gábor Hojtsy’s picture

Issue summary: View changes

Updated meeting times and added anchor.