Problem/Motivation

We need to build an upgrade path for Drupal 7 to Drupal 8.

There is much good work in the sandbox repository and its issues, but the sandbox is woefully behind HEAD. We can still use what's in there, but we'll need to a lot of re-rolling/refactoring.

The work done in the sandbox can be divided into two parts:

  1. D7->D8 stuff committed to the sandbox repo - this is what is contained in the patch that kedechant posted below in comment 2.
  2. D7->D8 patches in the sandbox issue queue, but not committed to the sandbox repo. There is **a lot of stuff** here. I spoke with Eduardo (Enzo) Garcia at DrupalCon Los Angeles about this - he and his fellow developers have most of D7->D8 **already working** and they have contributed it all back via patches posted in the sandbox. It appears that these patches contains mainly source and process plugins and migration configurations (no tests :( ). Regardless, between kdechant's patch and these sandbox patches, a good portion of D7->D8 may be ready for review and testing (and automated tests).

Proposed Resolution

The Migrate API, and the Drupal 6 upgrade path, are stable enough start getting an upgrade path from Drupal 7 into core.

I propose we attack this by module. Modules are a clean, understandable way of separating functionality, and I think they'd be a sensible way to group related migrations. There can be a separate issue for each core module that needs an upgrade path.

Remaining tasks

Certain parts of this upgrade path are decidedly more important than others. I've divided the list of core D7 modules into three tiers of priority:

Top Priority

Medium Priority

Low Priority

These migrations will mostly involve moving variables into configuration and are generally appropriate for new contributors.

API changes

Perhaps some, depending on #2462233: Migrations should not use the configuration entity system or #2463909: Migrations should support non-installed default configurations (templates).

Files: 
CommentFileSizeAuthor
#7 interdiff-2456259-6-7.txt2.85 KBphenaproxima
#7 2456259-7.patch56.53 KBphenaproxima
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 95,688 pass(es), 4 fail(s), and 0 exception(s). View
#6 2456259-6.patch54.27 KBphenaproxima
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Invalid PHP syntax in core/modules/migrate_drupal/src/Tests/d7/MigrateFieldInstanceTest.php. View
#5 2456259-5.patch76.27 KBphenaproxima
#2 drupal-d7-d8-migrations-2456259-2.patch29.78 KBkdechant

Comments

benjy’s picture

Issue summary: View changes
kdechant’s picture

Here is a patch of all the D7 migration definitions, source plugins, and process plugins from the old IMP sandbox. I have personally tested the d7_user, d7_taxonomy_vocabulary, d7_taxonomy_term, and d7_file migrations. I have not tested the other migrations (d7_field, d7_field_instance, and the variables).

ultimike’s picture

Issue summary: View changes
kdechant’s picture

I'm looking at #2469623: Process for creating migration source DBs for automated tests and I realized that my patch in #2 is breaking the SimpleTest tests created in that issue's patches. I might need to get the D7 source DB situation figured out sooner rather than later. Stay tuned.

phenaproxima’s picture

FileSize
76.27 KB

Interim patch that tunes up #2 and brings it in line with HEAD. And adds tests for the migrations therein. I'm not marking this for review because it won't pass the testbot (although the tests I wrote in there do pass, but they depend on another patch that is currently RTBC).

I offer this up for initial review, to make sure that I'm on the right track.

This depends on and extends the patch at https://www.drupal.org/node/2410625#comment-9948249.

phenaproxima’s picture

Status: Active » Needs review
FileSize
54.27 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Invalid PHP syntax in core/modules/migrate_drupal/src/Tests/d7/MigrateFieldInstanceTest.php. View

Updated patch, with moar assertions. These tests should now pass testbot.

phenaproxima’s picture

FileSize
56.53 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 95,688 pass(es), 4 fail(s), and 0 exception(s). View
2.85 KB

Updated the patch with a test of the d7_file migration.

phenaproxima’s picture

Status: Needs review » Active

I have begun splitting patch #7 into smaller, piecemeal patches in the various child issues. I like the idea of being able to get this done a bit at a time, instead of waiting on one giant patch which will take eons to merge into 8.0.x.

The last submitted patch, 6: 2456259-6.patch, failed testing.

Status: Active » Needs work

The last submitted patch, 7: 2456259-7.patch, failed testing.

phenaproxima’s picture

Status: Needs work » Active

This patch no longer needs to be tested, since I've split it out into several smaller sub-issues.

phenaproxima’s picture

phenaproxima’s picture

As I've started fiddling with the child issues, writing tests for them and trying to bring them in line with HEAD, I've found two hard blockers before any of them can proceed:

#2495755: Create MigrateDrupal7TestBase
Every single D7-D8 migration will of course need a test. Those tests will need a base class.

#2495851: migrate_source data type needs to define the constants mapping
This one, I'm less certain about. When writing tests for the D7-D8 migrations on my localhost, I kept running to configuration schema exceptions. The solution I present in that issues solves it for me, but I have to confess that I don't really know why it solves it, or what the real nature of the problem is.

Both these issues need to be resolved before we can start committing D7-D8 migrations into core.

phenaproxima’s picture

phenaproxima’s picture

Issue summary: View changes

Added a prioritized list of D7 core modules to the issue summary.

phenaproxima’s picture

Issue summary: View changes

Updated the issue summary again.

phenaproxima’s picture

Issue summary: View changes

Crossed out modules that were taken out of core.

phenaproxima’s picture

Issue summary: View changes

Added dedicated issues for low-priority modules.

phenaproxima’s picture

Issue summary: View changes

Added Views to the list of high priority modules.

phenaproxima’s picture

Issue summary: View changes

Added an issue for the Profile module.

phenaproxima’s picture

Issue summary: View changes

On second thought, the System module is pretty damn major.

phenaproxima’s picture

Issue summary: View changes
webchick’s picture

Priority: Normal » Major
Issue tags: +Migrate critical

Adding standard metadata for "Big, Important Migrate Stuff"™

quietone’s picture

Issue summary: View changes

Change link to Comment issue to the current one, not the duplicate

webchick’s picture

Just some house-keeping, ignore me.

googletorp’s picture

Issue summary: View changes
benjy’s picture

Issue summary: View changes

Fixed the link.

googletorp’s picture

Issue summary: View changes

Put new link into correct sorting.

skyredwang’s picture

I just tested a 7.39->8.0.0-rc3 upgrade/migrate. Everything worked out expectedly, except that Image Field doesn't get migrated. More specifically: image fields got created in D8, field files were copied to the new site, but no value were created or stored in the new image fields in nodes. Drush reported some errors below, but I don't see them directly related to image fields:

Upgrading d7_filter_format
Missing filter plugin: filter_null.                                                                                                               [error]
The "text_plain" plugin does not exist. (/var/www/html/core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php:57)                          [error]
The "link_url" plugin does not exist. (/var/www/html/core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php:57)                            [error]
The "i18n_taxonomy_term_reference_link" plugin does not exist. (/var/www/html/core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php:57)   [error]
The "i18n_taxonomy_term_reference_link" plugin does not exist. (/var/www/html/core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php:57)   [error]

I am not sure where to debug this broken image field migration. Could anyone point the direction?

skyredwang’s picture

Found the related issue. I didn't realize it's a task: #2604484: Migrate Drupal 7 image and file fields

googletorp’s picture

This is probably not the right place to talk about random migration errors.

Anyways, regarding your errors.

Missing filter plugin: filter_null.

I'm not sure about this one, there is a filter_null but it's not usable. You need to save the filters that has this and they will be usable. filter_null plugin is intended for migrations.

text_plain, link_url, i18n_taxonomy_term_reference_link missing plugins is probably all field formatters that doesn't exist. I'm not sure about text_plain, but the others I know. This code snippet can probably help you, it removes the errors but I haven't tested much on the data. Basically you add some mapping from D7 to D8 field formatters:

/**
 * Implement hook_ENTITY_TYPE_create().
 */
function module_migration_create($migration) {
  if ($migration->id() == 'd7_field_formatter_settings') {
    $process = $migration->getProcess();

    if (!empty($process['options/type'][0]['map']) && empty($process['options/type'][0]['map']['link_plain'])) {
      $process['options/type'][0]['map']['link_plain'] = 'link';
    }
    if (!empty($process['options/type'][0]['map']) && empty($process['options/type'][0]['map']['link_url'])) {
      $process['options/type'][0]['map']['link_url'] = 'link';
    }
    if (!empty($process['options/type'][0]['map']) && empty($process['options/type'][0]['map']['taxonomy_term_reference_plain'])) {
      $process['options/type'][0]['map']['taxonomy_term_reference_plain'] = 'entity_reference_label';
    }
    if (!empty($process['options/type'][0]['map']) && empty($process['options/type'][0]['map']['i18n_taxonomy_term_reference_link'])) {
      $process['options/type'][0]['map']['i18n_taxonomy_term_reference_link'] = 'entity_reference_label';
    }

    $migration->setProcess($process);
  }
}
mikeryan’s picture

mauriciomedeiros’s picture

Hi guys,

I am kinda lost here. I am trying to figure it out how to migrate books from d7 to d8, but the drupal 8.0.2 does not migrate it yet and I am having no luck trying to get the necessary files to patch it from the issues listed in this post: https://www.drupal.org/node/2409435. Looks like someone got it already :)

Can somebody help me out please?

Thanks in advance!

Mauricio

Jo Fitzgerald’s picture

I have come across some bugs when migrating users from D7 that do not have a language set - I have created an issue for this:
#2671312: No default value for User langcode when migrating D7 users with no language

Jo Fitzgerald’s picture

Jo Fitzgerald’s picture

Here is issue regarding migrating custom fields on users:
#2674152: D7 User fields content not migrating first time

quietone’s picture

Issue tags: +migrate-d7-d8

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

DocDJ-forum’s picture

Hello,
Let me 1st apologize if this is not the correct place to post my migration problem.

I'm a fairly new user, trying to migrate my development 7.43 database to my clean install of 8.1. I used Drush migrate_upgrade --legacy (etc.).
My structure and block definitions got migrated, but there was no content copied over. I have checked the forums and this set of postings and have not seen anything about how to proceed to determine whether my content got copied and how to enable it or to find out what I've done wrong.
I am hosting both versions (7.43, 8.1) on my own PC, running Windows 10 Pro.
Any tips you can offer will be greatly appreciated.

DocDJ-forum’s picture

I have found that some of my content does appear to have migrated, but my "front" page layout is not being applied. ONE navigation link (left sidebar) for HOME took me to SOME of that layout, but it is incomplete.

steinmb’s picture

@DocDJ-forum please create a new issue. This is a [META] – issue tracking the overall work on migration and not for random support req.

Gábor Hojtsy’s picture

Is this issue still being used to track migrations development? Adding the master issue for the migration system as parent.

hussainweb’s picture

DocDJ-forum’s picture

I apologize (again) for posting here. It was unclear to me that this forum was only for developers. I have posted my problems and help-requests elsewhere.

hussainweb’s picture

@DocDJ-forum: Just clarifying FYI, this forum is for everyone, but this issue was only meant to track the overall work on migration. You are very welcome to create a new 'Support Request' issue in this forum (issue queue) to get help. :) We do this so that we stay on topic and don't get distracted with unrelated conversation.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.