[META] Drupal 7 to Drupal 8 Migration path

Here is a patch of all the D7 migration definitions, source plugins, and process plugins from the old IMP sandbox. I have personally tested the d7_user, d7_taxonomy_vocabulary, d7_taxonomy_term, and d7_file migrations. I have not tested the other migrations (d7_field, d7_field_instance, and the variables).

I'm looking at #2469623: Process for creating migration source DBs for automated tests and I realized that my patch in #2 is breaking the SimpleTest tests created in that issue's patches. I might need to get the D7 source DB situation figured out sooner rather than later. Stay tuned.

Interim patch that tunes up #2 and brings it in line with HEAD. And adds tests for the migrations therein. I'm not marking this for review because it won't pass the testbot (although the tests I wrote in there do pass, but they depend on another patch that is currently RTBC).

I offer this up for initial review, to make sure that I'm on the right track.

This depends on and extends the patch at https://www.drupal.org/node/2410625#comment-9948249.

Updated patch, with moar assertions. These tests should now pass testbot.

Updated the patch with a test of the d7_file migration.

I have begun splitting patch #7 into smaller, piecemeal patches in the various child issues. I like the idea of being able to get this done a bit at a time, instead of waiting on one giant patch which will take eons to merge into 8.0.x.

This patch no longer needs to be tested, since I've split it out into several smaller sub-issues.

As I've started fiddling with the child issues, writing tests for them and trying to bring them in line with HEAD, I've found two hard blockers before any of them can proceed:

#2495755: Create MigrateDrupal7TestBase
Every single D7-D8 migration will of course need a test. Those tests will need a base class.

#2495851: migrate_source data type needs to define the constants mapping
This one, I'm less certain about. When writing tests for the D7-D8 migrations on my localhost, I kept running to configuration schema exceptions. The solution I present in that issues solves it for me, but I have to confess that I don't really know why it solves it, or what the real nature of the problem is.

Both these issues need to be resolved before we can start committing D7-D8 migrations into core.

Added a prioritized list of D7 core modules to the issue summary.

Updated the issue summary again.

Crossed out modules that were taken out of core.

Added dedicated issues for low-priority modules.

Added Views to the list of high priority modules.

Added an issue for the Profile module.

On second thought, the System module is pretty damn major.

Adding standard metadata for "Big, Important Migrate Stuff"™

Change link to Comment issue to the current one, not the duplicate

Just some house-keeping, ignore me.

Put new link into correct sorting.

I just tested a 7.39->8.0.0-rc3 upgrade/migrate. Everything worked out expectedly, except that Image Field doesn't get migrated. More specifically: image fields got created in D8, field files were copied to the new site, but no value were created or stored in the new image fields in nodes. Drush reported some errors below, but I don't see them directly related to image fields:

Upgrading d7_filter_format
Missing filter plugin: filter_null.                                                                                                               [error]
The "text_plain" plugin does not exist. (/var/www/html/core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php:57)                          [error]
The "link_url" plugin does not exist. (/var/www/html/core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php:57)                            [error]
The "i18n_taxonomy_term_reference_link" plugin does not exist. (/var/www/html/core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php:57)   [error]
The "i18n_taxonomy_term_reference_link" plugin does not exist. (/var/www/html/core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php:57)   [error]

I am not sure where to debug this broken image field migration. Could anyone point the direction?

Found the related issue. I didn't realize it's a task: #2604484: Migrate Drupal 7 image and file fields

This is probably not the right place to talk about random migration errors.

Anyways, regarding your errors.

Missing filter plugin: filter_null.

I'm not sure about this one, there is a filter_null but it's not usable. You need to save the filters that has this and they will be usable. filter_null plugin is intended for migrations.

text_plain, link_url, i18n_taxonomy_term_reference_link missing plugins is probably all field formatters that doesn't exist. I'm not sure about text_plain, but the others I know. This code snippet can probably help you, it removes the errors but I haven't tested much on the data. Basically you add some mapping from D7 to D8 field formatters:

/**
 * Implement hook_ENTITY_TYPE_create().
 */
function module_migration_create($migration) {
  if ($migration->id() == 'd7_field_formatter_settings') {
    $process = $migration->getProcess();

    if (!empty($process['options/type'][0]['map']) && empty($process['options/type'][0]['map']['link_plain'])) {
      $process['options/type'][0]['map']['link_plain'] = 'link';
    }
    if (!empty($process['options/type'][0]['map']) && empty($process['options/type'][0]['map']['link_url'])) {
      $process['options/type'][0]['map']['link_url'] = 'link';
    }
    if (!empty($process['options/type'][0]['map']) && empty($process['options/type'][0]['map']['taxonomy_term_reference_plain'])) {
      $process['options/type'][0]['map']['taxonomy_term_reference_plain'] = 'entity_reference_label';
    }
    if (!empty($process['options/type'][0]['map']) && empty($process['options/type'][0]['map']['i18n_taxonomy_term_reference_link'])) {
      $process['options/type'][0]['map']['i18n_taxonomy_term_reference_link'] = 'entity_reference_label';
    }

    $migration->setProcess($process);
  }
}

Hi guys,

I am kinda lost here. I am trying to figure it out how to migrate books from d7 to d8, but the drupal 8.0.2 does not migrate it yet and I am having no luck trying to get the necessary files to patch it from the issues listed in this post: https://www.drupal.org/node/2409435. Looks like someone got it already :)

Can somebody help me out please?

Thanks in advance!

Mauricio

I have come across some bugs when migrating users from D7 that do not have a language set - I have created an issue for this:
#2671312: No default value for User langcode when migrating D7 users with no language

I have added a few more issues & patches regarding d7 migration:

• #2673960: Unable to migrate D7 User cck fields
• #2674088: Unable to migrate D7 image fields
• #2674090: Unable to migrate D7 link fields

Here is issue regarding migrating custom fields on users:
#2674152: D7 User fields content not migrating first time

Hello,
Let me 1st apologize if this is not the correct place to post my migration problem.

I'm a fairly new user, trying to migrate my development 7.43 database to my clean install of 8.1. I used Drush migrate_upgrade --legacy (etc.).
My structure and block definitions got migrated, but there was no content copied over. I have checked the forums and this set of postings and have not seen anything about how to proceed to determine whether my content got copied and how to enable it or to find out what I've done wrong.
I am hosting both versions (7.43, 8.1) on my own PC, running Windows 10 Pro.
Any tips you can offer will be greatly appreciated.

I have found that some of my content does appear to have migrated, but my "front" page layout is not being applied. ONE navigation link (left sidebar) for HOME took me to SOME of that layout, but it is incomplete.

@DocDJ-forum please create a new issue. This is a [META] – issue tracking the overall work on migration and not for random support req.

Is this issue still being used to track migrations development? Adding the master issue for the migration system as parent.

Adding a couple of issues and patches:

• #2763637: D7 taxonomy term fields are not migrated with allowed vocabularies
• #2763629: cckPluginManager not explicitly declared in CckMigration

I apologize (again) for posting here. It was unclear to me that this forum was only for developers. I have posted my problems and help-requests elsewhere.

@DocDJ-forum: Just clarifying FYI, this forum is for everyone, but this issue was only meant to track the overall work on migration. You are very welcome to create a new 'Support Request' issue in this forum (issue queue) to get help. :) We do this so that we stay on topic and don't get distracted with unrelated conversation.

Good job us. It looks like we are done with all the Top and Medium priority issues for D7=>D8. We should probably review the IS and update. There are some entity reference and CCK migration issues we should add into the list.

I've just started work on upgrading my real-world D7 site, and the upgrade path looks quite good. At least I got some content in the D8 site. This D7 site contains ~15K nodes with about 1K being multilingual. For that in D7 I used the Internationalization, Entity Translation and Title. Most of these are now in core.

The problems I found after the migration:

• body summary didn’t get migrated
• entity translations didn't migrate - only the base node language was migrated. This included the title translations.
• title replacement field was not identified as a special case, and was migrated as a separate field. The resulting nodes now have the core title and the field_title fields.

I'd expect that these D7 modules that were being worked on by the core i18n team to be supported in the d7 to d8 upgrade paths, but I don't see them int the issue summary.

One final note, which may be related to the missing summary, I have to remove a null filter from each text format after the migration.

@jcnventura: we recently completed #2669964: Migrate Drupal 7 core node translations to Drupal 8 which supports core translations migration. We did not yet get around to even looking at supporting entity translation migrations (issue at #2073467: Migrate Drupal 7 Entity Translation settings to Drupal 8). All help is welcome there for sure :) We are working on more i18n module migrations like menu items, taxonomy, etc at the moment. See more at https://www.drupal.org/project/issues/search?issue_tags=i18n-migrate

@gábor, see you in Sevilla :)

Updated the issue for the Search path.

Please see below link , which is very useful for those who are going for upgrade.
how to estimate drupal 8 migration

Is it true we only have 2 more things for D7? Then we can close down this meta?
#2763637: D7 taxonomy term fields are not migrated with allowed vocabularies
#2587063: Variable to config: search_active_modules [d7]

@heddn: re #56 that first one is not even in the issue summary. It would be nice to keep this meta so long as there is more than one children.

We should update the IS. Looks like it is already tagged.

Add #2993367: Add the migration from D7 Picture to Responsive Image

Poll module is crossed out in the list but many people will need it to migrate from Drupal 6/7.

I have created an issue for it but the migration path is not complete: #2638406: Poll migrate support

The main remaining issue is figureing out how to create a reference to the poll from the poll node type during migration.

Updated the IS. Added all the Drupal 7 issues to the IS even though I'm not sure if they are all blockers to migration path.

@nbz, thanks for working on the migration for poll. It isn't included in the list here because this module list here is limited to those in core.

@nbz, I have added it though to #2906878: [Meta] Support for D7 -> D9 contrib migrate.

Removing fixed issues from the remaining tasks

Another one committed, #2993367: Add the migration from D7 Picture to Responsive Image

Only a few more to go!

Assigning to self to update IS.

Removing #3008202: Migrating more than 1 entity reference field instance fails because entity reference fields have a migration and if that is a problem it is a bug.

#3187320: Migrate Drupal 7 user settings was committed, removing from IS>

Only four left now.

#2565931: Handle long comment bundle names committed

Three more to go.

Make that four more to go

Fix typo in IS.

Back to three more to finish

#2814953: Migrate Drupal 7 node/user reference fields committed!

Thanks larowlan!

Removing fixed items from the IS #2814953: Migrate Drupal 7 node/user reference fields and #3005969: Add support for migrate d7 telephone field.

There is only one issue remaining, #3051252: Upgrade path for Multiupload Filefield Widget and Multiupload Imagefield Widget, so there the need for a Meta to track them all is no longer needed.

Thank you to everyone who working on the Drupal 7 to Drupal 8 or later upgrade path!

Cheers!

Tagging this issue for a 9.2.0 release highlight, because a bunch of additions and fixes for various contrib sources, such as #2814953: Migrate Drupal 7 node/user reference fields, landed into 9.2.x. Unfortunately, I don't have the full list in front of me, so this might need to be a vague statement rather than a list of specific issues, unless someone is able to put together that list.

@effulgentsia, here are the ones that are source related, or improve something at /upgrade.

• 21f096acfd Issue #3051252 by quietone, Matroskeen, Wim Leers: Upgrade path for Multiupload Filefield Widget and Multiupload Imagefield Widget
• 9c5da7cb4a Issue #3005969 by tstoeckler, quietone, longwave, ayushmishra206, Ante890, gbirch,zaporylie: Add support for migrate d7 telephone field
• 5e8e2edb22 Issue #3187320 by Wim Leers, shreyakaushik11, huzooka, quietone: Migrate Drupal 7 user settings
• 2a2f2de328 Issue #2814953 by quietone, huzooka, Nathaniel, Nixou, NickDickinsonWilde, benjifisher, cosolom, davidsickmiller, mradcliffe, catch: Migrate Drupal 7 node/user reference fields
• 42aa2b1b8c Issue #2925899 by quietone, Wim Leers, ravi.shankar, vsujeetkumar, benjifisher, alexpott, mikelutz, larowlan, mpp: MigrateUpgradeImportBatch does not use source_private_file_path & source_base_path correctly, making it impossible to have public & private files in separate locations