Writing migrations for contributed and custom modules

This page complements Migrate API Overview by providing practical examples that can be used when writing migrations for contrib and custom modules.

Example: Migrating Configuration

A detailed example of how to migrate a Drupal 6 configuration variable to a Drupal 8 configuration object is provided in Migrating configuration entities.

Example: User Migration using Process Plugins and Process Pipelines

This example shows how the core User module is migrating users from Drupal 7 to Drupal 8. The migration is defined in core/modules/user/migrations/d7_user.yml. Let’s break this into pieces as we did with the first example:

id: d7_user
label: User accounts
migration_tags:
  - Drupal 7
class: Drupal\user\Plugin\migrate\User
source:
  plugin: d7_user
process:
  # If you are using this file to build a custom migration,
  # consider removing the uid field to allow
  # incremental migrations.
  uid: uid
  name: name
  pass: pass
  mail: mail
  created: created
  access: access
  login: login
  status: status
  timezone: timezone
  langcode:
    plugin: user_langcode
    source: language
    fallback_to_site_default: false
  preferred_langcode:
    plugin: user_langcode
    source: language
    fallback_to_site_default: true
  preferred_admin_langcode:
    plugin: user_langcode
    source: language
    fallback_to_site_default: true
  init: init
  roles:
    plugin: migration_lookup
    migration: d7_user_role
    source: roles
  user_picture:
    -
      plugin: default_value
      source: picture
      default_value: null
    -
      plugin: migration_lookup
      migration: d7_file
destination:
  plugin: entity:user
migration_dependencies:
  required:
    - d7_user_role
  optional:
    - d7_field_instance
    - d7_file
    - language
    - default_language
    - user_picture_field_instance
    - user_picture_entity_display
    - user_picture_entity_form_display

Again, we have ID of the migration, label and migration tags.

id: d7_user
label: User accounts
migration_tags:
  - Drupal 7
class: Drupal\user\Plugin\migrate\User

User migration has its own migrate plugin class User which extends FieldMigration. This is needed to migrate the custom fields that we might have added to the User entity on the Drupal 7 source site. We don’t get deeper to this topic in this tutorial but you are, of course, free to investigate the source code of the plugin class.

Defining the source plugin

source:
  plugin: d7_user

We’re defining here that we are using the User source plugin. The id ('d7_user') we use in the migration matches to the plugin annotation of the plugin class.

This source plugin will query the source data from the D7 database table ‘users’. For implementation details, see User::query().

Defining the migration process and transformations

process:
  uid: uid
  name: name
  pass: pass
  mail: mail
  created: created
  access: access
  login: login
  status: status
  timezone: timezone
  langcode:
    plugin: user_langcode
    source: language
    fallback_to_site_default: false
  preferred_langcode:
    plugin: user_langcode
    source: language
    fallback_to_site_default: true
  preferred_admin_langcode:
    plugin: user_langcode
    source: language
    fallback_to_site_default: true
  init: init
  roles:
    plugin: migration_lookup
    migration: d7_user_role
    source: roles
  user_picture:
    -
      plugin: default_value
      source: picture
      default_value: null
    -
      plugin: migration_lookup
      migration: d7_file

We have here a lot of 1:1 field mapping, for example D8 'uid' is mapped 1:1 from D7 'uid', 'name' is mapped 1:1 from 'name' and so on. 

The langcode mapping looks a bit different, let's have a look at that.

  langcode:
    plugin: user_langcode
    source: language
    fallback_to_site_default: false

• We are first saying that we’re talking about the transformation of the D8 destination property 'langcode'.
• The process plugin to be used for this is UserLangcode
  • This is a process plugin provided by the User module itself.
  • The process plugin class must be created in the MODULE/src/Plugin/migrate/process directory
  • The plugin ID used in the migration must be found in the plugin annotation of the class
• We’re saying that the source is coming from the source property 'language'
• The UserLangcode process plugin takes another argument 'fallback_to_site_default' which is set to 'false'. 

The real transformation logic is defined in UserLangcode::transform(). If we look at the source of this transform method, it contains the following logic:

• If the user's language is empty and the 'fallback_to_site_default' is 'false', it means the locale module was not installed and the user's langcode should be set to English.
• Then we check that the user's language actually exists in Drupal 8. If it does not exist in Drupal 8, we use the default language of the Drupal 8 site.
• If the language was found, we use that as user’s language.

Let’s next have a closer look at the user role mapping:

roles:
  plugin: migration_lookup
  migration: d7_user_role
  source: roles

• We are first saying that we’re talking about the D8 'roles' that the user should be given.
• In order to be able to give roles to the user, the roles must have been migrated already earlier using the 'd7_user_role' migration.
• We use the migration_lookup process plugin which takes the D7 roles as an input and returns the corresponding D8 role ids. 

The migration_lookup process plugin is one of the most fundamental process plugins.

Let’s then have a closer look at the processing for user picture.

user_picture:
  -
    plugin: default_value
    source: picture
    default_value: null
  -
    plugin: migration_lookup
    migration: d7_file

Here the Drupal 7 source value 'picture' must pass through two process plugins to end up with the correct Drupal 8 value. The anatomy of the process pipeline is explained on the migrate process overview documentation page. 

What happens here is as follows:

• We’re first passing the Drupal 7 source property 'picture' through the default_value process plugin. This property is a 1:1 relationship with the Drupal 7 file_managed.fid value.
• This plugin passes the given default value (null in this case) to the next phase of the process pipeline if the original source property had no value (NULL, zero or empty string). If the source property has a value, then that value is preserved and passed to the next phase of the process pipeline.
• The second phase of the process pipeline is the migration_lookup plugin which we already covered above in this article. It uses the value from the previous pipeline step as its source and checks the ID of the file which was previously migrated using the d7_file migration.

Defining the destination plugin

destination:
  plugin: entity:user

Here we are saying that the migration is generating Drupal 8 user entities.

Migration dependencies

 migration_dependencies:
  required:
    - d7_user_role
  optional:
    - d7_field_instance
    - d7_file
    - language
    - default_language
    - user_picture_field_instance
    - user_picture_entity_display
    - user_picture_entity_form_display

We are declaring here that the migration d7_user_role is a prerequisite for running this migration. The optional migrations do not necessarily have to be executed but Migrate Drupal will execute them before this one if you are running all migrations at one go.

Advanced example: Node migration uses deriver class to dynamically build migrations for each content type

The source Drupal 6 or Drupal 7 site will have the content types the site builder had created and all content types have their own fields. 

• There might be for example a content type Car with fields make and model.
• And content type Driver with fields name and age.

What we need to achieve is obviously to migrate all Drupal 7 Car nodes to Drupal 8 as Car nodes and all Drupal 7 Driver nodes to Drupal 8 as Driver nodes. If we inspect the migrations at admin/config/development/configuration/single/export after the migrations have been generated (select 'Migration' as 'Configuration type'), we can see that there is a separate migration generated for Car nodes and Driver nodes and these two migrations have mappings for the correct fields. Let’s have a look at the core/modules/node/migrations/d7_node.yml to investigate how did this magic happen.

id: d7_node
label: Nodes
migration_tags:
  - Drupal 7
deriver: Drupal\node\Plugin\migrate\D7NodeDeriver
source:
  plugin: d7_node
process:
  # If you are using this file to build a custom migration consider removing
  # the nid and vid fields to allow incremental migrations.
  # In D7, nodes always have a tnid, but it's zero for untranslated nodes.
  # We normalize it to equal the nid in that case.
  # @see \Drupal\node\Plugin\migrate\source\d7\Node::prepareRow().
  nid: tnid
  vid: vid
  langcode:
    plugin: default_value
    source: language
    default_value: "und"
  title: title
  uid: node_uid
  status: status
  created: created
  changed: changed
  promote: promote
  sticky: sticky
  revision_uid: revision_uid
  revision_log: log
  revision_timestamp: timestamp
destination:
  plugin: entity:node
migration_dependencies:
  required:
    - d7_user
    - d7_node_type
  optional:
    - d7_field_instance
    - d7_comment_field_instance

The secret lies in the deriver class which is D7NodeDeriver and to be more specific, in D7NodeDeriver::getDerivativeDefinitions(). Since this is a very advanced topic, reading the source is the best way to understand the flow. On high level:

• We first read all node types that the site has. This is needed to generate the different migrations for each content type (i.e. Cars and Drivers in the example above)
• For each node type, we generate the field mappings based on the fields that each node type has. 
• And finally we return the derivatives.

Field migrations: using email field migration as an example

Writing an upgrade path for a module providing a field type is not too complex. We'll use the Drupal 7 Email as an example. The Email module was moved to Drupal 8 core but the example is valid also for modules which remain in contrib space for Drupal 8.

Write a plugin class that extends FieldPluginBase

• Class \Drupal\Core\Field\Plugin\migrate\field\Email extends \Drupal\migrate_drupal\Plugin\migrate\field\FieldPluginBase 
• FieldPluginBase implements MigrateFieldInterface
• The namespace to use for your plugin class is Drupal\MODULE\Plugin\migrate\field and thus the file should be located in MODULE/src/Plugin/migrate/field directory (where MODULE is obviously the name of your module). 

Let's start by inspecting the annotation of this plugin class

@MigrateField(
  id = "email",
  core = {6,7},
  type_map = {
    "email" = "email"
  },
  source_module = "email",
  destination_module = "core"
)

• ID and core versions are straight forward.
• The field type names are 'email' in both Drupal 8 and Drupal 6/7 (left hand side of the 'type_map' is the Drupal 8 field type and right hand side is the Drupal 6/7 field type.
• 'source_module' and 'destination_module' define the module names that are shown in the Migrate Drupal UI. Note: These became required as part of #2859304: Show field type migrations correctly in Migrate Drupal UI, see change record.

Methods to implement

The methods the plugin class need to implement depend obviously on the field that is being migrated. Email field is quite typical and it implements getFieldWidgetMap(), getFieldFormatterMap() and defineValueProcessPipeline(). 

Widget and formatter maps are straight forward array maps:

  /**
   * {@inheritdoc}
   */
  public function getFieldWidgetMap() {
    return [
      'email_textfield' => 'email_default',
    ];
  }

• The email field migration maps Drupal 6/7 'email_textfield' to Drupal 8 'email_default'.

  /**
   * {@inheritdoc}
   */
  public function getFieldFormatterMap() {
    return [
      'email_formatter_default' => 'basic_string',
      'email_formatter_contact' => 'basic_string',
      'email_formatter_plain' => 'basic_string',
      'email_formatter_spamspan' => 'basic_string',
    ];
  }

• The email field migration maps all different Drupal 6/7 field formatter settings to Drupal 8 'basic_string'.  

  /**
   * {@inheritdoc}
   */
  public function defineValueProcessPipeline(MigrationInterface $migration, $field_name, $data) {
    $process = [
      'plugin' => 'iterator',
      'source' => $field_name,
      'process' => [
        'value' => 'email',
      ],
    ];
    $migration->setProcessOfProperty($field_name, $process);
  }

• The actual processing of field values is done by implementing processFieldValue()
• The email field migration uses the iterator process plugin which allows migration of multi-value fields.
  • Note: iterator process plugin is deprecated. Use SubProcess process plugin instead.
• For each value we apply mapping so that the Drupal 8 destination property 'value' is mapped from Drupal 6/7 source property 'email'.

Summary and further reading

The examples above demonstrate different aspects of how Drupal core modules are migrating to Drupal 8.

• Migrate source plugins page has more information on how to write your own source plugin if you need to define one.
• Migrate process plugins page has more information on the general purpose process plugins provided by Drupal 8 core and how to define your own process plugins.
• Migrate destination plugins page has more information on the destination plugins.