Problem/Motivation

I have a migration that is migrate English and Spanish content. It has multiple steps:

  • Migrating files and media
  • Migrating Taxonomies first in English and then in Spanish and linking the translations
  • Migrating Content first in English and then in Spanish and linking the translations

The majority of it works but I have discovered that when migrating the Spanish content, the fields which are entity relationships to taxonomy terms do not get migrated.

The snippet of the English content migration YAML which works is:

....
process:
....
  field_themes:
    plugin: migration_lookup
    migration: themes
    no_stub: true
    source: themecats
....
destination:
  plugin: entity:node
  default_bundle: resource
....

The snippet of the Spanish content migration YAML which doesn't work is:

....
process:
....
  langcode:
    plugin: default_value
    default_value: 'es'
....
  field_themes:
    plugin: migration_lookup
    migration: themes_es
    no_stub: true
    source: themecats
....
destination:
  plugin: entity:node
  default_bundle: resource
  translations: true
....

The English migration will create the nodes and link the terms in field_themes. The Spanish migration will create the nodes but the terms will not be linked.

I have done some debugging and discovered that if I get the value of field_themes before the entity is saved in the save() function in /core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php then the structure of the value is different for English and Spanish.

e.g. var_dump($entity->get('field_themes')->getValue()); produces

English

array(2) {
  [0]=>
  array(1) {
    ["target_id"]=>
    string(4) "1371"
  }
  [1]=>
  array(1) {
    ["target_id"]=>
    string(4) "1385"
  }
}

Spanish

array(1) {
  [0]=>
  array(2) {
    [0]=>
    string(4) "1371"
    [1]=>
    string(2) "es"
  }
}

Notice that for English the array is keyed on target_id but the Spanish are keyed on integers.

By adding (hacking!) code into the save() function and using $entity->get('field_themes')->getValue()to retrieve the values, process them into a new array, and then set them using $entity->get('field_themes')->setValue(). The value returned now contained an array item keyed on target_id and the taxonomy terms were created albeit in English not Spanish.

This test suggests to me that when creating the values for the Spanish fields the migration module code isn't creating the structure correctly. Unfortunately, I don't know enough to dig deeper to find where that might be happening.

Issue fork drupal-3244859

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

ded created an issue. See original summary.

ded’s picture

ded commented

Having done further debugging - I believe the structure creation is correct and that the langcode should be present. This is based on the code in the getIds function in core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php which is called from the doLookup function in core/modules/migrate/src/MigrateLookup.php

I have found that it is the filterEmptyItems function in core/lib/Drupal/Core/Field/FieldItemList.php which is stripping out the array items for the Spanish migration during the presave resulting in the terms not being created. It is called from the updateOriginalValues function in core/lib/Drupal/Core/Entity/ContentEntityBase.php which in turn is called from doPreSave function in core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php

quietone’s picture

quietone commented
Status: Active » Postponed (maintainer needs more info)

@ded. Thanks for the report and your work to isolate the problem. It is helpful. Can you provide some more information? What is the source of the data? Are you using the classic node migrations or the complete node migrations? It would help to see the themes and themes_es migration as well.

If field_themes is an entity reference field then the process pipeline should be using sub_process. See \Drupal\taxonomy\Plugin\migrate\field\TaxonomyTermReference.

ded’s picture

ded commented
Status: Postponed (maintainer needs more info) » Active

Thanks @quietone

The source is WordPress via its REST API using a custom source plugin which extends the migrate_plus URL source plugin

<?php

namespace Drupal\abcd_migration\Plugin\migrate\source;

use Drupal\migrate\Plugin\MigrationInterface;
use Drupal\migrate_plus\Plugin\migrate\source\Url;
use Drupal\migrate\MigrateException;

/**
 * Source plugin for retrieving data via URLs.
 *
 * @MigrateSource(
 *   id = "abcd_wp_api_url"
 * )
 */
class AbcdWordPressApiUrl extends Url {

  protected $base_url = "https://xxxxxxxxxx/wp-json/wp/v2/";
  protected $pagination = "_envelope&per_page=100&page=";

  /**
   * {@inheritdoc}
   */
  public function __construct(array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration) {
    // get the WordPress REST API query string
    $query = $configuration['urls'][0];
    // if root add ? if has query add &
    if(substr_count($query,'?') == 0) {
      $query .= '?';
    }
    else {
      $query .= '&';
    }
    // pass context=edit to get raw unless set to false
    if(!isset($configuration['raw'])) {
      $configuration['raw'] = true;
    }
    if($configuration['raw'] != false) {
      $this->pagination = 'context=edit&' . $this->pagination;
    }
    // construct full url for page 1
    $url = $this->base_url . $query . $this->pagination . "1";
    $opts = array('http' =>
      array(
        'header'  => 'Authorization: Basic ' . base64_encode($configuration['authentication']['username'] . ':' . $configuration['authentication']['password']),
      ),
      'ssl' => array(
        'verify_peer' => false,
        'verify_peer_name' => false,
      )
    );
    $context = stream_context_create($opts);
    $json = file_get_contents($url, false, $context);
    $obj = json_decode($json, true);
    if ($obj['status'] != 200) {
      throw new MigrateException('WordPress REST API returned status ' . $obj['status'] . ' for endpoint ' . $url);
    }
    // extract number of pages
    $pages = $obj['headers']['X-WP-TotalPages'];

    unset($configuration['urls']);
    // build URLS from number of pages
    for ($page = 1; $page <= $pages; $page++) {
      $configuration['urls'][] = $this->base_url . $query . $this->pagination . $page;
    }

    // Pass to parent URL class
    parent::__construct($configuration, $plugin_id, $plugin_definition, $migration);
  }

}

The themes migration is

uuid: 435a0117-9812-41d3-94b1-6455d36b97e4
id: themes
label: 'Import themes from WordPress taxonomy themecats'
migration_group: abcd

dependencies:
  enforced:
    module:
      - abcd_migration

source:
  plugin: abcd_wp_api_url
  data_fetcher_plugin: http
  data_parser_plugin: json
  raw: false
  urls:
    - themecats
  ids:
    id:
      type: integer
  item_selector: body/
  authentication:
    plugin: basic
    username: xxxx
    password: 'xxxxxxxxxxxxxxxx'
  constants:
    alias: '/themes'
  fields:
    -
      name: id
      label: 'id'
      selector: /id
    -
      name: name
      label: 'Name'
      selector: /name
    -
      name: description
      label: 'Description'
      selector: /description
    -
      name: slug
      label: 'Slug'
      selector: /slug
    -
      name: en_tid
      label: 'English TID'
      selector: /meta/tx-en_gb
    -
      name: es_tid
      label: 'Spanish TID'
      selector: /meta/tx-es_es
    -
      name: language
      label: 'Language'
      selector: /language
    -
      name: image
      label: 'Image'
      selector: /featured_image
process:
  _is_in_english:
    plugin: skip_on_value
    value: 'es'
    method: row
    source: language
  langcode:
    plugin: default_value
    default_value: 'en'
  name: name
  'description/value': description
  'description/format':
    plugin: default_value
    default_value: 'editor_html'
  uid:
    plugin: default_value
    default_value: 1
  field_hero_image:
    plugin: migration_lookup
    migration: files_media
    no_stub: true
    source: image
  field_image:
    plugin: migration_lookup
    migration: files_media
    no_stub: true
    source: image
  'path/pathauto':
    plugin: default_value
    default_value: 0 # Disable pathauto.
  'path/alias':
    plugin: concat
    source:
      - constants/alias
      - slug
    delimiter: /
destination:
  plugin: entity:taxonomy_term
  default_bundle: theme

migration_dependencies:
  required:
    - migration.files_media
  optional: {}

and themes_es is

uuid: a1197256-51fe-481e-88f6-0bfb1ba5eeff
id: themes_es
label: 'Import themes in Spanish from WordPress taxonomy themecats'
migration_group: abcd

dependencies:
  enforced:
    module:
      - abcd_migration

source:
  plugin: abcd_wp_api_url
  data_fetcher_plugin: http
  data_parser_plugin: json
  raw: false
  urls:
    - themecats
  ids:
    id:
      type: integer
  item_selector: body/
  authentication:
    plugin: basic
    username: xxxxx
    password: 'xxxxxxxxxxxxxxxx'
  constants:
    alias: '/themes'
  fields:
    -
      name: id
      label: 'id'
      selector: /id
    -
      name: name
      label: 'Name'
      selector: /name
    -
      name: description
      label: 'Description'
      selector: /description
    -
      name: slug
      label: 'Slug'
      selector: /slug
    -
      name: en_tid
      label: 'English TID'
      selector: /meta/tx-en_gb
    -
      name: es_tid
      label: 'Spanish TID'
      selector: /meta/tx-es_es
    -
      name: language
      label: 'Language'
      selector: /language
    -
      name: image
      label: 'Image'
      selector: /featured_image
process:
  _is_in_spanish:
    plugin: skip_on_value
    value: 'en'
    method: row
    source: language
  tid:
    -
      plugin: skip_on_empty
      source: en_tid
      method: process
      message: 'No translation available'
    -
      plugin: get
      source: en_tid
    -
      plugin: migration_lookup
      migration: themes
      no_stub: true
  langcode:
    plugin: default_value
    default_value: 'es'
  name: name
  'description/value': description
  'description/format':
    plugin: default_value
    default_value: 'editor_html'
  uid:
    plugin: default_value
    default_value: 1
  field_hero_image:
    plugin: migration_lookup
    migration: files_media
    no_stub: true
    source: image
  field_image:
    plugin: migration_lookup
    migration: files_media
    no_stub: true
    source: image
  'path/pathauto':
    plugin: default_value
    default_value: 0 # Disable pathauto.
  'path/alias':
    plugin: concat
    source:
      - constants/alias
      - slug
    delimiter: /
destination:
  plugin: entity:taxonomy_term
  default_bundle: theme
  translations: true

migration_dependencies:
  required:
    - migration.themes
    - migration.files_media
  optional: {}

I don't believe the classic/complete applies as the source is not Drupal. Let me know if I have that wrong.

I will look at \Drupal\taxonomy\Plugin\migrate\field\TaxonomyTermReference and see if I can debug further.

Thanks

ded’s picture

ded commented

Further debugging has not revealed anything new and I have been unable to find where the field's are populated which is where I think the issue is.

In order to unblock the project that this is holding up I have placed the following code in the save() function of Drupal\migrate\Plugin\migrate\destination\EntityContentBase class. 

    $fields = ['field_themes', 'field_regions', 'field_tags', 'field_type'];
    foreach($fields as $field) {
      if ($entity->hasField($field)) {
        $langcode = $entity->get($field)->getLangcode();
        if ($langcode == 'es') {
          $themes = $entity->get($field)->getValue();
          $ids = [];
          foreach ($themes as $key => $theme) {
            $ids[] = ['target_id' => $theme[0], 'langcode' => 'es'];
          }
          $entity->set($field, NULL);
          $entity->set($field, $ids);
        }
      }
    }

This deletes and recreates the field values which are created with the correct array keys and they are then migrated. It isn't a solution to the issue but provides a viable workaround for this project and also shows the field values array that the migration creates is incorrect.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.1.10 (June 4, 2021) and Drupal 9.2.10 (November 24, 2021) were the last bugfix releases of those minor version series. Drupal 9 bug reports should be targeted for the 9.3.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

naveed_jadoon made their first commit to this issue’s fork.

Version: 9.5.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

quietone commented
Status: Active » Postponed

The Migrate Drupal Module was approved for removal in #3371229: [Policy] Migrate Drupal and Migrate Drupal UI after Drupal 7 EOL.

This is Postponed. The status is set according to two policies. The Remove a core extension and move it to a contributed project and the Extensions approved for removal policies.

The deprecation work is in #3522602: [meta] Tasks to remove Migrate Drupal module and the removal work in #3522602: [meta] Tasks to remove Migrate Drupal module.

Migrate Drupal will not be moved to a contributed project. It will be removed from core after the Drupal 12.x branch is open.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.