Multilingual Migrations for Non-Drupal Sources [#2733431]

Motivation

So there's a flurry of activity concerning multilingual Drupal content from D6 & D7 being properly migrated (#2208401: [META] Remaining multilingual migration paths), but I'm not seeing much if anything concerning non-Drupal imports for multilingual content. Point of fact, I'm having trouble coming up with good examples of how multilingual migrations are intended to work, let alone actual working examples using Drupal or other sources.

Proposed resolution

I'd like to use this thread to compile migration solutions or examples that successfully implement multilingual content. Ultimately, that would become a documentation page, or depending on how things go a feature request or module.

Groundwork

So since I had an immediate need and was looking for solutions I ended up finding #2313265: How import Multilingual Content with Migrate Module. While it originated Pre-D8, @digitaldonkey had a good solution for a POST_ROW_SAVE event in D8. I liked it, but it wasn't flexible enough for me to I took it and made it a little more broadly based. I updated the 'updateTranslations' function of the original:

/**
   * MigrateEvents::POST_ROW_SAVE event handler.
   *
   * @param MigratePostRowSaveEvent $event
   *   Instance of Drupal\migrate\Event\MigratePostRowSaveEvent.
   */
  public function updateTranslations(MigratePostRowSaveEvent $event) {

    $row =  $event->getRow();

    if ($row->hasSourceProperty('constants/available_languages')) {

      // These are defined in migration.yml.
      $available_languages = $row->getSource()['constants']['available_languages'];
      $default_language = $row->getDestination()['langcode'];

      // Unset default language from available languages.
      unset($available_languages[$default_language]);

      $migrated_entity = $event->destinationIdValues[0];
      $dest_config = $event->getMigration()->getDestinationConfiguration();
      $dest_plugin = explode(':', $dest_config['plugin']);

      if ($dest_plugin[0] == 'entity') {
        $entity = \Drupal::entityTypeManager()
          ->getStorage($dest_plugin[1])
          ->load($migrated_entity);

        foreach ($available_languages as $key => $lang_map) {

          $translated_entity = $entity->addTranslation($key);
          foreach ($lang_map as $field => $source) {
            $translated_entity->$field = $row->getSourceProperty($source);
          }
          $translated_entity->save();

        }

        $map = $event->getMigration()->getIdMap();
        $map->saveIdMapping($event->getRow(), array($migrated_entity));
      }
    }
  }

What this allows you to do is declare translations in your YAML file by adding available translations as constants. So your source would look something like

source:
  ...
  constants:
    available_languages:
      en:
      es:
        title: title_spanish

Where title is the name of the field you're entering and title_spanish would be the field coming in from the source. This is perfect for a lightweight translation of content from the same source. Note that it only works on entities and that the main langcode MUST be set in the process.

For bulkier translations that need processing or those coming from a different source, I put together a custom destination. This takes a separate migration and appends identically ID'd translations to it.


namespace Drupal\custom_migrate\Plugin\migrate\destination;

use Drupal\Component\Plugin\DependentPluginInterface;
use Drupal\Core\Entity\DependencyTrait;
use Drupal\migrate\Plugin\MigratePluginManager;
use Drupal\migrate\Plugin\MigrationPluginManagerInterface;
use Drupal\Core\Language\LanguageManagerInterface;
use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Drupal\migrate\Plugin\MigrationInterface;
use Drupal\migrate\Row;
use Symfony\Component\DependencyInjection\ContainerInterface;
use Drupal\migrate\Plugin\migrate\destination\DestinationBase;

/**
 * Provides Configuration Management destination plugin.
 *
 * Persist data to the config system.
 *
 * When a property is NULL, the default is used unless the configuration option
 * 'store null' is set to TRUE.
 *
 * @MigrateDestination(
 *   id = "entity_translation"
 * )
 */
class EntityTranslation extends DestinationBase implements ContainerFactoryPluginInterface, DependentPluginInterface {

  use DependencyTrait;

  /**
   * The config object.
   *
   * @var \Drupal\migrate\Plugin\MigrationInterface
   */
  protected $old_migration;

  /**
   * The language manager.
   *
   * @var \Drupal\Core\Language\LanguageManagerInterface
   */
  protected $language_manager;

  /**
   * The process plugin manager.
   *
   * @var \Drupal\migrate\Plugin\MigratePluginManager
   */
  protected $processPluginManager;

  /**
   * The migration plugin manager.
   *
   * @var \Drupal\migrate\Plugin\MigrationPluginManagerInterface
   */
  protected $migrationPluginManager;

  /**
   * Constructs a Config destination object.
   *
   * @param array $configuration
   *   A configuration array containing information about the plugin instance.
   * @param string $plugin_id
   *   The plugin ID for the plugin instance.
   * @param mixed $plugin_definition
   *   The plugin implementation definition.
   * @param \Drupal\migrate\Plugin\MigrationInterface $migration
   *   The migration entity.
   * @param \Drupal\Core\Language\ConfigurableLanguageManagerInterface $language_manager
   *   The language manager.
   */
  public function __construct(array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration, MigrationPluginManagerInterface $migration_plugin_manager, MigratePluginManager $process_plugin_manager, LanguageManagerInterface $language_manager) {
    parent::__construct($configuration, $plugin_id, $plugin_definition, $migration);
    $this->migrationPluginManager = $migration_plugin_manager;
    $this->migration = $migration;
    $this->processPluginManager = $process_plugin_manager;
    $this->language_manager = $language_manager;
  }

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration = NULL) {
    return new static(
      $configuration,
      $plugin_id,
      $plugin_definition,
      $migration,
      $container->get('plugin.manager.migration'),
      $container->get('plugin.manager.migrate.process'),
      $container->get('language_manager')
    );
  }

  /**
   * {@inheritdoc}
   */
  public function import(Row $row, array $old_destination_id_values = array()) {
    
    $migration_id = $this->configuration['migration'];
    $migration = $this->migrationPluginManager->createInstance($migration_id);

    // TODO: I think this only works right now if the keys and key labels match, so I'll need to expand later
    $source_id_values = $row->getSourceIdValues();

    $destination_ids = $migration->getIdMap()->lookupDestinationID($source_id_values);

    $dest_config = $migration->getDestinationConfiguration();
    $dest_plugin = explode(':', $dest_config['plugin']);

    $entity = \Drupal::entityTypeManager()
      ->getStorage($dest_plugin[1])
      ->load($destination_ids[0]);

    $lang = $row->getDestinationProperty('langcode');

    // TODO: Validate langcode against list of site languages

    $translated_entity = $entity->addTranslation($lang, $row->getDestination());
    $translated_entity->save();

    return $destination_ids;

  }

  /**
   * {@inheritdoc}
   */
  public function fields(MigrationInterface $migration = NULL) {
    // @todo Dynamically fetch fields using Config Schema API.
  }

  /**
   * {@inheritdoc}
   */
  public function getIds() {
    $ids['config_name']['type'] = 'string';
    return $ids;
  }

  /**
   * {@inheritdoc}
   */
  public function calculateDependencies() {
    $provider = explode('.', $this->config->getName(), 2)[0];
    $this->addDependency('module', $provider);
    return $this->dependencies;
  }

}

It's still very rough, but once it's in place you can use it to translate entities by adding this destination:

destination:
  plugin: entity_translation
  migration: original_content

Where original_content is the targeted migration. Note that it's still very rough, but is certainly a step in the right direction. Once again, it will only work for entities and the langcode needs to be set in the process.

Comment	File	Size	Author
#21	core-multilingual-migrations-for-non-drupal-sources-2733431-11.patch	977 bytes	mrtndlmt
#21	8.7.x: PHP 7.2 & MySQL 5.5 24,407 pass
#10	core-multilingual-migrations-for-non-drupal-sources-2733431-10.diff	1.19 KB	kriboogh
#10

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

25 May 2016 at 13:59

dhansen created an issue. See original summary.

Comment #2

mikeryan

he/him

English

Murphysboro, IL, USA

CreditAttribution: mikeryan at Virtuoso Performance commented 27 May 2016 at 00:53

@dhansen - Please see (and better yet, review!) the work in progress at #2225775: Migrate Drupal 6 core node translation to Drupal 8. Among other things, it includes destination plugin support for calling addTranslation() where applicable.

Comment #3

vasi CreditAttribution: vasi at Evolving Web commented 10 June 2016 at 15:25

There's an example in the migrate_external_translated_test module in #2225775: Migrate Drupal 6 core node translation to Drupal 8 .

But I am concerned that our translated migrations may all have the pattern of:

One migration for the default-translation of each node.
One migration for other translations.

This is not something that will necessarily map well to other types of external content. I'm a bit concerned that we could back ourselves into a corner if we ignore this too long in our rush to get D6 -> D8 finished. Let's try to set up examples and tests early for at least a couple of other translation patterns, chosen from:

Field translation, as used in D8, and D7 with the right module. See #2669964: Migrate Drupal 7 core node translations to Drupal 8
One row per translation, with columns (ID, langcode). No concept of source/default translations.
One row per translation, with columns (ID, langcode, source_langcode).
One row per entity, with translations embedded somehow (like field translation)
Others?

@mikeryan, which pattern is most common in your experience? What does eg: Wordpress use?

Comment #4

mikeryan

he/him

English

Murphysboro, IL, USA

CreditAttribution: mikeryan at Virtuoso Performance commented 10 June 2016 at 16:30

I have not seen a translated WordPress site - doing a little research, it seems that WordPress has no builtin multilingual capability, there are several plugins that provide it (such as WPML - which looks like it would map well to our tnid module).

I think the main priority will be supporting field translation from D7 - I'm not sure, lacking concrete examples, it's worth trying to speculate how else translations might be represented.

Comment #5

kriboogh CreditAttribution: kriboogh at Calibrate commented 8 July 2016 at 14:14

Hi, not sure if it's appropriate to ask this here, but been struggling with this for days now and this is the first thread I found related to our problem. We are trying to do something like this. We have a JSON file which contains a an array of objects which describe basically a page. We map these onto a D8 page content types. It has these properties: id, translation id, title, body content, body format (which we map to html, plain text) and a language.

Ex:

{
  "pages": [
    {
      "id": 1,
      "tid": 1,
      "langcode": "en",
      "title": "Disclaimer",
      "field_content_value": "Lorem ipsum...EN",
      "field_content_format": 2
    },
    {
      "id": 3,
      "tid": 1,
      "langcode": "nl",
      "title": "NL Disclaimer",
      "field_content_value": "NL Lorem ipsum...NL",
      "field_content_format": 2
    },
  ]
}

This is how I configured my migration:

source
    ...
    ids:
       id:
          type: integer
destination:
  plugin: entity:node
process:
  type:
    plugin: default_value
    default_value: page
  langcode: langcode
  title: title
  'field_content/value': field_content_value
  'field_content/format':
    plugin: static_map
    source: field_content_format
    map:
      1: plain_text
      2: basic_html
      3: full_html

The source plugin is a custom extended "url" with a "http" and "json" reader and fetcher (based on the migrate wine example except we read the JSON directly from a local file in stead of a REST setup) .

With the JSON above this generates 2 node entities in D8, but how can I tell it, that the Dutch entity is a translation of the English ?

I also tried by giving the two sources the same id (so english and dutch have id=1). This generates 1 node (as it should), but it messes up the language settings of the entity. The first row (the source language), is created with a defaultLanguage=en. The moment the second row is processed however, during the processing of the langcode field, the defaultLanguage property of the entity is overwritten to 'nl' (the language of the current row). Thus not saving a translation, but overwriting the existing english settings.

If you trace this, you can see that in the content_translation.module file there is a presave hook which should normaly handle this case correctly. Doing this through the normal add content and translation UI, you can see this works. But during a migration, the moment the 2nd row source values are saved, the $source_langcode gets a 'und' and not the expected 'en' (which it does when using the translate UI).

Comment #6

mikeryan

he/him

English

Murphysboro, IL, USA

CreditAttribution: mikeryan at Virtuoso Performance commented 8 July 2016 at 14:20

Status:	Active	» Postponed
Related issues:		+#2225775: Migrate Drupal 6 core node translation to Drupal 8

Migrating multi-lingual entities (regardless of source) will require the entity destination changes that are part of #2225775: Migrate Drupal 6 core node translation to Drupal 8.

Comment #7

mikeryan

he/him

English

Murphysboro, IL, USA

CreditAttribution: mikeryan at Virtuoso Performance commented 14 July 2016 at 14:35

Status:

Postponed

» Active

Comment #8

sylus CreditAttribution: sylus commented 26 July 2016 at 19:16

I eventually got this to work but required a small patch/hack to the migration process plugin. ^_^

#2767643: Scalar to array migration returns NULL

Any idea why this was needed?

Comment #9

26 July 2016 at 19:16

Version:

8.2.x-dev

» 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Comment #10

kriboogh CreditAttribution: kriboogh at Calibrate commented 21 December 2016 at 08:24

File	Size
core-multilingual-migrations-for-non-drupal-sources-2733431-10.diff	1.19 KB

A small update on this, in case anybody is trying to do this. This is related to issue #2809315 and #2767643

Short recap:
Source is a custom database table having at least these colums: source_id, source_langcode, ...
Destination is a multilingual entity (node, term,...)

Migrations config:

langcode: en
status: true
dependencies:
  enforced:
    module:
      - my_migration_module
id: my_content_migration
migration_tags: null
migration_group: default
label: 'My migration'
source:
  plugin: my_migration_source
process:
  nid:
    plugin: migration
    source: source_id
    migration: my_content_migration
    no_stub: true
  type:
    plugin: default_value
    default_value: my_destination_content_type
  title: source_title
  langcode: source_langcode
  ...
destination:
  plugin: 'entity:node'
  translations: true
migration_dependencies:
  ...

The custom source implements these:

/**
 * Migrate source for my content type import.
 *
 * @MigrateSource(
 *   id = "my_content_migration"
 * )
 */
class MyContentMigration extends SqlBase {

  /**
   * {@inheritdoc}
   */
  public function query() {
    // Do whatever query you need to do, but make sure, you have at least all the necessary column (source_id and source_langcode)
    // Ideally your item in the default language are sorted before your other translation languages
    $query = $this->select('my_content_table', 'n')->fields('n');
    return $query;
  }

  /**
   * {@inheritdoc}
   */
  public function fields() {
    // You could also implement this in the config...
    $fields = array(
      'source_id' => $this->t('ID'),
      'source_langcode' => $this->t('Langcode'),
      ...
    );

    return $fields;
  }

  /**
   * {@inheritdoc}
   */
  public function getIds() {
    return [
      'source_id' => [
        'type'  => 'integer',
      ],
      'source_langcode' => [
        'type' => 'string',
      ],
    ];
  }
}

You also need to setup your migration database parameters in the settings.php file, so add an extra database entry:

$databases['migrate']['default'] = array (
  'database' => 'my_source_database',
  'username' => 'username',
  'password' => 'password',
  'prefix' => '',
  'host' => 'host',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
);

Now with the fix made in #2767643, we hit an other issue. In order for translations to be linked to the source node, the process configuration for nid is setup to lookup the already imported source_id. This is done in the getEntity method of the EntityContentBase class. The parent class, Entity tries to fetch the entity id. However the id's generated when using the destination 'translatable: true' options, are arrays. So the storage->load method fails and so it is assumed a new node has to be created. This then fails later on when trying to save the new node, because of a database primary key failure (the same id already exists).

I patched the EntityContentBase class, so that when translations is enabled, it retrieves only the entity id as a singular value, and no longer a array. The patch is against 8.2.4

Comment #11

meanderix CreditAttribution: meanderix commented 29 December 2016 at 16:24

#2073467: Migrate Drupal 7 Entity Translation settings to Drupal 8 has been a showstopper in my migration project. The current migration process works for the default "content translation" mechanism but not for Entity Translation (i.e. field level translation). When I perform a standard D7 -> D8 migration, no translations are added and the (untranslated) fields are populated with the wrong language.

I realized that it would be rather an involved process to add support for this migration path (both field migrations and node migrations would have to be added and I'm fairly new to the Migrate API).

In the end I decided to go with a CSV export using Entity Translation Export/Import. The patch in #2839848: CSV file invalid when exporting fields from multiple bundles was required for this to work. Additionally, I made some further changes:

diff --git a/entity_translation_export_import.admin.inc b/entity_translation_export_import.admin.inc
index 34a0d09..11f1e93 100644
--- a/entity_translation_export_import.admin.inc
+++ b/entity_translation_export_import.admin.inc
@@ -308,16 +312,19 @@ function _entity_translation_export_import_export_entity($entity_type, $bundle,
     foreach($items as $entity_id => $item){
       foreach($languages as $language){
         $entity = current(entity_load($entity_type, array($entity_id)));
-        $export[$entity_id]['Entity Type (Not Editable)'] = $entity_type;
-        $export[$entity_id]['Bundle (Not Editable)'] = $bundle;
-        $export[$entity_id]['Entity ID (Not Editable)'] = $entity_id;
+        $export[$entity_id]['entity_type'] = $entity_type;
+        $export[$entity_id]['bundle'] = $bundle;
+        $export[$entity_id]['nid'] = $entity_id;
 
         $handler = entity_translation_get_handler($entity_type, $entity, TRUE);
         $translations = $handler->getTranslations();
+        $export[$entity_id]['original_lang'] = $translations->original;
+        $export[$entity_id]['translations'] = implode(",", array_keys($translations->data));
         foreach($fields as $field){
           $value = '';
           if(isset($translations->data[$language->language])){
             $value = $entity->{$field}[$language->language][0]['value'];

I created two migrations "node_sv" and "node_en" (one for each language supported by the site).

Example of the "node_sv" migration:

status: true
id: node_sv
migration_group: translations
label: 'Import D7 nodes (sv)'
source:
  plugin: csv
  path: /path/to/exported_nodes.csv
  header_row_count: 1
  keys:
    - nid
process:
  nid: nid
  langcode:
    plugin: default_value
    source: language
    default_value: sv
  body:                                     body_sv
  name_field:                               name_field_sv
  title:                                    title_field_sv
  field_title:                              field_title_sv  
  description_field:                        description_field_sv
  field_area_of_research:                   field_area_of_research_sv
  ... # (list of all fields)
destination:
  plugin: entity_translation

Here's the code used in my EntityTranslation destination plugin:

  /**
   * {@inheritdoc}
   */
  public function import(Row $row, array $old_destination_id_values = array()) {

    drush_print_r("===================================================================================================");

    $source_id_values = $row->getSourceIdValues();
    $destination_ids = array($source_id_values['nid']);

    $bundle = 'node';
    $nid = $destination_ids[0];

    $entity = \Drupal::entityTypeManager()
      ->getStorage($bundle)
      ->load($nid);

    $srclang = $row->getSourceProperty('langcode');
    $lang = $row->getDestinationProperty('langcode');

    $uid = $entity->getOwnerId();

    $source = $row->getSource();
    drush_print_table(array(array("nid", "lang", "org", "bundle", "title"), array($nid, $lang, $source['original_lang'], $source['bundle'], $entity->getTitle())), TRUE, array(5, 5, 5, 30, 50));

    $translations = explode(",", $source['translations']);

    if (in_array($lang, $translations)) {
      if ($lang == $source['original_lang']) {
        foreach ($row->getDestination() as $field => $value) {
          $entity->$field = $value;
        }
        $entity->save();
      } else {
        $translated_entity = $entity->addTranslation($lang, $row->getDestination());
        $translated_entity->setOwnerId($uid);
        $translated_entity->save();
      }
    }
    return $destination_ids;
  }

All in all, these measures seem to provide me with an upgrade path from the field translation used in D7.

Comment #12

sylus CreditAttribution: sylus commented 5 January 2017 at 20:24

For #10 I might be wrong but as my own implementation is quite similar.

I was having a problem initially but just needed to take into account this issue #2746293: Migrate content_translation_source when migrating node translations and the introduction of content_translation_source. Maybe this will help?

Comment #13

kriboogh CreditAttribution: kriboogh at Calibrate commented 7 January 2017 at 18:04

@sylus can you post a (simplified) example of your migration (specially the yml and how you used the content_translation_source) ?

Comment #14

7 January 2017 at 18:04

Version:

8.3.x-dev

» 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Comment #15

jwilson3

English

Ecuador

CreditAttribution: jwilson3 at Bluespark commented 27 February 2017 at 02:56

I'm running into some issues migrating multilingual content containing shared images with different translations of the "alt" text from CSVs.

I get the standard error:

InvalidArgumentException: Invalid translation language (und) specified.
in Drupal\Core\Entity\ContentEntityBase->getTranslation() (line 745 of
web/core/lib/Drupal/Core/Entity/ContentEntityBase.php).
Invalid translation language (und) specified.
(web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php:770)

I'd appreciate feedback or any insight on what I'm doing wrong in my yaml configs here:

http://drupal.stackexchange.com/questions/229750/how-to-migrate-multilin...

Comment #16

vasi CreditAttribution: vasi at Evolving Web commented 28 February 2017 at 07:20

jwilson3's problem seems to be happening because of the following odd behaviour:

$node = Node::load($nid);
printf("Original: %s\n", $node->language()->getId()); // -> en

$translation = $node->addTranslation('fr');
$translation_manager = \Drupal::service('content_translation.manager');
$metadata = $translation_manager->getTranslationMetadata($translation);
printf("Source: %s\n", $metadata->getSource()); // -> und

It seems that if we want a sensible translation, we need to explicitly set the translation source. Is that intended? Could it be automatically set when we call addTranslation()?

Comment #17

28 February 2017 at 07:20

Version:

8.4.x-dev

» 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Comment #18

kriboogh CreditAttribution: kriboogh at Calibrate commented 25 October 2017 at 13:37

I still think that the patch I had for #10 handles the multi id's key situation. What @vasi describes in #16 is an other issue, that can be fixed by issue #2544696-#11.

Comment #19

25 October 2017 at 13:37

Version:

8.5.x-dev

» 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Comment #20

25 October 2017 at 13:37

Version:

8.6.x-dev

» 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Comment #21

mrtndlmt CreditAttribution: mrtndlmt at Calibrate commented 2 August 2018 at 09:51

File	Size
core-multilingual-migrations-for-non-drupal-sources-2733431-11.patch	977 bytes
8.7.x: PHP 7.2 & MySQL 5.5 24,407 pass

Update patch for drupal 8.5.6!

Comment #22

borisson_

Dutch

Mechelen, 🇧🇪

CreditAttribution: borisson_ as a volunteer and at Calibrate commented 26 September 2018 at 08:51

Status:

Active

» Needs review

Setting to needs review so that the testbot can have a look at the latest patch (in #21)

Comment #23

borisson_

Dutch

Mechelen, 🇧🇪

CreditAttribution: borisson_ as a volunteer and at Calibrate commented 26 September 2018 at 19:22

Issue tags:

-migrate

+Needs tests

The testbot agrees with the latest patch, now all we need is tests.

Comment #24

phenaproxima

he/him

English

Massachusetts

CreditAttribution: phenaproxima at Acquia commented 27 September 2018 at 21:57

Status:

Needs review

» Needs work

Marking "Needs work" for tests.

Comment #25

27 September 2018 at 21:57

Version:

8.7.x-dev

» 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Comment #26

27 September 2018 at 21:57

Version:

8.8.x-dev

» 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Comment #27

27 September 2018 at 21:57

Version:

8.9.x-dev

» 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Comment #28

quietone CreditAttribution: quietone as a volunteer commented 14 May 2020 at 06:02

Had a look at this to see what test is needed here. The patch applies to 9.1.x so maybe maybe it is needed but I just have questions.

Is anyone using this patch?

+++ b/core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php
@@ -155,6 +155,22 @@ public function import(Row $row, array $old_destination_id_values = []) {
+    $entity_id = parent::getEntityId($row);
+    if ($this->isTranslationDestination()) {
+      // If we are translating, we might get the id and language as a id value.
+      // So make sure we only get the ID.
+      if (is_array($entity_id)) {

The doc for the parent:getEntityId() state that it returns a string not an array. Is there an entity where the Id is an array?

Comment #29

quietone CreditAttribution: quietone as a volunteer commented 14 May 2020 at 11:07

Status:

Needs work

» Needs review

Setting NR to get an answer for #28

Comment #30

mikelutz

He/Him

English

Michigan, USA

CreditAttribution: mikelutz as a volunteer commented 21 May 2020 at 04:06

Status:

Needs review

» Needs work

I won't close this, but this isn't something to commit to core. Core dosn't include anything for migrating from non drupal sources, so we need a good documented reason to add a method like this. I'm not sure what the intent is here exactly, I assume to provide a means to get the original entity id for a specific custom purpose, but it needs more exploration/work.

Comment #31

quietone CreditAttribution: quietone as a volunteer commented 4 July 2020 at 04:49

Status:	Needs work	» Postponed (maintainer needs more info)
Issue tags:		+Needs issue summary update

Triaging the support request and saved this till last.

The proposed resolution in the IS is to add documentation and yet in December 2016, kriboogh added a patch (#10). The patch was last updated in August 2018 (#21) and was tagged as needs tests in September 2018. So, is this issue for documentation or a patch or both? Please update the IS with some clarification.

Can anyone confirm that that you have used the patch in this issue and it worked? If so, please provide details about the source data. As mikelutz said in #30 if the patch is to be committed "we need a good documented reason to add a method like this".

If anyone can add a link to blog about migrating from non drupal sources please do so. The search I did just now didn't find anything about nodes or anything that was using this patch.

Since this is a support request and hasn't had any progress in almost 2 years changing status to PPMNMI. As it stands now there doesn't seem to be a interest in, or perhaps need, for this anymore.

Comment #32

4 July 2020 at 04:49

Version:

9.1.x-dev

» 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Comment #33

4 July 2020 at 04:49

Version:

9.2.x-dev

» 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Comment #34

4 July 2020 at 04:49

Version:

9.3.x-dev

» 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Comment #35

kriboogh CreditAttribution: kriboogh at Calibrate commented 29 December 2021 at 11:57

Been going through some of our patches we still use and came across this old one. We still use it though, to migrate translated data from csv's to nodes.

Our source data has several translations (langcode) of the same data (id), each in their own rows.

So we have:
header: id, langcode
row 1: 1, nl, ....
row 2: 1, fr, ....
...
row x: x, nl, ....
row y: x, fr, ....

The way we use the migration is we define our unique 'ids' as combination of id, langcode

ids:
   id:
      'type': 'string'
   langcode:
       'type': 'string'

So to answer #28, when you specify your source ids like this, you get an array not a string.
In our case the returned array has the id at index 0, because we configured it as the first id key in 'ids'.

Maybe the solution proposed in the patch is indeed to specific and doesn't need to be merged. We can always use a local patch.

Comment #36

29 December 2021 at 11:57

Version:

9.4.x-dev

» 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Comment #37

29 December 2021 at 11:57

Version:

9.5.x-dev

» 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Comment #38

29 December 2021 at 11:57

Version:

10.1.x-dev

» 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Comment #39

mikelutz

He/Him

English

Michigan, USA

CreditAttribution: mikelutz as a volunteer commented 26 June 2023 at 12:44

Status:

Postponed (maintainer needs more info)

» Closed (works as designed)

Closing this, as it's ultimately not something we want to do to in the core plugins.

I'm still not quite sure the point of the patch, as the parent getEntityId is expected to always return a string, although it's technically returning whatever is in the destination array under the entity type's id key (like nid for nodes), so I suppose you could write a migration with a process that set nid to an array and the system would just return that. But that's not how the core destination plugins are designed to be used. The core destination plugins can import multilingual data when used properly, and they are indifferent to the source and processes used, provided the end result is a destination array in the format the destination plugins are expecting.

So in general, I would say if you are using the core plugins with custom sources and migrations, you are responsible for confirming the ultimate output of that into the format needed for the core destinations. If your source data isn't conducive to doing that easily and requires custom destination processing as well, by all means do so in a custom destination as well.

Multilingual Migrations for Non-Drupal Sources

Motivation

Proposed resolution

Groundwork

Comments

Parent issue

Related issues

Referenced by