Problem/Motivation

CCK field options are translated using i18n_cck submodule of the suite i18n, we need to migrate those respecting the language.

Once an the option translation were working and rollbacks could be tested, the rollback on d6_field_option caused fatals. Turns out that EntityFieldStorageConfig does not handle translations. To fix this, EntityFieldStroageConfig::getIds() now adds the language code for translation destinations and EntityFieldStroageConfig::rollback() adds the language code to the destination id array. This array is now unusual in that the first key is 0 and the second key is langcode. Maybe this should be changed.

This then exposed an error in EntityConfigBase::rollback() in a loop that is building the destination identifier where the comparison of $key == 'langcode' always returns true (because $key is an integer) and the entity_id is not built correctly. Changing that comparision to strict fixes that and the rollback work.

Proposed resolution

Patch
Review
Commit

Remaining tasks

Write a test
Write a patch to migrate field labels and options.

Comments

quietone created an issue. See original summary.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

quietone’s picture

Assigned: Unassigned » quietone

Making a start

quietone’s picture

Status: Active » Needs review
StatusFileSize
new30.96 KB

Here is a start. The interesting thing I learned while finding out how this is done in Drupal 8 is that the boolean option is treated differently. The translation labels are stored in the field (field.field) where as for the others I tested put the translations in field storage (field.storage).

Status: Needs review » Needs work

The last submitted patch, 4: 2961114-4.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new2.42 KB
new32.4 KB

The majority of the errors were caused by an error in drupal6.php. Just before I made the patch the IDE was hogging the CPU and being mostly unresponsive, and it seems it was then an extra line got into the fixture.

Also, updated the MigrateUpgrade tests and found that d6_field_option_translation.yml was using the wrong source plugin.

Status: Needs review » Needs work

The last submitted patch, 6: 2961114-6.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new32.76 KB
new1.37 KB

Fix a typo and move i18ncck to the will be upgrade list in MigrateUpgrade6Test

Status: Needs review » Needs work

The last submitted patch, 8: 2961114-8.patch, failed testing. View results

quietone’s picture

Aside from the error on the entity count rollback on d6_field_option_translation causes a fatal.

quietone’s picture

Assigned: quietone » Unassigned
Status: Needs work » Needs review
StatusFileSize
new34.64 KB
new3.66 KB

In order for rollback to work changes are made to EntityFieldStroageConfig and EntityConfigBase. For EntityFieldStroageConfig this was to add the language to the destination identifier array, keyed by langcode, if this is a translation destination. The this exposed that in EntityConfigBase at test was being done ($key == 'langcode') where $key could be an integer. When $key is an integer the if is always true and the destination value is always empty, so a fatal eventually occurs when trying to get an entity with an id of NULL. That comparison is now strict.

Status: Needs review » Needs work

The last submitted patch, 11: 2961114-11.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review

TODO:
Translate another item in field_test_integer_selectlist to check that more that option with a translation migrates successfully.
Learn why node counter test fail.

quietone’s picture

Status: Needs review » Needs work
quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new34.05 KB
new4.61 KB

Added another translation to field_test_integer_selectlist to make sure that multiple translations work properly.
Changed the option translations to have the prefix 'fr - ' or 'zu - ' like the other translations in the fixture. For unknown reasons, I didn't follow that pattern when adding these ones.
Removed changes to the test fixture that are not necessary, which include changes to the node counter table.

quietone’s picture

Issue summary: View changes

Update IS

quietone’s picture

maxocub’s picture

Status: Needs review » Needs work

I was testing this manually and I stumble upon a couple exceptions. First, if you don't have the Profile module enabled on D6 you get this:

PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupal.profile_fields' doesn't exist

Then if you enable the Profile module but don't create any profile you get this:

Drupal\Core\Database\InvalidQueryException: Query condition 'objectid NOT IN ()' cannot be empty. 

I don't see why this migration requires Profile, maybe there's a reason I don't know. In any case, I think we should avoid those exceptions.

Otherwise, I haven't completed my review yet, but I can say that after enabling the Profile module and adding a profile, the migration seems to be working well.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new33.57 KB
new2.09 KB

Yes, the profile stuff can be removed. It was left over from when I first setup the source plugin and didn't escape the underscore in ' ->condition('property', 'option\_%', 'LIKE'). Without escaping the underscore the source plugin was returning profile field options as well as the CCK field options. When I learned about that '_' has special meaning I changed the LIKE statement but overlooked the code relating to profile. This patch fixes that.

heddn’s picture

Status: Needs review » Needs work

Discussed in the migrate maintainers meeting w/ @quietone and @phenaproxima and @heddn and none of think there is a BC break with the destination getIds because we don't have any existing field config destination migrations.

+++ b/core/modules/migrate/src/Plugin/migrate/destination/EntityFieldStorageConfig.php
@@ -58,6 +58,9 @@ class EntityFieldStorageConfig extends EntityConfigBase {
+    if ($this->isTranslationDestination()) {
+      $ids['langcode']['type'] = 'string';
+    }

We should never use conditional logic in a getIds. Fix this in 9.0. Should add an @TODO and link to an issue.

quietone’s picture

As part of doing #20 I decided to make a list of places where ($this->isTranslationDestination()) { is used is getIds() in migrate.

core/modules/migrate/src/Plugin/migrate/destination/Config.php
core/modules/migrate/src/Plugin/migrate/destination/EntityConfigBase.php
core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php
core/modules/migrate/src/Plugin/migrate/destination/EntityFieldInstance.php
core/modules/migrate/src/Plugin/migrate/destination/EntityRevision.php

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new32.11 KB
new750 bytes

Issue made for point raised in #20 and added an @todo.

This needed a reroll as well as adding the @todo comment.

masipila’s picture

Re-parenting so that all multilingual migrations can be found from the same meta. Moved original parent to related issues.

quietone’s picture

Status: Needs review » Needs work
    +++ b/core/modules/field/tests/src/Kernel/Plugin/migrate/source/d6/FieldOptionTranslationTest.php
    @@ -0,0 +1,202 @@
    +        'label' => ' 	Integer Select List Field',
    

    Remove extra whitespace

  1. +++ b/core/modules/field/src/Plugin/migrate/process/d6/FieldOptionTranslation.php
    @@ -0,0 +1,56 @@
    +            if ((isset($value[0]) && ($value[0] == $option)) ||
    

    Document what is expected in value.

  2. +++ b/core/modules/field/src/Plugin/migrate/process/d6/FieldOptionTranslation.php
    @@ -0,0 +1,56 @@
    +    return ['settings.allowed_values.' . $i, $allowed_values];
    

    Use double quotes

In the migrate meeting phenaproxima asked for more documentation on the source plugins as they are very similar. And to look at having the source plugins not be so specific, i.e. gather all data from the tables.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new64.75 KB
new37.04 KB

Fixes for the items in #24, except for the source plugin.

While working on this again I realized that this is only testing list fields with an allowed value list with just a key and not any field with an allowed value list of key|label. So, this adds a new field to the source and updates the test accordingly. I haven't cleaned up the test fixture so that still need to be done.

TODO:

  1. Remove unneeded changes to drupal6.php
  2. Change the source plugin to be not so specific.

@heddn, @phenaproxima, can you remind me what needs to be done to the source plugin. Was the intention to have all the properties in the tables in the row or to remove/reduce the conditions?

Status: Needs review » Needs work

The last submitted patch, 25: interdiff-22-25.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new65.47 KB
new731 bytes

There was a failing test and this should fix it.

heddn’s picture

All properties.

quietone’s picture

StatusFileSize
new66.74 KB
new6.3 KB

Thank you. This updated the source plugins and tests as needed.

quietone’s picture

Status: Needs review » Needs work

Still need to remove unneeded changes to drupal6.php

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new43.24 KB
new45.75 KB

Remove what I hope is unnecessary changes in the fixture.

heddn’s picture

Status: Needs review » Needs work

This might just be a quick question and reply:

+++ b/core/modules/config_translation/migrations/d6_field_instance_option_translation.yml
@@ -0,0 +1,178 @@
+    map:
+      userreference:
+        userreference_select: entity_reference
+        userreference_buttons: entity_reference
+        userreference_autocomplete: entity_reference
+      nodereference:
+        nodereference_select: entity_reference

+++ b/core/modules/config_translation/migrations/d6_field_option_translation.yml
@@ -0,0 +1,144 @@
+    map:
+      userreference:
+        userreference_select: entity_reference
+        userreference_buttons: entity_reference
+        userreference_autocomplete: entity_reference
+      nodereference:
+        nodereference_select: entity_reference
+      number_integer:
+        number: integer

I assume this is a pretty standard list of mappings. And with our work on a base class for entity reference, node reference, user reference, instead of constantly copy/pasting this all over the place, would it make sense to move this in to a trait and do this in code? Or should this be a follow-up to clean these things up?

quietone’s picture

Definitely a follow up, there is enough going on here.

Leaving at NW for the followup to be made.

heddn’s picture

I feel woefully unqualified to RTBC this thing, but I cannot find anything else to comment about. Let's get that follow-up opened and on to RTBC then.

quietone’s picture

Status: Needs work » Needs review
heddn’s picture

Status: Needs review » Reviewed & tested by the community

No more blockers.

gábor hojtsy’s picture

Status: Reviewed & tested by the community » Needs review

Thanks, great work. Some notes:

  1. +++ b/core/modules/config_translation/migrations/d6_field_instance_option_translation.yml
    @@ -0,0 +1,178 @@
    +label: Field option translation
    

    Field instance option translation

  2. +++ b/core/modules/config_translation/migrations/d6_field_instance_option_translation.yml
    @@ -0,0 +1,178 @@
    +      fr_phone:
    +        phone_textfield: telephone
    +      be_phone:
    +        phone_textfield: telephone
    +      it_phone:
    +        phone_textfield: telephone
    +      el_phone:
    +        phone_textfield: telephone
    +      ch_phone:
    +        phone_textfield: telephone
    +      ca_phone:
    +        phone_textfield: telephone
    +      cr_phone:
    +        phone_textfield: telephone
    +      pa_phone:
    +        phone_textfield: telephone
    +      gb_phone:
    +        phone_textfield: telephone
    +      ru_phone:
    +        phone_textfield: telephone
    +      ua_phone:
    +        phone_textfield: telephone
    +      es_phone:
    +        phone_textfield: telephone
    +      au_phone:
    +        phone_textfield: telephone
    +      cs_phone:
    +        phone_textfield: telephone
    +      hu_phone:
    +        phone_textfield: telephone
    +      pl_phone:
    +        phone_textfield: telephone
    +      nl_phone:
    +        phone_textfield: telephone
    +      se_phone:
    +        phone_textfield: telephone
    +      za_phone:
    +        phone_textfield: telephone
    +      il_phone:
    +        phone_textfield: telephone
    +      nz_phone:
    +        phone_textfield: telephone
    +      br_phone:
    +        phone_textfield: telephone
    +      cl_phone:
    +        phone_textfield: telephone
    +      cn_phone:
    +        phone_textfield: telephone
    +      hk_phone:
    +        phone_textfield: telephone
    +      mo_phone:
    +        phone_textfield: telephone
    +      ph_phone:
    +        phone_textfield: telephone
    +      sg_phone:
    +        phone_textfield: telephone
    +      jo_phone:
    +        phone_textfield: telephone
    +      eg_phone:
    +        phone_textfield: telephone
    +      pk_phone:
    +        phone_textfield: telephone
    +      int_phone:
    +        phone_textfield: telephone
    
    +++ b/core/modules/config_translation/migrations/d6_field_option_translation.yml
    @@ -0,0 +1,144 @@
    +      fr_phone:
    +        phone_textfield: telephone
    +      be_phone:
    +        phone_textfield: telephone
    +      it_phone:
    +        phone_textfield: telephone
    +      el_phone:
    +        phone_textfield: telephone
    +      ch_phone:
    +        phone_textfield: telephone
    +      ca_phone:
    +        phone_textfield: telephone
    +      cr_phone:
    +        phone_textfield: telephone
    +      pa_phone:
    +        phone_textfield: telephone
    +      gb_phone:
    +        phone_textfield: telephone
    +      ru_phone:
    +        phone_textfield: telephone
    +      ua_phone:
    +        phone_textfield: telephone
    +      es_phone:
    +        phone_textfield: telephone
    +      au_phone:
    +        phone_textfield: telephone
    +      cs_phone:
    +        phone_textfield: telephone
    +      hu_phone:
    +        phone_textfield: telephone
    +      pl_phone:
    +        phone_textfield: telephone
    +      nl_phone:
    +        phone_textfield: telephone
    +      se_phone:
    +        phone_textfield: telephone
    +      za_phone:
    +        phone_textfield: telephone
    +      il_phone:
    +        phone_textfield: telephone
    +      nz_phone:
    +        phone_textfield: telephone
    +      br_phone:
    +        phone_textfield: telephone
    +      cl_phone:
    +        phone_textfield: telephone
    +      cn_phone:
    +        phone_textfield: telephone
    +      hk_phone:
    +        phone_textfield: telephone
    +      mo_phone:
    +        phone_textfield: telephone
    +      ph_phone:
    +        phone_textfield: telephone
    +      sg_phone:
    +        phone_textfield: telephone
    +      jo_phone:
    +        phone_textfield: telephone
    +      eg_phone:
    +        phone_textfield: telephone
    +      pk_phone:
    +        phone_textfield: telephone
    +      int_phone:
    +        phone_textfield: telephone
    

    Hm, where do these come from? They both look to excessive and too limited if so specific :)

  3. +++ b/core/modules/config_translation/migrations/d6_field_option_translation.yml
    @@ -0,0 +1,144 @@
    +label: Field configuration
    

    Field option translation

  4. +++ b/core/modules/field/src/Plugin/migrate/process/d6/FieldInstanceOptionTranslation.php
    @@ -0,0 +1,53 @@
    +    list($field_type, $global_settings) = $value;
    

    It is not global settings when its about an instance only when its about a field, no?

  5. +++ b/core/modules/field/src/Plugin/migrate/process/d6/FieldInstanceOptionTranslation.php
    @@ -0,0 +1,53 @@
    +    if (isset($global_settings['allowed_values'])) {
    +      $list = explode("\n", $global_settings['allowed_values']);
    
    +++ b/core/modules/field/src/Plugin/migrate/process/d6/FieldOptionTranslation.php
    @@ -0,0 +1,59 @@
    +    if (isset($global_settings['allowed_values'])) {
    +      $list = explode("\n", $global_settings['allowed_values']);
    

    Is allowed values the only thing we need to translate / handle? Prefix/suffix, etc. are not options we would need to translate or is that already covered or covered by another issue?

  6. +++ b/core/modules/field/src/Plugin/migrate/source/d6/FieldOptionTranslation.php
    @@ -0,0 +1,78 @@
    +      'translation' => $this->t('Translation of either the title or explanation.'),
    

    What's an explanation here?

quietone’s picture

StatusFileSize
new43.25 KB
new2.48 KB

37.1. Fixed
37.2. These source migration yml are based on d6_field.yml and that particular process is copied from d6_field.yml. I don't know their history, I just assume they are correct.
37.3 Fixed
37.4 This is really using the global settings from content_node_field. The translations for the booleans are in the field_config not in field_storage so a whole separate migration is needed.
37.5 Good question. I meant to follow that up!
37.6 Fixed.

Also fixed formatting in the yml files.

TODO:
37.5

quietone’s picture

37.5 I brought up a D6 site using the test fixture and tested whether the prefix and suffix need to be translated and migrated. I found that the 'integer field' in the story content type has values in the prefix and suffix field. I refreshed the translation strings but then searched for these strings ('pref' and 'suf) and they are not found. To double check I went back and added prefix ("XYZZY') and suffix ("SUFFIXXYZZY') to the Decimal field, refreshed the translation strings and search for XYZZY and they were not found. So it seems there is no more work to do here.

Unless, of course, I was not doing the correct process to translate those values.

heddn’s picture

Status: Needs review » Reviewed & tested by the community

For #37.2, see #3008328: Create Field Plugins for Drupal 6 phone and number fields. where we plan to fix that up. I think that covers all the questions brought up since last RTBC, so let's try it again.

gábor hojtsy’s picture

Status: Reviewed & tested by the community » Needs review

Thanks.

#3008328: Create Field Plugins for Drupal 6 phone and number fields. does not explain why would phone numbers have such country/language(?) specific variants, is that a phone module thing and not a translation thing to support different phone number formats? I went ahead and checked that at https://cgit.drupalcode.org/phone/tree/phone.module?h=6.x-1.x#n28 and is apparently a funky implementation on Drupal 6 where country codes are represented with one field type each. Hum. Ok. You are doing the right thing here.

I looked at https://cgit.drupalcode.org/i18n/tree/i18ncck/i18ncck.module?h=6.x-1.x and it does indeed only do label/description translation for the fields (migrations for which I believe are already done earlier in Drupal 8). Even Drupal 7's i18n at https://cgit.drupalcode.org/i18n/tree/i18n_field/i18n_field.i18n.inc only extends that with supporting default value translation for text type fields. Again you are doing the right thing here. One minor thing left then:

+++ b/core/modules/config_translation/migrations/d6_field_instance_option_translation.yml
@@ -1,5 +1,5 @@
-label: Field option translation
+label: Field instance option translation

+++ b/core/modules/config_translation/migrations/d6_field_option_translation.yml
@@ -1,5 +1,5 @@
-label: Field configuration
+label: Field option translation configuration

I don't believe "configuration" is used as a migration suffix (for other field config related things or otherwise). Either both should have it or neither.

heddn’s picture

Issue tags: +Novice

Tagging novice. Let's call them configuration in both cases. I checked and we have precedent on that in d7_field_instance.yml and d7_field.yml

quietone’s picture

Status: Needs review » Needs work

Tagging needs work for the task in #41/#42 to improve the label in the yml files.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new43.26 KB
new1.13 KB

Right, let's get this done.

The labels are now "Field option configuration translation" and "Field instance option configuration translation" because they are configuration and translation is normally at the end of the label string for translation migrations.

  • Gábor Hojtsy committed a007373 on 8.7.x
    Issue #2961114 by quietone, heddn, Gábor Hojtsy, maxocub, masipila:...

  • Gábor Hojtsy committed 94b0472 on 8.6.x
    Issue #2961114 by quietone, heddn, Gábor Hojtsy, maxocub, masipila:...
gábor hojtsy’s picture

Status: Needs review » Fixed

Superb, thanks, landed!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.