Migrate vocabulary translations and taxonomy term references for Drupal 7 node translations

This is just the D7 version, #2859297: Migrate taxonomy term references for D6 Node translations ss the D6 versin.

I've got a patch for this but need to do more testing.

Started from the d6 issue and made migration for Vocabulary and Taxonomy Term. I also added tests to MigrateNode to confirm that the correct translation was attached to the nodes. I discovered that the drupal7 fixture, despite my efforts, had wonky lids in the locales_target table. I found that out when clicking around the translations via the UI. So there are changes here to fix that.

There are two patches here, both of which fail but for different reasons. The no-node-translation removed d7_node_translation from the MigrateNode test. In this case the test fails when testing if there is an Icelandic translation for node 2, which is correct. The point is that before that assertion there is an assertion for the taxonomy term reference 'field_vocab_fixed' on node 2 that passes, which is what the taxonomy_term migration does. However, when the d7_node_translation is run, the assertion for the 'field_vocab_fixed' fails because it is NULL. And what is curious about that is that field_vocab_fixed is correct after the run of d7_node. But after d7_node_translation the value of that field is removed. There is NOT a translation of field_vocab_fixed for node 2, icelandic, and when that row is saved there isn't a property for that field in the row and somehow that removes the field contents on the node. I hope that makes sense to you.

This is odd. Working with the D7 fixture, I have seen that the fixed language terms do not work as I would expect based on this from
/admin/structure/taxonomy/vocabfixed/edit
Fixed Language. Terms will have a global language and they will only show up for pages in that language.

The term, which is in French, can appear on the English and Icelandic version of the article. That doesn't make sense to me.

What is meant by 'Per language' vocabulary? The options are

• Localize. Terms are common for all languages, but their name and description may be localized.
• Translate. Different terms will be allowed for each language and they can be translated.
• Fixed Language. Terms will have a global language and they will only show up for pages in that language.

Kia ora masipila!

Thanks for the explanations. For #1, I expected that 'per language' was the same as 'translate'. Good to have that confirmed. For #2/3, Yes, he vocabulary (vocabfixed) is fixed, with a language of 'fr' in the 'taxonomy_vocabulary' table. It has one term, 'FR - Crewman' which also has a 'fr' language in the 'taxonomy_term_data' table. And yet when I add an 'en' language node, that French language term is a valid choice. A bit strange, but that is how D8 behaves.

So, moving on. This patch correctly migrates localized and translated vocabulary are per the proposed resolution in the IS. However, d7_node_translation breaks the migration of the FixedVocabulary. Details of that are in #12.

Not sure how to fix this one. Any ideas?

And yet when I add an 'en' language node, that French language term is a valid choice. A bit strange, but that is how D8 behaves.

Drupal 8 does not filter selectable terms based on language in core. You could install a contrib to do that. The thinking is you may want to actually translate the terms (or some terms) and still be able to select them. Same goes for menu item parents, etc. Such UI filtering is not available in core.

Wait a minute, rechecked with both D7 and D8 with a vocabulary fixed taxonomy and find that D7 behaves differently than D8. In D8, the term is always the same on all the translations of the node whereas in D7 the term can be NULL on one of the translated nodes. I can't get that to happen in D8. Plus, the D7 fixed has that case, where the term is 'FR - Crewman' on one of the translated nodes and NULL on another, that is the problem. If that is all true then term reference for fixed vocabularies can't be migrated to match the source site. The term reference migrated will be the one in the last migrated translated node.

@quietone: are you comparing D7 node or entity translations to D8? If the term was fixed language and cannot be assigned to translations it makes sense that the D7 translation would not have it assigned. D8 does not have a setting to prohibit assigning it, but it does have a setting to fix it to a specific language. So whether you assign it or not depends on whether the field value was set optional (not required) and whether it was set to be translatable (can be different per language). In the case where the D8 field is optional and can be different per language, it can be unset for a certain language. There is no UI or other mechanism to enforce that you cannot pick a different language term though if you want to. So the migration cannot migrate into the same UI behavior but it can migrate to the same data state IMHO.

With that in mind, cleanup the patch from #12 to remove test garbage I left in and modify MigrateNodeTest so that the taxonomy term ref for vocab fixed is tested with a NULL value.

So, to be clear the English version of the node has the term ref for the fixed vocab as 'FR - Crewman' and the Icelandic term ref is empty/NULL. The migration will use NULL for the term for for both the English and Icelandic versions of the node. That is not a match for the D7 source but the D7 and D8 behave differently so I don't see what else can be done. This will need some documentation.

And correct the migration states.

Here is snippet of D7 code for translation vocabulary fixed, from i18n_taxonomy.module, https://git.drupalcode.org/project/i18n/blob/7.x-1.x/i18n_taxonomy/i18n_...

        // We translate just some vocabularies: translatable, fixed language
        // Fixed language ones may have terms translated, though the UI doesn't support it
        if ($mode & I18N_MODE_LANGUAGE || $mode & I18N_MODE_TRANSLATE) {
          $translation[$index] = i18n_taxonomy_translate_terms($tdata, $langcode, $fullterms);
        }
        elseif ($fullterms) {
          $translation[$index] = array_map('_i18n_taxonomy_filter_terms', $tdata);
        }
        else {
          $translation[$index] = array_map('_i18n_taxonomy_filter_tids', $tdata);
        }

Here is states that for the fixed mode (i18N_MODE_LANGUAGE) that that, in fact, the terms can be translated though it can't be done via the UI. So, the source can have translations for a vocabulary fixed field but D8 can't. The way the migration runs the last translated value will be migrated. In a way that is what is seen in the test where the first value migrated is 'FR - Crewman' and that is overwritten by the last translated value, which is NULL. AFAICT there will be data loss here.

Maybe the migration of fixed vocabulary needs to be smarter and get the value of the translation from the node of the same language. I don't know.

Does the fixture having translation mean that the i18n module itself would allow those translations or is it just that the fixture has them added? If the later (which I assume) then we need the fixture fixed not the migration.

I created the fixed vocabulary from the UI and from the UI I can set the English version of the node to use the fixed vocab term 'FR - Crewman' and the Icelandic term ref to empty/NULL. Yes, just from the UI and I have triple checked that.

We seem stuck here because of the behavior of vocab fixed. I have checked that behavior with the fixture and with a vanilla D7 site and they behave the same, which his different that D8. Can anyone confirm that?

If my results are correct, we need to make a decision about how to migrate terms with a fixed vocabulary.

OK, Thanks to Gábor Hojtsy and simplytestme I finally see what is going on. Pretty sure this is about the 'Users may translate this field setting' in regard to the vocabulary fixed.

Found an error in the test where it was testing the node and not the translation. Fixed that and changed the term references for the fixed vocabulary to properly reflect what is in the D7 source db. It will fail tests until #3080492: Migrate fixed vocabulary term reference fields as translatable to conform to data expectations when nodes get merged is in.

Coding standard fix

Now apply the patch from #3080492: Migrate fixed vocabulary term reference fields as translatable to conform to data expectations when nodes get merged and see if everything is fixed.

The test only patch passed tests so just need to postpone this on #3080492: Migrate fixed vocabulary term reference fields as translatable to conform to data expectations when nodes get merged and when that happens use the patch in #31.

#3080492: Migrate fixed vocabulary term reference fields as translatable to conform to data expectations when nodes get merged is in 8.8 :) No time yet to review.

Reuploading #31 just for posterity to have before-after results kept/shown properly.

#3080492: Migrate fixed vocabulary term reference fields as translatable to conform to data expectations when nodes get merged was not yet pushed. Now that it is, sending for a retest.

@masiplia, thx for running tests against sqlite and postgres.

I can't get lando to bring up a postgres so I am taking my best guess at fixing this without the ability to test locally.

Arg, ignore that patch. On the bright side I've got postgres setup in one of my LXC containers.

Starting over from the patch in #36.

The test failure with postgresql is not related it is in Drupal\Tests\media_library\FunctionalJavascript\MediaLibraryTest

HEAD postgres fails for the same reason - see https://www.drupal.org/pift-ci-job/1402626 and #3055648: Frequent random fail in \Drupal\Tests\media_library\FunctionalJavascript\MediaLibraryTest

Updated title and summary based on what is included.

Two things I don't understand:

+    $query = parent::query();
+    $query->leftjoin('i18n_string', 'i18n', 'CAST (v.vid AS CHAR(222))= i18n.objectid');
+    $query->innerJoin('locales_target', 'lt', 'lt.lid = i18n.lid');
+    $query
+      ->condition('type', 'vocabulary')
+      ->fields('lt')
+      ->fields('i18n');
+    $query->addField('lt', 'language', 'lt.language');
+    $query->addField('lt', 'lid', 'lt_lid');

lt.language would already be added by adding all fields of lt, no?

+      'format' => $this->t('The {filter_format}.format of the string'),
+      'ltlanguage' => $this->t('Language code from locales_source table'),
+      'plid' => $this->t('Parent lid'),

That would be from the locales_target table, not source :)

+    if ($this->getDatabase()
+      ->schema()
+      ->fieldExists('taxonomy_vocabulary', 'language')) {
+      $query->addField('v', 'language');
+    }
[...]
+  public function prepareRow(Row $row) {
+    parent::prepareRow($row);
+    // For ease of reading the migration use 'language' as the property name for
+    // the language.
+    $language = $row->getSourceProperty('ltlanguage');
+    $row->setSourceProperty('language', $language);
+  }

Would this unconditionally overwrite the language from the query with ltlanguage at all times? Why have the language in the query then?

@Gábor Hojtsy, well spotted and that exposed a problem with getIds() too.

The fields() entries are changed for the two language fields, the 'property' value is added to getIds() and the query adjusted. And now, the prepareRow() is not longer needed.

Oh dear, changing the source plugin requires changing the source plugin test. That is now done.

Ah, more failures with PostgreSQL and they are unrelated to the patch. None are the media problem mentioned in #46

One is failing on HEAD, https://www.drupal.org/pift-ci-job/1410286

Thanks for fixing those, looks good now :)

Updated the IS with the added vocabularies and their translations.

@masipila, Did I get everything?

Thanks, let's get this in!

Committed 932a453 and pushed to 8.8.x. Thanks!

🎉