Problem/Motivation

In Drupal 6, you can use the I18n Taxonomy module to translate a taxonomy vocabulary and taxonomy terms. There are two different multilang concepts: 'localized' and 'per language'. This issue is about the 'localized terms' concept.

The 'Localized terms' concept is explained at https://www.drupal.org/docs/8/upgrade/upgrading-multilingual-drupal-6-to...

Proposed resolution

Scope of this issue is to migrate D6 taxonomy term translations. In practice:

  • Have a vocabulary in D6 with 'Localized terms' setting.
  • Create a term in this D6 vocabulary.
  • Translate the term title and description in D6 at admin/build/translate
  • Expected result and scope of this issue: The translation of the term is migrated to Drupal 8.

Out of scope for this issue:

This issue is only about migrating the term translations as described above.

Remaining tasks

a) Make a patch with tests.
b) Test manually. jhodgdon's test instructions are in comment #12.

c) Fix the Postgres / SQLite issues

d) Commit.

User interface changes

N/A

API changes

N/A

Data model changes

N/A

Original issue description by @jhodgdon

  • You mark a particular vocabulary as having translatable terms. (In my site, I had one that was marked as translatable and one that wasn't).
  • If you go to the taxonomy page, like admin/content/taxonomy/5, you cannot see the translated taxonomy terms. But you can translate them by going to the central UI (locale) translation page (admin/build/translate), so the translated terms are stored in the same database tables as the UI translation strings. To translate, click on Search, and filter by Taxonomy, and if you have the i18n_taxonomy module enabled, you will see the translatable taxonomy terms there to translate.
  • After translating, if you view a translated node with one of the terms on it, and view it in Spanish, the term will show up in Spanish. Also, I made a block view that lists the tags (the translated vocabulary), and when viewing the site in Spanish, this block shows up in Spanish also.
  • In the D6 database:
    - term_data has the taxonomy term information
    - i18n_strings correlates term IDs to locale string IDs
    - locales_target has the actual translations in it
    Here's a query that illustrates the relationships:
    SELECT td.tid, td.name, lt.language, lt.translation FROM term_data td LEFT JOIN i18n_strings ist ON td.tid = ist.objectid LEFT JOIN locales_target lt ON ist.lid = lt.lid WHERE ist.type = 'term'

At this time, taxonomy terms translated in this way cannot be migrated into Drupal 8. If you have a Drupal 6 site with this setup, you get the English terms migrated in, and the English name for the vocabulary, but the translations are not migrated. The resulting site has the following in the database, after migrating from an actual D6 site with internationalized taxonomy terms (one vocabulary with English/Spanish terms, and one with just English terms):

  • Table migrate_message_d6_taxonomy_term_translation is there but empty, so Migrate thinks it did the translation migration with no messages.
  • Table migrate_map_d6_taxonomy_term_translation has 36 rows. 36 is the total number of taxonomy terms on the d6 site. 23 were from the translated vocabulary. The other 13 from the untranslated vocabulary.
  • Table taxonomy_term_field_data has 36 rows. Each one is an English taxonomy term. The langcode field is blank for all entries. There are no Spanish entries.

Note that #2784371: Migrate D6 i18n taxonomy term translation (but not yet localized translations) migrated term language, and #2225781: Migrate D6 i18n taxonomy vocabularies migrated vocabulary language, but neither of them migrated translations. Also, once this is done, we can look at also making sure the translated nodes get the translated taxonomy terms, on #2859297: Migrate taxonomy term references for D6 Node translations.

CommentFileSizeAuthor
#131 interdiff-128-131.txt1.23 KBquietone
#131 2886609-131.patch27.64 KBquietone
#128 2886609-128.patch27.1 KBjofitz
#120 2886609-114.patch27.54 KBheddn
#116 2886609-116-test-only.patch5.97 KBquietone
#114 interdiff-111-114.txt558 bytesquietone
#114 2886609-114.patch27.54 KBquietone
#113 2886609-113-test-only.patch6.24 KBquietone
#112 2886609-111.patch27.81 KBquietone
#111 interdiff-109-111.txt2.88 KBquietone
#109 interdiff-107-109.txt597 bytesquietone
#109 2886609-109.patch24.93 KBquietone
#107 2886609-107.patch24.97 KBquietone
#102 interdiff-101-102.txt3.28 KBquietone
#102 2886609-102.patch24.92 KBquietone
#101 interdiff-100-101.txt1.09 KBquietone
#101 2886609-101.patch22.56 KBquietone
#100 2886609-100.patch22.35 KBquietone
#96 interdiff.txt849 bytesquietone
#96 2886609-95.patch22.35 KBquietone
#92 2886609-92.patch22.35 KBquietone
#86 interdiff-2886609-66-86.txt885 bytesmasipila
#86 2886609-86.patch22.34 KBmasipila
#82 2886609-vocabulary-settings.png32.71 KBmasipila
#66 interdiff.txt2.72 KBquietone
#66 2886609-66.patch22.3 KBquietone
#64 interdiff.txt2.09 KBquietone
#64 2886609-64.patch22.33 KBquietone
#58 interdiff.txt3.37 KBquietone
#58 2886609-58.patch22.47 KBquietone
#50 warning.png29.83 KBjhodgdon
#45 2886609-45.patch22.44 KBjofitz
#45 interdiff-43-45.txt1.72 KBjofitz
#43 2886609-43.patch22.17 KBquietone
#35 interdiff.txt1.31 KBquietone
#35 2886609-35.patch22.43 KBquietone
#32 2886609-32.patch22.24 KBquietone
#30 2886609-30.patch42.32 KBquietone
#30 interdiff.txt687 bytesquietone
#28 2886609-28.patch42.33 KBquietone
#28 interdiff.txt1.65 KBquietone
#26 2886609-26.patch42.68 KBquietone
#24 interdiff.txt2.87 KBquietone
#24 2886609-24.patch208.15 KBquietone
#22 interdiff.txt1.98 KBquietone
#22 2886609-22.patch207.96 KBquietone
#20 2886609-20.patch207.94 KBquietone
#18 2886609-18-fixture.patch4.99 KBquietone
#9 2886609-9.patch4.16 KBquietone
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

jhodgdon created an issue. See original summary.

jhodgdon’s picture

mikeryan’s picture

Status: Active » Postponed (maintainer needs more info)

@jhodgdon: The other issue introduces a d6_taxonomy_term_translation migration, that should be migrating the terms. Did that migration run when you tested it? If so, did it generate any errors?

jhodgdon’s picture

Category: Task » Bug report
Status: Postponed (maintainer needs more info) » Needs work

On the advice of mikeryan in IRC, I found the following information in the database:

  • Table migrate_message_d6_taxonomy_term_translation is there but empty, so Migrate thinks it did the translation migration with no messages.
  • Table migrate_map_d6_taxonomy_term_translation has 36 rows. 36 is the total number of taxonomy terms on the d6 site. 23 were from the translated vocabulary. The other 13 from the untranslated vocabulary.
  • Table taxonomy_term_field_data has 36 rows. Each one is an English taxonomy term. The langcode field is blank for all entries. There are no Spanish entries.

So, making this a bug report because the migration is running but it's not working correctly.

One more piece of information... In the D6 site, my recollection is that if you use i18n taxonomy to translate terms (as was done in this site that was migrated), the translations are stored with the UI translations, not with the taxonomy terms. So for instance if you go in the d6 site to the taxonomy page like admin/content/taxonomy/5, you cannot see the translated taxonomy terms. But if you go to a node with one of the terms on it, and view it in Spanish, the term will show up in Spanish, and a view I had that made a tag list, when viewing the site in Spanish, shows up in Spanish also.

jhodgdon’s picture

Status: Needs work » Active
Gábor Hojtsy’s picture

@mikeryan: the translation migrations in #2784371: Migrate D6 i18n taxonomy term translation (but not yet localized translations) were done to account for explicit language info on taxonomy terms, which is one of the modes i18n can work. However when translations are directly on the term ("localizations" in i18n lingo) that is not yet supported.

jhodgdon’s picture

Yes, that is how the taxonomy terms on my d6 site were translated. The terms themselves apparently had no language on them in the D6 site, but as the default language of the site was English, they were in English. Then I used i18n Taxonomy to translate them, which basically put the translations of the terms into the same database tables that store translations of the UI.

I can provide database table dumps of various tables in my d6 site if that would be helpful.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

quietone’s picture

Status: Active » Needs review
FileSize
4.16 KB

Updated d6_taxonomy_term_translation.yml and the test. The migration needs translations: true on the destination. And then added a migration_lookup to the tid, renamed migration to migration_lookup. And finally skip the row if the language is empty. In the test fixture the language field only has data when the term has been translated.

jhodgdon’s picture

Do you think this is a viable patch, and would you like me to test it on the migration that caused me to create this issue? Let me know and I can test.

quietone’s picture

Yes, please test again.

jhodgdon’s picture

Status: Needs review » Needs work

OK, I tested as follows:

a) Downloaded latest drupal 8.5.x dev tgz file from drupal.org (easier than git for me for Drupal core at the moment).
b) Applied this patch using the patch command. 2 files patched.
c) Installed Drupal in this site from UI, using the Minimal install profile, configured site without Update module.
d) Turned on Seven/Bartik themes, and a bunch of core modules: Migrate, Migrate Drupal *UI, Migrate Drupal UI; all 4 multilingual modules; Toolbar; plus some my migrated d6 site would need: Taxonomy, Views, a bunch of field modules, custom blocks and custom menu items.
e) Went to /upgrade, clicked Confirm, and entered the site database/files information.
f) On the confirmation screen, the Available Updates section said it was ready to update taxonomy as well as i18ntaxonomy. Clicked Perform upgrade.
g) Waited for the 100 upgrade tasks to complete. The ones related to Taxonomy were nearly the end of the batch.
h) Got a message saying all 100 tasks completed successfully. Looked at the detailed log. There were no errors in the log related to taxonomy (the only errors I think were due to providing the wrong file path, oops).

So... I had one translated vocabulary on my D6 site. I looked at this vocabulary on the new site, and at the vocabulary level, the "Enable translation" checkbox was *not* checked. It said all the terms would be in English, the vocabulary language is English, and no language selector would be shown on the term page. Not good.

Then I looked at the list of terms. They're all English, and there's no obvious way to translate them (no Translate links in the Operations drop-down, just View and Edit). Not good.

So... I do not think this patch works for my site. See issue summary and/or comment #7 here for more about the setup of the site and how the terms are translated... but they're not getting translated in the migration. Setting to Needs Work, because this patch does not work for me to satisfy the issue's goal, which is to migrate translations of taxonomy terms. As far as I can tell, my translated taxonomy terms from D6 are not migrated as translated terms into D8.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

quietone’s picture

Status: Needs work » Needs review

I'm trying to figure out the next steps here. I'm thinking that data has to be added to the fixture.

In the D6 site, my recollection is that if you use i18n taxonomy to translate terms (as was done in this site that was migrated), the translations are stored with the UI translations, not with the taxonomy terms.

What does "the translations are stored with the UI translations" mean? What table(s) is this referring to?

So for instance if you go in the d6 site to the taxonomy page like admin/content/taxonomy/5, you cannot see the translated taxonomy terms

And this is true on the test fixture as well, for some taxonomies that are not set multilingual. For the ones that are multilingual the translated terms are listed.

However when translations are directly on the term ("localizations" in i18n lingo) that is not yet supported.

If this is true, how does one create translation 'directly on the term'? I have only found one way to translate a taxonomy term.

jhodgdon’s picture

Good questions!

Let's see... database tables. Here is a query that illustrates (hopefully) where the data is stored:

SELECT td.tid, td.name, lt.language, lt.translation FROM term_data td left join i18n_strings ist on td.tid = ist.objectid left join locales_target lt on ist.lid = lt.lid WHERE ist.type = 'term' 

So term_data has the taxonomy term information, then i18n_strings correlates term IDs to locale string IDs, then locales_target has the actual translations in it. That is what I meant above: locales_target also stores the translations of the Drupal UI, like what you download from localize.drupal.org.

Since the translations are stored with UI translations, you don't see them on the taxonomy page. And I don't know of any other way to translate terms in D6 other than using the i18n_taxonomy module.

Hope this helps...

quietone’s picture

@jhodgdon, thanks for that. It does help. I thought that is what you meant but I didn't want to make any assumptions.

I think the next step is to get translated terms of the type you have (Is there a name for that type of translation? Localization?) into the source fixture. And to do that I need to know how to do it from the UI. Curiously I too, at least I think I have, have used the i18ntaxonmy module to make the translations but they do not appear in i18n_strings.

Maybe Gábor Hojtsy can explain?

jhodgdon’s picture

Let's see. I still (barely!) have this D6 site running... it's getting harder as PHP updates on my local machine...

i18ntaxonomy_help says:

To search and translate strings, use the <a href="@translate-interface">translation interface</a> pages

which is admin/build/translate

So if you go to for example
admin/build/translate/search
and filter to Taxonomy in the Type filter (bottom of the list of filters), you can see the taxonomy terms to translate.

I don't think they get into i18n_strings until you actually provide translations for them.

Also, you have to edit the vocabulary first to say terms are translated. That is at
admin/content/taxonomy/edit/vocabulary/5
(or whatever number your vocabulary ID is). Under Multilingual Options, set to
Localize terms. Terms are common for all languages, but their name and description may be localized.

Hope this helps...

quietone’s picture

OK, I think I got it. Just uploading a patch for the test fixture, no migrations yet. There is some extra data in it that I think can be removed but I'll look at that later.

Status: Needs review » Needs work

The last submitted patch, 18: 2886609-18-fixture.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
FileSize
207.94 KB

A work in progress.

Status: Needs review » Needs work

The last submitted patch, 20: 2886609-20.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

quietone’s picture

Status: Needs work » Needs review
FileSize
207.96 KB
1.98 KB

Misspelled the class name.

Status: Needs review » Needs work

The last submitted patch, 22: 2886609-22.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

quietone’s picture

Status: Needs work » Needs review
FileSize
208.15 KB
2.87 KB

Fixed coding standard errors, upgrade taxonomy term entity count and change migration to Content (cut/paste error).

Status: Needs review » Needs work

The last submitted patch, 24: 2886609-24.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
FileSize
42.68 KB

Well, more fool me. I've been looking at the errors and not realizing that I wasn't working from HEAD. Starting over.

Status: Needs review » Needs work

The last submitted patch, 26: 2886609-26.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
FileSize
1.65 KB
42.33 KB

Remove an error from the fixture, update entity counts.

Status: Needs review » Needs work

The last submitted patch, 28: 2886609-28.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
FileSize
687 bytes
42.32 KB

Wrong namespace on test file.

quietone’s picture

Status: Needs review » Needs work

There are changes to the fixture that need to be removed.

quietone’s picture

Status: Needs work » Needs review
FileSize
22.24 KB

Removed unwanted changes from the test fixture.

jhodgdon’s picture

There have been a lot of patches in this issue (which is great! I like to see it moving forward)... It is hard for me to tell if each patch is really ready for review or if you just set to Needs Review to see if the tests would pass.

I'm happy to test migrating my D6 site to see if it all works, and review code, when you have a viable patch -- so please let me know when it's really ready for review.

quietone’s picture

Status: Needs review » Needs work

@jhodgdon, yes I should have set back to NW as I want to do some further checks.

quietone’s picture

Status: Needs work » Needs review
FileSize
22.43 KB
1.31 KB

I've done my checks and it all looks good. To avoid processing duplicate rows this add the same check as in #2225587: Migrate D6 i18n menu links to skip the duplicates in prepareRow.

quietone’s picture

Status: Needs review » Needs work

I want to run another manual test next

heddn’s picture

Priority: Normal » Major

Bumping to major as this is a pretty important milestone for i18n migrations. Not sure if it should also be 'Migrate Critical'.

quietone’s picture

Not sure if it should also be 'Migrate Critical'.

I went back and read the meeting notes and the summary in #2905736: Mark Migrate Drupal as stable According to masipila's summary this should be both 'Migrate Critical' and 'Migrate Drupal'.

All i18n upgrade issues must have the 'i18n-migrate' tag
Issues that are blocking migrate_drupal from being stable for multilingual sites must also be tagged as 'Migrate Critical' and 'Migrate Drupal'

Adding those tags, and let's see if I understood the notes correctly.

quietone’s picture

Then maybe not, in the next comment catch suggests these i18n issues do not block stability.

masipila’s picture

Tagging with Migrate critical and Migrate Drupal according to the policy discussion for Migrate Drupal stability criteria.

masipila’s picture

Quietone, just noticed that I cross-podted with your previous comment.

Taxonomt terms are content. I interpret the policy discussion comments by catch and gabor so that this needs to land before we csn declare Migrate Drupal stable for multilingual sites.

Markus

quietone’s picture

Issue tags: +Needs reroll

No longer applies

quietone’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
FileSize
22.17 KB

A reroll

Status: Needs review » Needs work

The last submitted patch, 43: 2886609-43.patch, failed testing. View results

jofitz’s picture

Status: Needs work » Needs review
FileSize
1.72 KB
22.44 KB

Corrected test failures.

quietone’s picture

@Jo Fitzgerald, thanks. I just found my error in making the reroll and was about to make a patch!

quietone’s picture

OK, thanks again Jo Fitzgerald. I think this is ready for manual testing.

@jhodgdon, I hope your offer to test still stands. This is ready for a test.

jhodgdon’s picture

Yes, I will be happy to test! I might be able to do it later today; if not, then I should have more time on Friday.

jhodgdon’s picture

Issue summary: View changes

Updating issue summary a bit, with notes on the database structure from previous comments. Now that I've located the test instructions I wrote out in comment #12 from the last time, I will test following the same script (except starting with 8.6.x this time).

jhodgdon’s picture

Status: Needs review » Needs work
FileSize
29.83 KB

OK, here are the notes and results of my testing, following the steps in comment #12 (except I started with 8.6.x instead of 8.5.x and applied the latest patch in #45):

- Step (f) -- after visiting /upgrade, clicking Continue, and entering the D6 site information, I reached a screen saying there might be ID conflicts when migrating translated content and I might lose data!!! To continue, I had to acknowledge I might lose data, which is pretty strange, because my D8 site is totally empty of content. So, I followed the link to
https://www.drupal.org/docs/8/upgrade/known-issues-when-upgrading-from-d...
and looked for information relevant to my situation. All I found was:

Possible ID conflicts with translations are not automatically detected.
- If you are executing a complete upgrade at once and you are performing the upgrade to an empty Drupal 8 site, everything should be OK.

So why did I get this warning? There's nothing in my database yet. Here's a screenshot:
Warning when attempting to import translated data
I don't understand why it says it might overwrite data, when I have no data in my database to overwrite? I guess that's a separate issue though.

- End: it said it completed 96 tasks successfully. I looked at the migrate log, and it had a bunch of errors like
Source ID 773: Unable to get entity
but nothing about taxonomy, and it said taxonomy completed successfully, including "Upgraded Taxonomy localized term translations (processed 23 items total)".

So I went to admin/structure/taxonomy ... When I go to the vocabulary that was translated on the old site:
-- The settings for the vocabulary are incorrect. It does not have the "Enable translation" box checked.
-- Individual terms do not appear to be translated (because of this setting), and there is no Translate link in the operations for individual terms.

So, I edited the vocabulary settings -- checked "Enable translation" and saved. Now, when I list the terms, each one has a Translate link in its operations, and if I click through, it shows that the Spanish translations are there. So, that's good -- they were migrated. But this patch needs work, because if it migrates translations, it should also mark the vocabulary as having translated terms, wouldn't you think?

I also checked on the nodes. I can view a node in English that has 2 of these translated taxonomy terms on it. But when I view it in Spanish, it has no taxonomy terms shown. So, that's still a bug, but it's a separate issue #2859297: Migrate taxonomy term references for D6 Node translations (which is currently postponed on this issue).

Also, is that confirmation screen a bug? Should I file an issue? It is crazy that it warns me about LOSING DATA ?!?!?!?!? when I have no data in my site that I might lose.

quietone’s picture

@jhodgdon, thanks for the quick response. I'm not awake enough to respond to your questions but I have one for you! Was the empty D8 site installed with the standard install or minimal?

jhodgdon’s picture

Minimal. I followed the steps in comment #12, so: installed with Minimal, then turned on a bunch of modules (which didn't include forum, which would have added its own taxonomy).

heddn’s picture

The reason for the warning is because we do not audit for conflicts on i18n. Period. It was spun out of #2876085-92: Before upgrading, audit for potential ID conflicts into #2905759: Before upgrading, audit for potential i18n ID conflicts. Since it isn't clear with the current wording, we should update the handbook page to make it more clear about potential issues.

jhodgdon’s picture

I understand there can be potential issues. But the warning says something like "WATCH OUT YOU MAY LOSE DATA" and I don't see how that can be if I have no existing data to lose? The message is misleading, as is the button I have to click to continue that sounds like a legal disclaimer.

Updating the docs page to explain that the message is misleading would be helpful, but it isn't going to help people who cannot read English and would be reading just that page (translated into their own language), and it just seems like the message shouldn't talk about data loss if there is no possibility of losing data I don't have?

quietone’s picture

Status: Needs work » Needs review

Here are the points I see #50 and then my thoughts.

1) Troublesome Language in the UI.
I agree with jhodgson that this needs improvement. I checked the existing UI issues and there is none that is specific to this problem so a new issue is needed.
2) Source ID 773: Unable to get entity.
Appears unrelated to this issue. Maybe a separate issue to track down what entity is causing the problem?
3) Vocabulary translation not enabled at /admin/config/regional/content-language.
Move this to a follow up. Why? One is that it is a separate migration and will have its own challenges that should not hold this up. Two, these i18n tend to be complicated and best to be done is small steps. Three, I have taken a look and it is not obvious where to the source data will come from for the vocabulary content language settings. There is no equivalent 'translate this vocabulary' in the Drupal 6 dump nor do I see such a thing when reading the i18ntaxonomy module files. Four, this patch hasn't had a review yet so I expect there will be things here to do. Five, I really dislike rerolling patches that have fixture changes and they just get harder the longer it takes to get it committed.
4) Taxonomy terms not shown on node.
Being handled in #2859297: Migrate taxonomy term references for D6 Node translations
5) The translations are present on the edit taxonomy page.
Yay! This is what this issue was to do, migrate the translated taxonomy terms.

Setting to NR for feedback.

jhodgdon’s picture

Regarding item 3, I created a new issue #2975509: Migrate D6 vocabulary language settings and added it to the parent issue.

I also created #2975518: Add audit of translation content but wasn't sure if I should put it on the parent issue, so I didn't.

Agreed that "unable to get entity" message is not related to this issue. I think it's about files, not sure.

Anyway, I guess if the scope of this issue is only to migrate the term translations, my manual testing shows they are migrated. I haven't reviewed the code or the tests.

masipila’s picture

Status: Needs review » Needs work

A couple of documentation nits.

1. Why are we talking about menu links in the context of taxonomy term migration?

+++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/TermLocalizedTranslation.php
@@ -0,0 +1,99 @@
...
+  public function query() {
+    // Ideally, the query would return rows for each language for each menu link
+    // with the translations for both the title and description or just the
+    // title translation or just the description translation.

2. Same thing here

+  public function fields() {
+    $fields = [
+      'language' => $this->t('Language for this menu.'),

3. Typo

+      'name_translated' => $this->t('Term nametranslation.'),

4. Class documentation should start with a third person verb i.e. 'Tests migration...'

+++ b/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateTermLocalizedTranslationTest.php
@@ -0,0 +1,140 @@
+ * Test migration of translated localized taxonomy terms.

5. Same thing applies to method descriptions

+   * Validate a migrated term contains the expected values.

6. And here

+   * Assert that a term is present in the tree storage, with the right parents.

Cheers,
Markus

quietone’s picture

Status: Needs work » Needs review
FileSize
22.47 KB
3.37 KB

@masipla, thanks for the review.

This should fix all points in #57.

masipila’s picture

Hi! Superb as always @quietone!

@jhodgdon has done manual testing, I have reviewed the patch and my feedback has been addressed. The interdiff shows that the changes were doc changes only since the manual testing so the manual test results are still valid. The test coverage looks good to me.

Of course it doesn't hurt if somebody else can give this another pair of eyeballs but this looks good to me.

Markus

quietone’s picture

thanks masipila.

One thing about this though is that the migration is in content_translation but the migration does a lookup on d6_taxonomy_vocabulary. If that doesn't exist then migration_lookup returns NULL and the row is skipped. There should probably be a comment to that effect in the migration.

+++ b/core/modules/migrate_drupal/tests/fixtures/drupal6.php
@@ -47247,8 +47453,8 @@
+  'access' => '1521962400',
+  'login' => '1521961881',

This change can be reverted.

heddn’s picture

+++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/TermLocalizedTranslation.php
@@ -0,0 +1,99 @@
+class TermLocalizedTranslation extends Term {

Based on #60, should we do a module_handler check that taxonomy module is enabled on D8? In this class' checkRequirements?

quietone’s picture

Since the migration is in content_translation it won't be discovered unless content_translation is enabled. While thinking and poking around I see that d6_node_translation.yml declares 'destination_module: content_translation' which none of the i18n migrations due. That looks like the way to do and then we can put the i18 migrations and source plugins in their respective module directory instead of content_translation. I still just want to check what the UI does in that case, just to be sure.

heddn’s picture

Assigned: Unassigned » quietone

Assigning to quietone for review.

quietone’s picture

Yea, that wasn't right. What I didn't realize is that d6_node_translation.yml can be in the node module because the deriver will prevent the creation of the derivatives when content_translation is not enabled. There isn't something similar for i18n migrations, so they go into content_translation and will only be discovered when that is enabled.

should we do a module_handler check that taxonomy module is enabled on D8? In this class' checkRequirements

Surely the source plugin should only have requirements based on the source. Unlike most source plugins the i18n ones require several source tables and there is no check that all those tables exist. Are there any source plugins that check that the source tables exists? Yes, 3, in d6/Node.php, d7/user.php and d6/User.php so it is not a common practice.

So, this patch now puts the files in directories just like the already committed d6_menu_links. The migration goes into content_translation and all other files in the respective module directory. The UI shows the new taxonomy migration only if content_translation is enabled. And there is a bug fix, to get the source plugin to return a count a change was made to the query.

Status: Needs review » Needs work

The last submitted patch, 64: 2886609-64.patch, failed testing. View results

quietone’s picture

Hmm, I thought I ran the tests. Anyway, correct the namespaces and make sure that the source row has a property 'language'.

quietone’s picture

Assigned: quietone » Unassigned

Ready for review

jhodgdon’s picture

Do you need me to test again?

quietone’s picture

@jhodgdon, thanks for the offer. I don't think it is necessary, the last set of changes didn't affect the migration process itself and were done to be consistent with the structure of the existing menu_link translation files and for the UI.

quietone’s picture

masipila asked me about the changes in #66, specifically about the use of 'ltlanguage' in the query and I see I didn't explain that at all. What I notices was that 'drush ms' was returning a count of 'N/A' for this migration and on closer inspection it wasn't happy about the two language fields in the query, on in locales_target and the other in term data. Adding a field for 'ltlanguage' to the query and then `drush ms` returned to the correct count. And finally, so that a nice property name is available for use, 'ltlanguage' is copied to to the source row as 'language'

masipila’s picture

Status: Needs review » Reviewed & tested by the community

I tested this manually to double check that I'm understanding everything correctly.

Drupal 6 setup

1. Installed D6.38
2. Installed https://ftp.drupal.org/files/projects/i18n-6.x-1.10.tar.gz
3. Enabeld Taxonomy translation and the required Internationalization, String translation, Locale, Content translation
4. Added Finnish as a second language with path prefix and language fallback
5. Added a new vocabulary, vocabulary_a with these settings:
- Localize terms. Terms are common for all languages, but their name and description may be localized.
- Added this vocabulary to Story content type
6. Created a new Story node
- Added a new term 'Yellow' and added a description to this term.
7. I can translate this term (title and description) at admin/build/translate
- I translated both Term title and description to Finnish
8. When I view fi/node/1 I can see the Finnish translation for the term i.e. Keltainen

Drupal 8 setup and migrations

1. Applied https://www.drupal.org/files/issues/2018-06-05/2886609-66.patch on D8
- D8 had Finnish added as a second language

2. I executed the following migrations using Drush:
- upgrade_d6_taxonomy_vocabulary
- upgrade_d6_taxonomy_term
- upgrade_d6_taxonomy_term_localized_translation

Migrations executed without errors. The count on drush migrate-status was shown correctly (Re: #70)

3. The vocabulary 'Enable translation' setting was not migrated as pointed out earlier by @jhodgdon but that will be handled as part of #2975509: Migrate D6 vocabulary language settings and not this issue.
- I enabled this setting manually to my vocabulary

4. When I view the taxonomy term translations on D8, I can see that both taxonomy term name and description translations are correctly migrated.

Conclusion

- I have reviewed the patch and all my feedback has been addressed.
- There is test coverage.
- @jhodgdon has tested this manually as well as I have. Both have reported that the manual tests are OK.
- Follow-ups have been created.
- Testbot is green.
- Let's land this. RTBC!

Thank you very much @jhodgdon and @quietone! And thanks @Jo Fitzgerald for your help as well!

Cheers,
Markus

quietone’s picture

Title: Migrate D6 i18n translations of taxonomy vocabulary/terms » Migrate D6 i18n loacalized translations of taxonomy terms

Gábor Hojtsy pointed in Slack that this isn't migrating vocabularies, just the localized taxonomy term therefor updating the title.

masipila’s picture

Issue summary: View changes

Further updates to IS to help committers.

  • Gábor Hojtsy committed 8c457c9 on 8.6.x
    Issue #2886609 by quietone, Jo Fitzgerald, jhodgdon, masipila, heddn,...

  • Gábor Hojtsy committed b4e8740 on 8.5.x
    Issue #2886609 by quietone, Jo Fitzgerald, jhodgdon, masipila, heddn,...
Gábor Hojtsy’s picture

Version: 8.6.x-dev » 8.5.x-dev
Status: Reviewed & tested by the community » Fixed

Superb, looks good to me too now. Thanks for making the corrections in the issue summary as well, it is important for everyone reviewing what got committed to understand the exact scope and not complain that vocabularies were not even here for example. Now they have pointers to look at.

tacituseu’s picture

Looks like this introduced test failures on all PostgreSQL 9.1 environments:
https://www.drupal.org/pift-ci-job/989536
https://www.drupal.org/pift-ci-job/989684

  • alexpott committed befb1c1 on 8.6.x
    Revert "Issue #2886609 by quietone, Jo Fitzgerald, jhodgdon, masipila,...

  • alexpott committed 478ff5a on 8.5.x
    Revert "Issue #2886609 by quietone, Jo Fitzgerald, jhodgdon, masipila,...
alexpott’s picture

Status: Fixed » Needs work

@tacituseu yep you're right. Reverted. This is likely to need an issue to fix \Drupal\taxonomy\TermStorage::loadTree() to be DB agnostic. ugh.

tacituseu’s picture

Title: Migrate D6 i18n loacalized translations of taxonomy terms » Migrate D6 i18n localized translations of taxonomy terms

Fixing typo.

masipila’s picture

Title: Migrate D6 i18n localized translations of taxonomy terms » Migrate translations for D6 i18n taxonomy 'localized' terms
Issue summary: View changes
FileSize
32.71 KB

Added clarification that this issue is about the 'localized' concept that we have in D6, not about 'per language' concept.

masipila’s picture

If I'm not mistaken, we can fix the PostgreSQL error by using explicit type casting, see https://stackoverflow.com/questions/42269987/understanding-explicit-type...

I remeber seeing this approach in other patches but can't remeber exactly where (and I'm writing this with my mobile on my way to work so can't search the codebase easily).

Should we add the explicit type casting to our query or postpone this to the issue @alexpott was thinking out loud in #80? Does that issue already exist?

Note: this issue is tagged as 'Migrate critical' i.e. this migration is needed to declare Migrate Drupal stable.

Markus

phenaproxima’s picture

Should we add the explicit type casting to our query or postpone this to the issue @alexpott was thinking out loud in #80? Does that issue already exist?

Since it's Migrate critical, I think we should add explicit type casting to work around the Postgres problem, with a comment referencing the follow-up issue (creating it if it doesn't already exist :) That should hopefully be enough to land this.

masipila’s picture

Assigned: Unassigned » masipila

The issue seems to be (as the test fail error message suggests) that term_data.tid is INT(10) and i18n_strings.objectid is VARCHAR(255). We need to cast tid as VARCHAR and then we should be fine. Assigning this to myself.

Cheers,
Markus

masipila’s picture

Let's fasten the seatbelts and see what the testbot thinks of this. This works on MariaDB and should work also on PostgreSQL.

Follow-up created as suggested, see #2980294: Make \Drupal\taxonomy\TermStorage::loadTree() to be DB agnostic.

Cheers,
Markus

masipila’s picture

Status: Needs review » Needs work

Patch #86 resolved one of the PostgreSQL errors but others remain. I can't see how the Sqlite failure would be related.

I can't work on this in the coming days so if anyone else can, it's all yours.

Markus

tacituseu’s picture

The SQLite failure is unrelated and present since at least January (see https://www.drupal.org/pift-ci-job/866534).

jhodgdon’s picture

Also, do we need to add another issue to take care of migrating the vocabulary translations, since this is now not part of this issue?

masipila’s picture

Re #89

Do you mean migrating the vocabulary settings i.e. Localized terms vs. Per language terms? This is handled in #2975509: Migrate D6 vocabulary language settings.

Or do you mean migrating the translation of vocabulary name and description? Was that covered in #2225781: Migrate D6 i18n taxonomy vocabularies? I have not tested this part myself so I don't know if that already works.

Or did you mean something else?

Cheers,
Markus

jhodgdon’s picture

Oh yeah, sorry, forgot about the other issue.

quietone’s picture

Status: Needs review » Needs work

The last submitted patch, 92: 2886609-92.patch, failed testing. View results

quietone’s picture

Version: 8.5.x-dev » 8.7.x-dev
Status: Needs work » Needs review

Change to 8.7.x

quietone’s picture

Status: Needs review » Needs work

Needs work for the Postgres Failure

quietone’s picture

I like what masiplia did in #86 but I think the cast should be in objectid instead. The errors are showing that the tid is a string where it should be an int. So, lets keep the tid as an int but cast the objectid to a int.

Status: Needs review » Needs work

The last submitted patch, 96: 2886609-95.patch, failed testing. View results

quietone’s picture

Arg, now I see that I mislabel my screen sessions and was in the wrong environment when running tests locally. Need to start over!

masipila’s picture

Issue summary: View changes

Updated issue summary as part of clarifying the scope for each of the multilingual vocabulary / term migration issues.

quietone’s picture

This needs another reroll and I started with the patch in #92, deliberately skipping that last patch which just cause more errors.

quietone’s picture

Lets sort the term parents before the assertion. I don't think the order of the parent ids is critical, if that is true, this should be OK.

quietone’s picture

Oh, spring has gone to my head and I put those changes in the wrong file. This add some sorting in the tests to make sure the parent ids are in the same order before asserting.

masipila’s picture

Patch #102 looks promising. Queued for PostgreSQL and SQLite, lets see if the testbot likes it...

quietone’s picture

Thanks masipila, I thought I had started the PostgreSQL and SQLite tests.

+++ b/core/modules/taxonomy/src/Plugin/migrate/source/d6/TermLocalizedTranslation.php
--- a/core/modules/taxonomy/tests/src/Kernel/Migrate/d6/MigrateTaxonomyTermTest.php
+++ b/core/modules/taxonomy/tests/src/Kernel/Migrate/d6/MigrateTaxonomyTermTest.php

I must admit I am not sure why the sorting is necessary in this test since this migrations this test runs are not altered by this patch.

catch’s picture

Is it because of the new terms being added to the test database? Might be worth a patch with just that change, and with and without the test changes to verify either way.

quietone’s picture

Status: Needs review » Needs work

Setting NW for the idea in #105

quietone’s picture

Status: Needs work » Needs review
FileSize
24.97 KB

This needs a reroll

Status: Needs review » Needs work

The last submitted patch, 107: 2886609-107.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
FileSize
24.93 KB
597 bytes

Let's try that again. I have no idea how I made that mistake.

Status: Needs review » Needs work

The last submitted patch, 109: 2886609-109.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
FileSize
2.88 KB

Those failures are unexpected. What has happened is that #2975509: Migrate D6 vocabulary language settings was committed and that had changes to the same variable, i18ntaxonomy_vocabulary, specifically vocabulary_3_i_2_ , in the fixture which required changes to those two tests. This updates the tests for the new i18ntaxonomy_vocabulary value used in this patch.

quietone’s picture

It helps to upload the patch too.

quietone’s picture

Status: Needs review » Needs work
FileSize
6.24 KB
+++ b/core/modules/migrate_drupal/tests/fixtures/drupal6.php
@@ -47421,8 +47627,8 @@
+  'access' => '1521962400',
+  'login' => '1521961881',

This change is not necessary, needs to be removed.

quietone’s picture

Darn. I didn't mean to upload that test only patch. Ignore it.

First, remove the fixture change.

quietone’s picture

Add PostgreSQL and SQLite tests

quietone’s picture

Now add a test only patch. This removes everything from the patch in #114 except the test fixture.

Status: Needs review » Needs work

The last submitted patch, 116: 2886609-116-test-only.patch, failed testing. View results

quietone’s picture

Let's look at the failures with the test fixture only patch in #116.

All three, MySQL, PostgreSQL and SQLite have failures in these test files. These failures are the same across all three database types and make sense given the changes to the fixture.

  • MigrateUpgrade6Test
  • MigrateLanguageContentTaxonomyVocabularySettingsTest
  • MigrateVocabularyFieldInstanceTest

Now, PostgreSQL has two extra failures. in MigrateTaxonomyTermTranslationTest and MigrateTaxonomyTermTest. Both of there are sorting type errors as shown in the follow excerpts. So, making changes in the patch in #114 here makes sense.

 MigrateTaxonomyTermTranslationTest
  Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    0 => '4'
-    1 => '5'
+    0 => '5'
+    1 => '4'
 1) Drupal\Tests\taxonomy\Kernel\Migrate\d6\MigrateTaxonomyTermTest::testTaxonomyTerms
Term 6 has correct parents in vocabulary tree
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    0 => 4
-    1 => 5
+    0 => '5'
+    1 => '4'

Now, the patch in #114 has an error with PostgreSQL in Drupal\KernelTests\Core\Database\SchemaTest which appears to be unrelated to the changes in #114. Starting a retest of that now.

quietone’s picture

Status: Needs work » Needs review

Good, the retest of the patch in #114 with PostgreSQL passed, it must have been a testbot problem.

So this is all ready for a review. The patch to review is #114

The patch does the necessary migration and to accommodate the different databases the parent id are sorted before asserting them in the tests. This includes sorting in MigrateTaxonomyTermTest, which was unexpected, but a result of the changes to the test fixture , see #118.

heddn’s picture

FileSize
27.54 KB

Adding #114 as the last patch.

quietone’s picture

Add postgres and sqlite tests

masipila’s picture

Assigned: Unassigned » masipila

Assigning for self for review

masipila’s picture

Status: Needs review » Reviewed & tested by the community

Patch #66 was RTBCd in #71 and originally committed in #74-75. It was reverted in #78-79 because of PostgreSQL issues (the patch was originally tested only against MySQL).

The PostgreSQL failures were as follows:

1. PostrgreSQL requires columns to be of same type when joining tables

  • The issue here was that we were doing a join on term_data.tid = i18n_strings.objectid.
  • The type of term_data.tid is INT(10) and the type of i18n_strings.objectid is VARCHAR(255).
  • PostgreSQL requires that the types match.
  • This was resolved by explicitly type casting term_data.tid.

2. Array sorting errors in the tests when using PostgreSQL

Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    0 => '4'
-    1 => '5'
+    0 => '5'
+    1 => '4'

This was resolved in #102 by sorting the arrays in the tests before the asserts:

+++ b/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateTaxonomyTermTranslationTest.php
@@ -108,7 +108,12 @@ protected function assertHierarchy($vid, $tid, array $parent_ids) {
 
     $this->assertArrayHasKey($tid, $this->treeData[$vid], "Term $tid exists in taxonomy tree");
     $term = $this->treeData[$vid][$tid];
-    $this->assertEquals($parent_ids, array_filter($term->parents), "Term $tid has correct parents in taxonomy tree");
+    // PostgreSQL, MySQL and SQLite may not return the parent terms in the same
+    // order so sort before testing.
+    sort($parent_ids);
+    $actual_terms = array_filter($term->parents);
+    sort($actual_terms);
+    $this->assertEquals($parent_ids, $actual_terms, "Term $tid has correct parents in taxonomy tree");
   }

And

+++ b/core/modules/taxonomy/tests/src/Kernel/Migrate/d6/MigrateTaxonomyTermTest.php
@@ -104,7 +104,14 @@ public function testTaxonomyTerms() {
 
       $this->assertArrayHasKey($tid, $tree_terms, "Term $tid exists in vocabulary tree");
       $tree_term = $tree_terms[$tid];
-      $this->assertEquals($values['parent'], $tree_term->parents, "Term $tid has correct parents in vocabulary tree");
+
+      // PostgreSQL, MySQL and SQLite may not return the parent terms in the
+      // same order so sort before testing.
+      $expected_parents = $values['parent'];
+      sort($expected_parents);
+      $actual_parents = $tree_term->parents;
+      sort($actual_parents);
+      $this->assertEquals($expected_parents, $actual_parents, "Term $tid has correct parents in vocabulary tree");
     }

The sorting topic was investigated further in #103 - #118. The final fix is in #114 and there is a test-only patch in #116. Analysis of this in #118 is accurate in my eyes.

I reviewed patch #114 and it looks good to me. I compared it to the patches #66 which was originally committed.

  • The explicit type casting change is in the source plugin, which I manually tested in #86.
  • All other changes are in the tests i.e. no need for manual testing.

Tests are green on all three DB backends so this is good to go. RTBC.

Markus

masipila’s picture

Assigned: masipila » Unassigned
masipila’s picture

Queued the last patch for a re-test to double check that the patch still applies without re-rolls.

The last submitted patch, 114: 2886609-114.patch, failed testing. View results

masipila’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs re-roll

And it didn't apply anymore... Needs re-roll.

jofitz’s picture

jofitz’s picture

Status: Needs work » Needs review
Issue tags: -Needs re-roll
masipila’s picture

Status: Needs review » Needs work

Thanks for the re-roll Jo!

We currently have tests in core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateTaxonomyTermTranslationTest.php

Patch #114 moves this from content_translation module to taxonomy module:

diff --git a/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateTaxonomyTermTranslationTest.php b/core/modules/taxonomy/tests/src/Kernel/Migrate/d6/MigrateTermLocalizedTranslationTest.php
similarity index 63%
copy from core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateTaxonomyTermTranslationTest.php
copy to core/modules/taxonomy/tests/src/Kernel/Migrate/d6/MigrateTermLocalizedTranslationTest.php
index 3e32406d28..4c5e216163 100644
--- a/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateTaxonomyTermTranslationTest.php
+++ b/core/modules/taxonomy/tests/src/Kernel/Migrate/d6/MigrateTermLocalizedTranslationTest.php

In patch #114 we preserve the history of the original tests and only do a diff to the changes.

In the re-rolled patch #128 we're adding everything as a new file:

diff --git a/core/modules/taxonomy/tests/src/Kernel/Migrate/d6/MigrateTermLocalizedTranslationTest.php b/core/modules/taxonomy/tests/src/Kernel/Migrate/d6/MigrateTermLocalizedTranslationTest.php
new file mode 100644
index 0000000..4c5e216
--- /dev/null
+++ b/core/modules/taxonomy/tests/src/Kernel/Migrate/d6/MigrateTermLocalizedTranslationTest.php

1. I believe the approach in #114 is how we should do thing.

2. Question: I'm a bit rusty but how did our Migrate Drupal Multilingual things go?
At least in MigrateTermLocalizedTranslationTest we don't enable Migrate Drupal Multilingual at the moment so have we missed something?

+
+  /**
+   * {@inheritdoc}
+   */
+  public static $modules = [
+    'content_translation',
+    'language',
+    'menu_ui',
+    'node',
+    'taxonomy',
+  ];

Markus

quietone’s picture

130.1. Personally, I find it isn't always possible to make an interdiff on a reroll. Certainly interdiff itself will fail. Anyway, that is my experience.
130.2 Yes, this should be added. The Multilingual tag is checked in MigrationConfigurationTrait which is for the UI. Originally, that check for the Multilingual tag was in checkRequirements on the source plugin, but that was a bit of a hack and didn't catch the followup migrations so it all got moved to the previously mentioned trait. The tags are now added and test updated.

quietone’s picture

maxocub’s picture

Status: Needs review » Reviewed & tested by the community

I've read through the patch, IS & comments, I think everything has been address (automated testing, manual testing, postgres failures).
Let's get back at RTBC.

catch’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/core/modules/taxonomy/src/Plugin/migrate/source/d6/TermLocalizedTranslation.php
@@ -0,0 +1,100 @@
+
+    // Add in the property, which is either name or description.
+    // Cast td.tid as char for PostgreSQL compatibility.
+    $query->leftJoin('i18n_strings', 'i18n', 'CAST(td.tid AS CHAR(255)) = i18n.objectid');

This could use an additional explanation as to why it's necessary for postgres compatibility, it's a bit strange having postgres specific code in core, although maybe it's OK to workaround like this for a migration.

heddn’s picture

Status: Needs work » Reviewed & tested by the community

We do that in several places in migrations. See MenuLinkTranslation as one example and reference #3001749-23: Migrate D7 i18n custom blocks.

$query->leftJoin(static::I18N_STRING_TABLE, 'i18n', 'CAST(ml.mlid as CHAR(255)) = i18n.objectid');

This code was already committed once and reverted around #77 for postgres failures. This casting just gets us past that.

catch’s picture

Version: 8.7.x-dev » 8.6.x-dev
Status: Reviewed & tested by the community » Fixed

OK that's fair enough.

Committed and pushed a3072cba96 to 8.7.x and 36ba6ba3de to 8.6.x. Thanks!

  • catch committed a3072cb on 8.7.x
    Issue #2886609 by quietone, Jo Fitzgerald, masipila, heddn, jhodgdon,...

  • catch committed 36ba6ba on 8.6.x
    Issue #2886609 by quietone, Jo Fitzgerald, masipila, heddn, jhodgdon,...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.