[META] Remaining multilingual migration paths

> Best case scenario, we detect which settings have the override set, and then pull all the language versions for those config and apply them to the translatable config entities in D8.

That will be slightly interesting because we do not have a special mechanism that would map a variable to a config rather we just have separate migrations for each config file so this would mean we would need to duplicate them for i18n? Check for example core/modules/migrate_drupal/config/migrate.migration.d6_system_site.yml which creates the config file system.site

Tests are superb easy. I'd recommend looking at MigrateAggregatorFeedTest as your starter. You need a DBTNG-format database dump in a class (Drupal6AggregatorFeed), a pretty much copy-paste setUp and then assert that whatever you migrated makes sense.

@Ryan Weal - thanks again for the outline above.

I know I get discouraged when I find something hard and other people tell me how easy it is. Test writing can eventually become routine, but it also requires a different way of looking at things that can take time to wrap your head around!

Your plan to wait for the camp in New York to personally dig in makes sense to me.

In addition to the example @chx pointed to, I believe the following documentation is still relevant for getting started: https://groups.drupal.org/node/387488 I'm not positive it's up-to-date, though.

Added roadmap after discussion with penyaskito, chx & gábor.
We're now about to create a set of issues to get this started.

We discussed with chx and work will happen on drupal6-i18n branch, so we can split the mega-d6-migration-patch and the multilingual parts.

What is the "the drupal6-i18n branch" mentioned at the top of the IS? Do I need to use that to work on Multilingual migrations?

That one is not used anymore, I just edited the issue summary.
Patches are against Drupal core now.

Can we get issue summary update? What is remaining for i18n at this point? With D6 sunset fast approaching, how many of the D6 i18n issues are still a concern?

@heddn, Nearly everything remains to be done and I'm fairly sure there are more issues to create, at least of the variable to config variety.

I've been researching these issues using D6 (never used multilingual before) and the source plugins should be easy, it is the destinations that need input from someone more experienced. Recently, I've been looking into the user profile fields and the destination EntityConfigBase needs to handle language overrides. How to do that 'properly'?

A review of the Config destination plugin at #2225477: Add migrate sources and destinations for D6 i18n variables is needed. It is blocking novice level work.

Added issue links to the IS, primarily to find what is missing.

Created sub issues and moved the original summaries by Ryan Weal to the new issues. Now that it is easier to see all the issues, maybe assigning some priorities is the next step.

Add issue for D7 test fixture.

Just discovered that i18n blocks is different that the blocks data in the i18n_strings tables. So, adding a todo.

@TODO:

• Make sure i18n_string issues are clear and with sensible titles.
• Make sure there is an issue for i18n_string blocks and one i18n_blocks.

Let's keep the individual issues on the D8MI sprint. Unless you keep the sprint tag for migrate sprint issues, we are fine to remove it from here IMHO. This meta should be discoverable from each individual issue.

@morningtime: I agree that is unfortunate, but that is sadly how it is expected at this point. We focused on making the Drupal 6 to Drupal 8 migration path work generally and especially for multilingual. The Drupal 7 core translation and Drupal 6 contrib entity translation to Drupal 8 core translation migrations are still to be done. As per #2225775-188: Migrate Drupal 6 core node translation to Drupal 8 we don't even have an entity_translation issue yet, we do have at least #2669964: Migrate Drupal 7 core node translations to Drupal 8 which would already be a step forward.

@morningtime, the reason the migration does not understand your title translations is because that is a custom module in Drupal 7, just like entity translation and all of those would need migration paths built. Help is very welcome, maybe starting on #2669964: Migrate Drupal 7 core node translations to Drupal 8 would be a good first step towards a solution :)

IS update: Corrected the issue for i18n menu.

Why is "Menu Links: #2669978: Migrate D7 Menu Links" listed here with the i18n issues. Admittedly, I just took a brief look.

Corrected link to taxonomy term issue.

@quietone: it could be that #2669978: Migrate D7 Menu Links was a dependency for the menu translation migration? Not sure why else would it be here, but its complete nonetheless. I think we can remove it now.

I'm running in to the same problems as in #33: the D7->D8 upgrade path does not work when using Entity Translation with "field translation". I guess this related to #2073467: Migrate Drupal 7 Entity Translation settings to Drupal 8.

#2669964 was just committed, so that's another great step forwards!

Added #2073467: Migrate Drupal 7 Entity Translation settings to Drupal 8 to issue summary.

Added #2859297: Migrate taxonomy term references for D6 Node translations

Added child issue #2886609: Migrate translations for D6 i18n taxonomy 'localized' terms to summary.

This hasn't been updated in a while and needs to checked that it includes all the issues tagged i18n-migrate.

Add a few i18n issues not already listed here.

Fix typo and add new experimental module in top priority section.

Formatting

Noting that an issue to migrate CCK field labels and options needs to be added. I can't do that now, maybe tomorrow.

Add issue for D6 CCK fields

Add D7 i18n Variable issues

I updated the issues summary based on the discussion last Thu in migrate mainatainers' weekly and following discussion in #7 - #8 in #2905736: Mark Migrate Drupal as stable.

I added a new subheading for those migrations that are about migrating multilingual content to separate them from migrations where site builder csn easily do the i18n setup manually.

Feel free to propose promoting any issue to this category, I just moved the most obvious ones.

Markus

p.s. Thumbs up @quietone! I noticed you've been very active lately in these i18n issues!

Moved user entity translations to the high priority (content) list.

Adding new issue to the taxonomy section.

I made a minimal change to issue summary. I believe it will refresh the colors of the issues listed in the IS. Sorry for the email notifications that this triggers.

Re-titling to bring in the concept of entity translation in addition to just i18n.

Bumping to major.

I cleaned up the issue summary, the list items were duplicating the issue titles that are automatically rendered.

Bumped #2859297: Migrate taxonomy term references for D6 Node translations to the high priority list as it is content migration and is blocking Migrate Drupal stability according to the current definition.

Markus

Added #2979964: Migrate translations for D7 i18n taxonomy 'localized' terms which is the same thing as #2886609: Migrate translations for D6 i18n taxonomy 'localized' terms but for Drupal 7.

Added #2979966: Migrate D7 i18n taxonomy term language

Added #2979970: Migrate D7 vocabulary language settings

Entity translation issue was split, added the follow-ups:
#2980996: Migrate Drupal 7 taxonomy term entity translations data to Drupal 8
#2669984: Migrate Drupal 7 user entity translations data to Drupal 8
#2981000: Migrate Drupal 7 comment entity translations data to Drupal 8

Woops, listed one of the issues twice.

Added the two new comment related issues + organized the lists in the IS a bit for clarity.

Added #2985716: Cannot save language negotiation settings after upgrade which is critical as it corrupts data

Just a bunk comment to update the issue statuses in the issue summary to their proper current values.

Pseudo update to refresh the status colors.

Refreshing the issue colors so that I can write a summary of the current situation.

Added remaining child issues to the issue summary.

The upgrade path for multilingual Drupal 6 to Drupal 8 is quite good. The biggest single remaining bundle is related to multilingual taxonomies / vocabularies. We have also node revision translations left.

For multilingual Drupal 7 to Drupal 8, the migrations for the Entity Translation concept is done except one remaining issue! The multilingual taxonomies / vocabularies (done without Entity Translation) and node revision translations is also remaining.

When these land, I would be ready to start considering also the multilingual upgrade paths stable i.e. the other issues that we have left should not block stability. If somebody has other views, please comment here!

Thanks @maxocub and @quietone once more for the productive mini-sprint this week and keep up the great work!

Cheers,
Markus

Retitle to be more appropriate now that we have its own module.

I was just comparing the d6 i18n issues with what exists for D7 to see what D7 migrations still need to be created . This is what I have as a working list. This does not mean that these are all the ones needed for D7, just the ones the would bring in the same type of data as the D6 ones. Still a lot to do.

Block
block_translation

Fields
cck fields label and description
cck field option translations

Menu
menu_links_translation

Taxonomy
taxonomy_vocabulary_translation
taxonomy_term_translation
i18n taxonomy localized terms
vocabulary language settings
"per" language multilingual vocabularies and terms

User
user_profile_field_instance_translation
user_settings_translation
user profile field value option translations

Adding new issues to the IS and put the ones still to do at the top of each section. Still needs more cleanup but I want to understand the priorities first.

Taxonomy and User migrations still need issues.

Updating the list of migrations still needing issues.

Taxonomy
taxonomy_vocabulary_translation
taxonomy_term_translation
vocabulary language settings
"per" language multilingual vocabularies and terms

User
user_profile_field_instance_translation
user profile field value option translations

Just rearranging the issues to make it easier for me to find things.

IS update - removed completed issues from the list and moved all the d6 to high priority. Much easier to read now.

Removed two other landed just now if we are removing things now :)

Two subheadings were not accurate anymore, removed.

Updated IS to put blocked issues below the issue blocking it and indented.

@masipila, you recently changed the parent of #2746517: Automatically derive auxiliary migrations to this one but this has changed to focus on stability of the multilingual. And I don't think that one is required for stability. Shall we just 'un parent' it and rely on the tags?

#2886609: Migrate translations for D6 i18n taxonomy 'localized' terms landed.

#2979970: Migrate D7 vocabulary language settings is not postponed anymore.

And one of taxonomy one was committed too! #2886609: Migrate translations for D6 i18n taxonomy 'localized' terms

Remove fixed issue from the IS

Add D7 language content comment migration
Remove fixed D6 user profile issue

Discussed in the migration maintainers call in slack this week this issue. TLDR; the sub-system maintainers are going to review all the child issues for multilingual for BC break potential. After we've done that and ONLY IF we don't find any potential, we'll mark this issue here for framework and release manager review. Review for marking multilingual as stable.

The longer version...
Due to our cautious nature in Drupal, we want to make sure we don't cause *any* BC breaks to folks. We take great pride in limiting the BC breaks in Migrate for a long time. Even before it was officially stable. This is a slow process though and multi-lingual migrations are complicated and only serves about 20% of the community. And of that 20%, there are multiple ways to implement multilingual sites in D6 and D7, all adding to the complexity. So we've traditionally stated we'd like to finish everything related to multi-lingual before we mark all of it stable. This is contrary to what we have done in the non-multilingual side of Migrate Drupal. On that side, we got to a point where we don't think we'll have any more BC breaks and even though there are few small holes here on the upgrade path... we felt it was far better to mark the whole thing stable and give folks the opportunity to upgrade to D8, then to wait a really long time until all upgrade paths are filled.

So, we're proposing to do the same thing now for multilingual. Review the child issues. Look for potential BC breaks. We know that name changes are the bane of BC. Keep that and other things in mind when reviewing. If we do not find any potential BC breaks, then let's move on and mark this thing stable.

Reviewed #3008028-24: Migrate D7 i18n menu links for BC breaks and found none. Adjusting the IS.

Hi!

I honestly don't know how we could declare stability without these three issues being solved:

#2979964: Migrate translations for D7 i18n taxonomy 'localized' terms
#2859297: Migrate taxonomy term references for D6 Node translations
#2979966: Migrate D7 i18n taxonomy term language

Without these, the upgrade path for multilingual sites that use taxonomy terms is more or less totally incomplete. The only workaround would be to go manually through all the nodes that used to have terms and manually add those terms back (after manually fixing the language to all those terms). This can be thousands of nodes and humanly impossible task.

Markus

Add issue for D7 synchronized fields

Add issue

Just moving the D6 issues to the top of this list in the IS

Format fix for IS.

Updating issue color coding

Added #3035392: Migrate vocabulary translations and taxonomy term references for Drupal 7 node translations

Removing this issue from the issue summary todo list as it landed. #3008030: Migrate D7 i18n fields label and description

Just finished going through all the high priority issues. There is plenty to review.

Refreshing the status colors in the issue summary

Retitled for Must/Should/Could/Nice to have.

Further clarified the issue summary and removed 2 done issues.

#3024460: Migrate D7 comment type language settings just landed, removing from IS

According to #2608062: [META] Requirements for tagging Drupal 9.0.0-alpha1, this is a "must have" for opening the Drupal 9 branch, which must be done by the end of 2020, so escalating to critical and tagging it with "Drupal 9."

Fixing a mistake

Remove duplicate issue from IS

Per @catch, one of the core release managers, we should try and focus here on migration paths for multilingual functionality provided by core (e.g. node revisions), since that's core violating its own standard to provide upgrade paths for all of its own data, vs. other things such as taxonomy terms that needed a contrib module (i18n) to provide multilingual functionality... those are still critical from the standpoint of calling the migration path "done," but the core-provided stuff specifically blocks the opening of the D9 branch, so... "hyper-critical." ;D

I split the issue summary up into 'must have -> core migrations' and 'must have -> contrib to core migrations'. Also bumped those two obvious core to core issues to critical.

Add #3073050: Migrate D7 i18n taxonomy term field translations to the IS.

Refreshing colors

#2979970: Migrate D7 vocabulary language settings and #3008029: Migrate D7 i18n field option translations have landed, removing from IS.

And #2979964: Migrate translations for D7 i18n taxonomy 'localized' terms

#2979966: Migrate D7 i18n taxonomy term language just landed, removed from remaining tasks.

One more edit to refresh status colors as #3035392: Migrate vocabulary translations and taxonomy term references for Drupal 7 node translations is unpostponed.

Add issue for migrating revisions for d7 entity translation method.

I closed #2980551: Errors When Migrating Books From Multilingual Drupal 7 as the reported did not follow up on questions from almost a month ago (and repeated questions over a week ago). Since it was not otherwise reproducible, it does not seem like we can do anything about that.

Updating the IS because #3030937: Ensure language is not Null in translation source queries has been committed

Reparenting as this is a direct requirement of beta.

#3031727: Migrate D7 synchronized fields also just landed, removing from issue summary.

#3035392: Migrate vocabulary translations and taxonomy term references for Drupal 7 node translations landed, removed from issue summary

Added issues that were found when working on the node revision translation and that need to be committed first.

Great both Upgrade6 and Upgrade7 are passing tests.

Testers: Sorry, I found a problem late last night so please don't test with the latest patch. I will post a new patch within the hour.

Add instructions for drush

Sorry for the noise; adding an ID attribute to the "Testing" heading for easier link-age.

Remove #3082719: migrate_drupal_migration_plugins_alter() should only alter definitions that exist from the IS since it was committed.

Remove #3073050: Migrate D7 i18n taxonomy term field translations and #3086238: getHighestId() should not fail when there is a destination id with type string from the IS because they were committed.

Update the IS with the Drupal 8 version to install.

Remove committed issue from the IS

Add more testing info to IS

For those following along here: #2746541: Migrate D6 and D7 node revision translations to D8 is now RTBC 🥳

Add menu translation issue

Moving entity_translation issue to could have based on discussion in #2966856-46: Hide and disable Drupal Migrate Multilingual.

Re-titling. We have a stable multingual migration path for core, but we still have migrations to add for functionality that required contrib modules in Drupal 6/7 and has been moved to core.

Update issues in IS.

this is blocking #3010983: Deprecate Drupal 6 and Drupal 7 migrations and move to contrib

We'll still want to backport these to 8.9 patch releases if possible I think, so switching the version back.

#2746517: Automatically derive auxiliary migrations has been closed(outdated) so removing from the IS.

All must haves are completed.

#3110669: Migrate d7 menu language content settings committed!

#3112249: Migrate d7 menu translation committed.

Not a word: "migation". ;)

Just realised #3008028: Migrate D7 i18n menu links and #3030939: D7 user profile translation migration are the last open issues here, both should haves, one RTBC - I think we can close this? Going to go ahead and do that (adding some triage credit in the process). Please re-open if I've missed something.

🥳

Wim Leers pointed out in #3175729: Mark i18n migrations as finished that there are todos in core referring to this issue. Those todo can't be done until all the i18n issues are committed and they are not. We are waiting on, at least, #3008028: Migrate D7 i18n menu links. So, let's leave this open until all the issues are done.

Thanks for the coordination here, @quietone!

#3008028: Migrate D7 i18n menu links is committed at least in Drupal 9.1. That would mean this can be continued?

I've checked the migration of user profile translation for D7 and am pleased so say that it was already done, there is nothing to do there. Therefor, I have closed #3030939: D7 user profile translation migration.

That means that all the multilingual migrations have been committed to HEAD, 9.1.x.

Wow! Did I really say that?

The remaining work is to backport #3110669: Migrate d7 menu language content settings and #3112249: Migrate d7 menu translation. The first one needs to be committed first as it has fixture changes needed in the second. There is no reason to wait on these issue to be backported, or not, as the case may be. We can continue on with the followup work.

Thanks to Wim Leers, the follow up to this, which is to review the files in migrates/state and remove @todos, already exists with a patch, #3175729: Mark i18n migrations as finished.

A very big THANK YOU to the many, many, many people, too many to name, who helped get this job done. And a special thank you to Gábor Hojtsy who was patient when I was struggling to figure out how to use translations in Drupal 6 and Drupal 7.

Time to Celebrate!!

Oops, seems I missed, #3008028: Migrate D7 i18n menu links, despite it being mentioned in #177. I will check those issues and update them as needed tonight.