Problem/Motivation
CCK field options are translated using i18n_cck submodule of the suite i18n, we need to migrate those respecting the language.
Once an the option translation were working and rollbacks could be tested, the rollback on d6_field_option caused fatals. Turns out that EntityFieldStorageConfig does not handle translations. To fix this, EntityFieldStroageConfig::getIds() now adds the language code for translation destinations and EntityFieldStroageConfig::rollback() adds the language code to the destination id array. This array is now unusual in that the first key is 0 and the second key is langcode. Maybe this should be changed.
This then exposed an error in EntityConfigBase::rollback() in a loop that is building the destination identifier where the comparison of $key == 'langcode' always returns true (because $key is an integer) and the entity_id is not built correctly. Changing that comparision to strict fixes that and the rollback work.
Proposed resolution
Patch
Review
Commit
Remaining tasks
Write a test
Write a patch to migrate field labels and options.
| Comment | File | Size | Author |
|---|---|---|---|
| #44 | interdiff-38-44.txt | 1.13 KB | quietone |
| #44 | 2961114-44.patch | 43.26 KB | quietone |
| #38 | interdiff.txt | 2.48 KB | quietone |
| #38 | 2961114-38.patch | 43.25 KB | quietone |
| #31 | interdiff.txt | 45.75 KB | quietone |
Comments
Comment #3
quietone commentedMaking a start
Comment #4
quietone commentedHere is a start. The interesting thing I learned while finding out how this is done in Drupal 8 is that the boolean option is treated differently. The translation labels are stored in the field (field.field) where as for the others I tested put the translations in field storage (field.storage).
Comment #6
quietone commentedThe majority of the errors were caused by an error in drupal6.php. Just before I made the patch the IDE was hogging the CPU and being mostly unresponsive, and it seems it was then an extra line got into the fixture.
Also, updated the MigrateUpgrade tests and found that d6_field_option_translation.yml was using the wrong source plugin.
Comment #8
quietone commentedFix a typo and move i18ncck to the will be upgrade list in MigrateUpgrade6Test
Comment #10
quietone commentedAside from the error on the entity count rollback on d6_field_option_translation causes a fatal.
Comment #11
quietone commentedIn order for rollback to work changes are made to EntityFieldStroageConfig and EntityConfigBase. For EntityFieldStroageConfig this was to add the language to the destination identifier array, keyed by langcode, if this is a translation destination. The this exposed that in EntityConfigBase at test was being done ($key == 'langcode') where $key could be an integer. When $key is an integer the if is always true and the destination value is always empty, so a fatal eventually occurs when trying to get an entity with an id of NULL. That comparison is now strict.
Comment #13
quietone commentedTODO:
Translate another item in field_test_integer_selectlist to check that more that option with a translation migrates successfully.
Learn why node counter test fail.
Comment #14
quietone commentedComment #15
quietone commentedAdded another translation to field_test_integer_selectlist to make sure that multiple translations work properly.
Changed the option translations to have the prefix 'fr - ' or 'zu - ' like the other translations in the fixture. For unknown reasons, I didn't follow that pattern when adding these ones.
Removed changes to the test fixture that are not necessary, which include changes to the node counter table.
Comment #16
quietone commentedUpdate IS
Comment #17
quietone commentedAnd this is blocking #2845975: Migrate Drupal 6 user profile field value option translations.
Comment #18
maxocub commentedI was testing this manually and I stumble upon a couple exceptions. First, if you don't have the Profile module enabled on D6 you get this:
Then if you enable the Profile module but don't create any profile you get this:
I don't see why this migration requires Profile, maybe there's a reason I don't know. In any case, I think we should avoid those exceptions.
Otherwise, I haven't completed my review yet, but I can say that after enabling the Profile module and adding a profile, the migration seems to be working well.
Comment #19
quietone commentedYes, the profile stuff can be removed. It was left over from when I first setup the source plugin and didn't escape the underscore in '
->condition('property', 'option\_%', 'LIKE'). Without escaping the underscore the source plugin was returning profile field options as well as the CCK field options. When I learned about that '_' has special meaning I changed the LIKE statement but overlooked the code relating to profile. This patch fixes that.Comment #20
heddnDiscussed in the migrate maintainers meeting w/ @quietone and @phenaproxima and @heddn and none of think there is a BC break with the destination getIds because we don't have any existing field config destination migrations.
We should never use conditional logic in a getIds. Fix this in 9.0. Should add an @TODO and link to an issue.
Comment #21
quietone commentedAs part of doing #20 I decided to make a list of places where
($this->isTranslationDestination()) {is used is getIds() in migrate.core/modules/migrate/src/Plugin/migrate/destination/Config.php
core/modules/migrate/src/Plugin/migrate/destination/EntityConfigBase.php
core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php
core/modules/migrate/src/Plugin/migrate/destination/EntityFieldInstance.php
core/modules/migrate/src/Plugin/migrate/destination/EntityRevision.php
Comment #22
quietone commentedIssue made for point raised in #20 and added an @todo.
This needed a reroll as well as adding the @todo comment.
Comment #23
masipila commentedRe-parenting so that all multilingual migrations can be found from the same meta. Moved original parent to related issues.
Comment #24
quietone commentedRemove extra whitespace
Document what is expected in value.
Use double quotes
In the migrate meeting phenaproxima asked for more documentation on the source plugins as they are very similar. And to look at having the source plugins not be so specific, i.e. gather all data from the tables.
Comment #25
quietone commentedFixes for the items in #24, except for the source plugin.
While working on this again I realized that this is only testing list fields with an allowed value list with just a key and not any field with an allowed value list of key|label. So, this adds a new field to the source and updates the test accordingly. I haven't cleaned up the test fixture so that still need to be done.
TODO:
@heddn, @phenaproxima, can you remind me what needs to be done to the source plugin. Was the intention to have all the properties in the tables in the row or to remove/reduce the conditions?
Comment #27
quietone commentedThere was a failing test and this should fix it.
Comment #28
heddnAll properties.
Comment #29
quietone commentedThank you. This updated the source plugins and tests as needed.
Comment #30
quietone commentedStill need to remove unneeded changes to drupal6.php
Comment #31
quietone commentedRemove what I hope is unnecessary changes in the fixture.
Comment #32
heddnThis might just be a quick question and reply:
I assume this is a pretty standard list of mappings. And with our work on a base class for entity reference, node reference, user reference, instead of constantly copy/pasting this all over the place, would it make sense to move this in to a trait and do this in code? Or should this be a follow-up to clean these things up?
Comment #33
quietone commentedDefinitely a follow up, there is enough going on here.
Leaving at NW for the followup to be made.
Comment #34
heddnI feel woefully unqualified to RTBC this thing, but I cannot find anything else to comment about. Let's get that follow-up opened and on to RTBC then.
Comment #35
quietone commentedFollow up created, #3007780: Replace reference field mappings in yml with a trait
Comment #36
heddnNo more blockers.
Comment #37
gábor hojtsyThanks, great work. Some notes:
Field instance option translation
Hm, where do these come from? They both look to excessive and too limited if so specific :)
Field option translation
It is not global settings when its about an instance only when its about a field, no?
Is allowed values the only thing we need to translate / handle? Prefix/suffix, etc. are not options we would need to translate or is that already covered or covered by another issue?
What's an explanation here?
Comment #38
quietone commented37.1. Fixed
37.2. These source migration yml are based on d6_field.yml and that particular process is copied from d6_field.yml. I don't know their history, I just assume they are correct.
37.3 Fixed
37.4 This is really using the global settings from content_node_field. The translations for the booleans are in the field_config not in field_storage so a whole separate migration is needed.
37.5 Good question. I meant to follow that up!
37.6 Fixed.
Also fixed formatting in the yml files.
TODO:
37.5
Comment #39
quietone commented37.5 I brought up a D6 site using the test fixture and tested whether the prefix and suffix need to be translated and migrated. I found that the 'integer field' in the story content type has values in the prefix and suffix field. I refreshed the translation strings but then searched for these strings ('pref' and 'suf) and they are not found. To double check I went back and added prefix ("XYZZY') and suffix ("SUFFIXXYZZY') to the Decimal field, refreshed the translation strings and search for XYZZY and they were not found. So it seems there is no more work to do here.
Unless, of course, I was not doing the correct process to translate those values.
Comment #40
heddnFor #37.2, see #3008328: Create Field Plugins for Drupal 6 phone and number fields. where we plan to fix that up. I think that covers all the questions brought up since last RTBC, so let's try it again.
Comment #41
gábor hojtsyThanks.
#3008328: Create Field Plugins for Drupal 6 phone and number fields. does not explain why would phone numbers have such country/language(?) specific variants, is that a phone module thing and not a translation thing to support different phone number formats? I went ahead and checked that at https://cgit.drupalcode.org/phone/tree/phone.module?h=6.x-1.x#n28 and is apparently a funky implementation on Drupal 6 where country codes are represented with one field type each. Hum. Ok. You are doing the right thing here.
I looked at https://cgit.drupalcode.org/i18n/tree/i18ncck/i18ncck.module?h=6.x-1.x and it does indeed only do label/description translation for the fields (migrations for which I believe are already done earlier in Drupal 8). Even Drupal 7's i18n at https://cgit.drupalcode.org/i18n/tree/i18n_field/i18n_field.i18n.inc only extends that with supporting default value translation for text type fields. Again you are doing the right thing here. One minor thing left then:
I don't believe "configuration" is used as a migration suffix (for other field config related things or otherwise). Either both should have it or neither.
Comment #42
heddnTagging novice. Let's call them configuration in both cases. I checked and we have precedent on that in
d7_field_instance.ymlandd7_field.ymlComment #43
quietone commentedTagging needs work for the task in #41/#42 to improve the label in the yml files.
Comment #44
quietone commentedRight, let's get this done.
The labels are now "Field option configuration translation" and "Field instance option configuration translation" because they are configuration and translation is normally at the end of the label string for translation migrations.
Comment #47
gábor hojtsySuperb, thanks, landed!