Problem/Motivation
On a global website, you might end up with a single language (like english) with multiple language records for regional variations (like english england, english scotland, english ireland).
So the same site might show a global region selection (england, scotland, ireland) and then allow to pick a local language (english) resulting in 3 different english variations. The user might not need to understand all this, but the admins / publishers need to be able to deal with these variations.
Because a translator might not support all these variations, we might end up ordering translations for a generic language "english".
This is not possible currently, as the reverse mapping is required to be a 1:1 mapping started with the remote (translator) language key.
That's why validation is requested to prohibit this setting:
#2256959: Validate language mapping selection
Hint: Setting to minor as it would lead to major refactoring of our language mapping functions... This is for a later major release.
Proposed resolution
Don't use translation target languages in early phases. We should deal with local language keys as much as possible.
The local language list can then be used to check availability on a certain translator and thus be filtered.
Reverse mapping should also almost never be used.
For instance if a job returns for review, the job defines the local language to process.
Remaining tasks
Identify APIs that are affected
Define better APIs
Define, if we support the new and the old way in parallel for some time. Otherwise it will break everything.
User interface changes
API changes
| Comment | File | Size | Author |
|---|---|---|---|
| #61 | allow_multiple_local-2257033-61.patch | 14.18 KB | edurenye |
| #57 | interdiff-allow_multiple_local-2257033-52-57.txt | 1.75 KB | edurenye |
| #57 | allow_multiple_local-2257033-57.patch | 13.62 KB | edurenye |
| #52 | interdiff-allow_multiple_local-2257033-49-52.txt | 1.88 KB | edurenye |
| #52 | allow_multiple_local-2257033-52.patch | 15.37 KB | edurenye |
Comments
Comment #1
miro_dietikerPushing this to 8.x-1.x.
This has been discussed in the TMGMT team and i'm working on this.
When this is completed, the mapping can happen in the base translator. The translator implementation will have a much easier life.
Comment #2
miro_dietikerThere are more related tasks.
Comment #3
miro_dietikerNot so minor. Because we discussed the need of a significant API change that also makes the whole relationships more simple and implementations of translators easier since they don't need to take care of all the language mappings explicitly.
We can't do this in 7.x because APIs should be stable. But in 8.x we want to do it right this time.
Comment #4
miro_dietikerStarted a refactoring issue: #2538198: Wire getSource/RemoteLanguage via Job
Comment #5
miro_dietikerThe refactoring issue mentioned in #24 also dropped the mapToLocalLanguage and all translators are updated.
With this step, the origin of the problem is solved.
What is missing, is test coverage that translates two jobs into related target languages that end up with the same language code on the translator.
Comment #6
edurenye commentedAdded a test for this, as the test failed I created a patch that solves this issue.
I'm still not sure that my fix is the correct option, as I think this mappings should be added automaticaly, without the need of adding it in our custom save, so maybe there's still an error in my schema fix. But I'm not sure if I'm right with it or which should be the correct solution.
Comment #9
juanse254 commentedThere are some leftovers there ;)
Comment #10
edurenye commentedYes, sorry. I was using that to save time, I forgot to delete.
Comment #11
juanse254 commentedWorking for me, seems good.
Comment #12
miro_dietikerAwesome.
Still this is not enough. We still need to submit the job and receive the translation saved back to the source. Then we need to check the written source to make sure it was updated with the correct target language. In past, it happened that if you order pt-br (ending in pt remote) then it was written as pt-pt because of the reverse mapping. This no more happens after the refactoring at #2538198: Wire getSource/RemoteLanguage via Job, but a test should proof it.
Comment #13
edurenye commentedDone.
Comment #16
edurenye commentedUnrelated failing test.
Comment #17
miro_dietikerYou still only test the job submission.
You really need to load the source item that you added to the job and check if it has the translation attached after the job came back and was accepted (in the right language).
Comment #18
edurenye commentedI found an error in the translator test, it was using the local langcode to translate, and it should use the remote langcode, so I fixed.
Now I'm checking that the item is accepted, it contains the translation and the translated language is the correct.
Comment #20
edurenye commentedFixed failing tests.
Comment #22
edurenye commentedMissed some tests before.
Comment #24
edurenye commentedI don't know why the test fails here and not in local.
Comment #25
edurenye commentedComment #31
edurenye commentedMaybe for some strange reason, in the testbot the id is diferent or somthing, so I improved the test, to have dynamic values.
Comment #34
mbovan commentedMaybe this is a strange reason. :P
Comment #35
edurenye commentedyes, lol
Comment #36
juanse254 commentedEverything works as expected and seems fine to me.
Comment #37
miro_dietikerWhy did you suddenly decide to switch the language code in the translator? This is possibly troublesome.
Let's change it the way so that we have all target codes visible:
Example: "ch_de(de): The original text..."
The value in brackets is the reffective remote target language of the translator.
Let's have the TestTranslator add the brackets in case the mapping makes the remote code different to the local target code.
Comment #38
edurenye commentedI did because represents that the translator (microsoft, google, gengo...) doesn't know the local language, so has no sense that it shows like it was translated to that language.
Comment #39
miro_dietikerI understood that intention and that's why it makes sense.
Note though that the plugin is the connector of both worlds, just with a very well prepared environment for passing things to the remote side. But it is still in between.
And that's why i think it should also state that in its response in test.
Comment #40
edurenye commentedok, done.
Comment #42
edurenye commentedFixed the failing test.
Comment #44
berdirI don't get this change.
We explicitly moved the remote language mappings *out* of the settings. This moves them back in.
Same here. It has to work without this.
If it does't then we need to identify what code is still trying to access them as settings.
This however is correct and should be removed.
Comment #45
edurenye commentedI changed that, because was not working without the point 2. So to make the point to work I had to change the schema.
I don't know how to solve this otherwise.
Why should not be in settings? Isn't it a setting?
Comment #46
edurenye commentedI know this fails, it's just to shou it to @Berdir, I don't know why it fails.
Comment #47
miro_dietikerI don't like the term "de(de)". My proposal was to only output the mapping info if the languages differ. See my original quote in #37
Comment #48
miro_dietikerIf you look at some sensor config saved from translator creation, you will see that the remote_languages_mappings is not inside the settings key.
And it seems we have missed some cases to move this out of the setting...
In Translator.php
And in TranslatorTest.php
Also tmgmt.translator.settings.test_translator should already have remote_languages_mappings from tmgmt.translator.*
So the schema seems to contain a duplicate definition.
Hint: The TranslatorInterface now has getRemoteLanguagesMappings, but it has no interface to change it.
The form values should be updated in EntityForm::copyFormValuesToEntity() through the entity setter. This should just work..
At least it looks to me like Translator.php is lacking config_export annotation for remote_languages_mappings.
So still a bunch to cleanup from the previous issue about remote_languages_mapping. Looking forward to have that clean finally!
Comment #49
edurenye commentedI think adding
'remote_languages_mappings' => []everywhere is not the best solution, but it works, so I upload this to you to check the progres and maybe suggest a better solution.Comment #52
edurenye commentedFixed those failing tests.
Comment #53
miro_dietikerI think this is how it should be.
Comment #56
berdirThis should be removed.
unnecessary change?
This should *not* be necessary. If it's not then it means that the property on Translator is not defined. Define it there as an empty array instead.
Comment #57
edurenye commentedAs I said, that was a patch work in progres, so I had some things that I tried commented just to get feedback about what was better.
Now should be everything ok.
Comment #60
miro_dietikerHmm, some mappings now seem to be null:
"Schema errors for tmgmt.translator.tw9mzwfb with the following errors: tmgmt.translator.tw9mzwfb:remote_languages_mappings variable type is NULL"
Comment #61
edurenye commentedRebased
Added again this line, is defined but still don't work without this.
Comment #62
miro_dietikerCommitting this intermediate to make the mapping work as expected.
Keeping open to discuss how we resolve the empty mapping more clean as a second step in this issue. Could also be moved into a followup.
Comment #64
edurenye commentedWe discussed and there is an issue where we will try to fix it #2654958: Default remote language mappings are not recreated
So I'm closing it.
Comment #65
edurenye commented