Problem/Motivation

On a global website, you might end up with a single language (like english) with multiple language records for regional variations (like english england, english scotland, english ireland).

So the same site might show a global region selection (england, scotland, ireland) and then allow to pick a local language (english) resulting in 3 different english variations. The user might not need to understand all this, but the admins / publishers need to be able to deal with these variations.

Because a translator might not support all these variations, we might end up ordering translations for a generic language "english".
This is not possible currently, as the reverse mapping is required to be a 1:1 mapping started with the remote (translator) language key.

That's why validation is requested to prohibit this setting:
#2256959: Validate language mapping selection

Hint: Setting to minor as it would lead to major refactoring of our language mapping functions... This is for a later major release.

Proposed resolution

Don't use translation target languages in early phases. We should deal with local language keys as much as possible.
The local language list can then be used to check availability on a certain translator and thus be filtered.
Reverse mapping should also almost never be used.
For instance if a job returns for review, the job defines the local language to process.

Remaining tasks

Identify APIs that are affected
Define better APIs

Define, if we support the new and the old way in parallel for some time. Otherwise it will break everything.

User interface changes

API changes

CommentFileSizeAuthor
#61 allow_multiple_local-2257033-61.patch14.18 KBedurenye
#57 interdiff-allow_multiple_local-2257033-52-57.txt1.75 KBedurenye
#57 allow_multiple_local-2257033-57.patch13.62 KBedurenye
#52 interdiff-allow_multiple_local-2257033-49-52.txt1.88 KBedurenye
#52 allow_multiple_local-2257033-52.patch15.37 KBedurenye
#49 interdiff-allow_multiple_local-2257033-46-49.txt5.42 KBedurenye
#49 allow_multiple_local-2257033-49.patch15.4 KBedurenye
#46 interdiff-allow_multiple_local-2257033-42-46.txt3.46 KBedurenye
#46 allow_multiple_local-2257033-46.patch12.7 KBedurenye
#42 interdiff-allow_multiple_local-2257033-40-42.txt1.66 KBedurenye
#42 allow_multiple_local-2257033-42.patch11.44 KBedurenye
#40 interdiff-allow_multiple_local-2257033-35-40.txt6.14 KBedurenye
#40 allow_multiple_local-2257033-40.patch10.68 KBedurenye
#35 interdiff-allow_multiple_local-2257033-31-35.txt1.24 KBedurenye
#35 allow_multiple_local-2257033-35.patch10.03 KBedurenye
#31 interdiff-allow_multiple_local-2257033-22-31.txt1.24 KBedurenye
#31 allow_multiple_local-2257033-31.patch10.06 KBedurenye
#22 interdiff-allow_multiple_local-2257033-20-22.txt3.07 KBedurenye
#22 allow_multiple_local-2257033-22.patch9.91 KBedurenye
#20 interdiff-allow_multiple_local-2257033-18-20.txt3.31 KBedurenye
#20 allow_multiple_local-2257033-20.patch9.14 KBedurenye
#18 interdiff-allow_multiple_local-2257033-13-18.txt3.05 KBedurenye
#18 allow_multiple_local-2257033-18.patch5.83 KBedurenye
#13 interdiff-allow_multiple_local-2257033-10-13.txt1.14 KBedurenye
#13 allow_multiple_local-2257033-13.patch4.54 KBedurenye
#10 interdiff-allow_multiple_local-2257033-6-10.txt1.16 KBedurenye
#10 allow_multiple_local-2257033-10.patch4.32 KBedurenye
#6 allow_multiple_local-2257033-6.patch5.47 KBedurenye
#6 allow_multiple_local-2257033-6-test_only.patch3.1 KBedurenye

Comments

miro_dietiker’s picture

Version: 7.x-1.x-dev » 8.x-1.x-dev
Assigned: Unassigned » miro_dietiker

Pushing this to 8.x-1.x.
This has been discussed in the TMGMT team and i'm working on this.

When this is completed, the mapping can happen in the base translator. The translator implementation will have a much easier life.

miro_dietiker’s picture

miro_dietiker’s picture

Priority: Minor » Major

Not so minor. Because we discussed the need of a significant API change that also makes the whole relationships more simple and implementations of translators easier since they don't need to take care of all the language mappings explicitly.

We can't do this in 7.x because APIs should be stable. But in 8.x we want to do it right this time.

miro_dietiker’s picture

miro_dietiker’s picture

The refactoring issue mentioned in #24 also dropped the mapToLocalLanguage and all translators are updated.
With this step, the origin of the problem is solved.

What is missing, is test coverage that translates two jobs into related target languages that end up with the same language code on the translator.

edurenye’s picture

Status: Active » Needs review
StatusFileSize
new3.1 KB
new5.47 KB

Added a test for this, as the test failed I created a patch that solves this issue.
I'm still not sure that my fix is the correct option, as I think this mappings should be added automaticaly, without the need of adding it in our custom save, so maybe there's still an error in my schema fix. But I'm not sure if I'm right with it or which should be the correct solution.

The last submitted patch, 6: allow_multiple_local-2257033-6-test_only.patch, failed testing.

The last submitted patch, 6: allow_multiple_local-2257033-6-test_only.patch, failed testing.

juanse254’s picture

Status: Needs review » Needs work

There are some leftovers there ;)

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new4.32 KB
new1.16 KB

Yes, sorry. I was using that to save time, I forgot to delete.

juanse254’s picture

Status: Needs review » Reviewed & tested by the community

Working for me, seems good.

miro_dietiker’s picture

Status: Reviewed & tested by the community » Needs work

Awesome.

+++ b/src/Tests/TranslatorTest.php
@@ -125,4 +125,39 @@ class TranslatorTest extends TMGMTTestBase {
+    $this->assertFieldByXPath('//select[@id="edit-translator"]//option[@value="test_translator"]', t('Test translator (auto created)'), 'Translator maps correctly');
...
+    $this->assertFieldByXPath('//select[@id="edit-translator"]//option[@value="test_translator"]', t('Test translator (auto created)'), 'Translator maps correctly');

Still this is not enough. We still need to submit the job and receive the translation saved back to the source. Then we need to check the written source to make sure it was updated with the correct target language. In past, it happened that if you order pt-br (ending in pt remote) then it was written as pt-pt because of the reverse mapping. This no more happens after the refactoring at #2538198: Wire getSource/RemoteLanguage via Job, but a test should proof it.

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new4.54 KB
new1.14 KB

Done.

Status: Needs review » Needs work

The last submitted patch, 13: allow_multiple_local-2257033-13.patch, failed testing.

The last submitted patch, 13: allow_multiple_local-2257033-13.patch, failed testing.

edurenye’s picture

Status: Needs work » Needs review

Unrelated failing test.

miro_dietiker’s picture

Status: Needs review » Needs work

You still only test the job submission.
You really need to load the source item that you added to the job and check if it has the translation attached after the job came back and was accepted (in the right language).

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new5.83 KB
new3.05 KB

I found an error in the translator test, it was using the local langcode to translate, and it should use the remote langcode, so I fixed.
Now I'm checking that the item is accepted, it contains the translation and the translated language is the correct.

Status: Needs review » Needs work

The last submitted patch, 18: allow_multiple_local-2257033-18.patch, failed testing.

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new9.14 KB
new3.31 KB

Fixed failing tests.

Status: Needs review » Needs work

The last submitted patch, 20: allow_multiple_local-2257033-20.patch, failed testing.

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new9.91 KB
new3.07 KB

Missed some tests before.

Status: Needs review » Needs work

The last submitted patch, 22: allow_multiple_local-2257033-22.patch, failed testing.

edurenye’s picture

I don't know why the test fails here and not in local.

edurenye’s picture

Status: Needs work » Needs review

The last submitted patch, 18: allow_multiple_local-2257033-18.patch, failed testing.

The last submitted patch, 20: allow_multiple_local-2257033-20.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 22: allow_multiple_local-2257033-22.patch, failed testing.

The last submitted patch, 22: allow_multiple_local-2257033-22.patch, failed testing.

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new10.06 KB
new1.24 KB

Maybe for some strange reason, in the testbot the id is diferent or somthing, so I improved the test, to have dynamic values.

Status: Needs review » Needs work

The last submitted patch, 31: allow_multiple_local-2257033-31.patch, failed testing.

The last submitted patch, 31: allow_multiple_local-2257033-31.patch, failed testing.

mbovan’s picture

+++ b/src/Tests/TranslatorTest.php
@@ -151,10 +151,15 @@ class TranslatorTest extends TMGMTTestBase {
+    $this->drupalGet('http://d8.dev/admin/tmgmt/items/' . 1);

@@ -162,6 +167,11 @@ class TranslatorTest extends TMGMTTestBase {
+    $this->drupalGet('http://d8.dev/admin/tmgmt/items/' . 2);

Maybe this is a strange reason. :P

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new10.03 KB
new1.24 KB

yes, lol

juanse254’s picture

Status: Needs review » Reviewed & tested by the community

Everything works as expected and seems fine to me.

miro_dietiker’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/sources/content/src/Tests/ContentEntitySourceUiTest.php
@@ -101,9 +101,9 @@ class ContentEntitySourceUiTest extends EntityTestBase {
-    $this->assertText('de_' . $node->body->value);
...
+    $this->assertText('de-ch_' . $node->body->value);

+++ b/tmgmt_test/src/Plugin/tmgmt/Translator/TestTranslator.php
@@ -86,7 +86,7 @@ class TestTranslator extends TranslatorPluginBase implements TranslatorRejectDat
-          $tdata[$key]['#text'] = $job->getTargetLangcode() . '_' . $value['#text'];
+          $tdata[$key]['#text'] = $job->getRemoteTargetLanguage() . '_' . $value['#text'];

Why did you suddenly decide to switch the language code in the translator? This is possibly troublesome.

Let's change it the way so that we have all target codes visible:
Example: "ch_de(de): The original text..."
The value in brackets is the reffective remote target language of the translator.
Let's have the TestTranslator add the brackets in case the mapping makes the remote code different to the local target code.

edurenye’s picture

I did because represents that the translator (microsoft, google, gengo...) doesn't know the local language, so has no sense that it shows like it was translated to that language.

miro_dietiker’s picture

I understood that intention and that's why it makes sense.
Note though that the plugin is the connector of both worlds, just with a very well prepared environment for passing things to the remote side. But it is still in between.

And that's why i think it should also state that in its response in test.

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new10.68 KB
new6.14 KB

ok, done.

Status: Needs review » Needs work

The last submitted patch, 40: allow_multiple_local-2257033-40.patch, failed testing.

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new11.44 KB
new1.66 KB

Fixed the failing test.

The last submitted patch, 40: allow_multiple_local-2257033-40.patch, failed testing.

berdir’s picture

Status: Needs review » Needs work
  1. +++ b/config/schema/tmgmt.schema.yml
    @@ -39,11 +39,6 @@ tmgmt.translator.*:
         settings:
           type: tmgmt.translator.settings.[%parent.plugin]
    -    remote_languages_mappings:
    -      type: sequence
    -      sequence:
    -        type: string
    -        label: Language key
     
     views.access.tmgmt_job:
       type: mapping
    @@ -55,3 +50,8 @@ tmgmt.translator_base:
    
    @@ -55,3 +50,8 @@ tmgmt.translator_base:
         auto_accept:
           type: boolean
           label: Automatically accept
    +    remote_languages_mappings:
    +      type: sequence
    +      sequence:
    +        type: string
    +        label: Language key
    diff --git a/sources/content/src/Tests/ContentEntitySourceUiTest.php b/sources/content/src/Tests/ContentEntitySourceUiTest.php
    
    diff --git a/sources/content/src/Tests/ContentEntitySourceUiTest.php b/sources/content/src/Tests/ContentEntitySourceUiTest.php
    index b457e4c..001c7a2 100644
    

    I don't get this change.

    We explicitly moved the remote language mappings *out* of the settings. This moves them back in.

  2. +++ b/src/Form/TranslatorForm.php
    @@ -233,6 +233,7 @@ class TranslatorForm extends EntityForm {
         $entity = $this->entity;
    +    $entity->setSetting('remote_languages_mappings', $form_state->getValue('remote_languages_mappings'));
    

    Same here. It has to work without this.

    If it does't then we need to identify what code is still trying to access them as settings.

  3. +++ b/tmgmt_test/config/schema/tmgmt_test.schema.yml
    @@ -16,8 +16,3 @@ tmgmt.translator.settings.test_translator:
           type: string
           label: Another key
    -    remote_languages_mappings:
    -      type: sequence
    -      sequence:
    -        type: string
    -        label: Language key
    diff --git a/tmgmt_test/src/Plugin/tmgmt/Translator/TestTranslator.php b/tmgmt_test/src/Plugin/tmgmt/Translator/TestTranslator.php
    

    This however is correct and should be removed.

edurenye’s picture

I changed that, because was not working without the point 2. So to make the point to work I had to change the schema.
I don't know how to solve this otherwise.
Why should not be in settings? Isn't it a setting?

edurenye’s picture

I know this fails, it's just to shou it to @Berdir, I don't know why it fails.

miro_dietiker’s picture

+++ b/tmgmt_test/src/Plugin/tmgmt/Translator/TestTranslator.php
@@ -86,7 +86,7 @@ class TestTranslator extends TranslatorPluginBase implements TranslatorRejectDat
-          $tdata[$key]['#text'] = $job->getTargetLangcode() . '_' . $value['#text'];
+          $tdata[$key]['#text'] = $job->getTargetLangcode() . '(' . $job->getRemoteTargetLanguage() . '): ' . $value['#text'];

I don't like the term "de(de)". My proposal was to only output the mapping info if the languages differ. See my original quote in #37

Let's have the TestTranslator add the brackets in case the mapping makes the remote code different to the local target code.

miro_dietiker’s picture

If you look at some sensor config saved from translator creation, you will see that the remote_languages_mappings is not inside the settings key.
And it seems we have missed some cases to move this out of the setting...

In Translator.php

  public function mapToRemoteLanguage($language) {
...
    if ($mapping = $this->getSetting(['remote_languages_mappings', $language])) {

And in TranslatorTest.php

    $this->default_translator->setSetting(['remote_languages_mappings', 'de'], 'de-de');
    $this->default_translator->setSetting(['remote_languages_mappings', 'en'], 'en-uk');
    $this->default_translator->save();

Also tmgmt.translator.settings.test_translator should already have remote_languages_mappings from tmgmt.translator.*
So the schema seems to contain a duplicate definition.

Hint: The TranslatorInterface now has getRemoteLanguagesMappings, but it has no interface to change it.
The form values should be updated in EntityForm::copyFormValuesToEntity() through the entity setter. This should just work..
At least it looks to me like Translator.php is lacking config_export annotation for remote_languages_mappings.

So still a bunch to cleanup from the previous issue about remote_languages_mapping. Looking forward to have that clean finally!

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new15.4 KB
new5.42 KB

I think adding 'remote_languages_mappings' => [] everywhere is not the best solution, but it works, so I upload this to you to check the progres and maybe suggest a better solution.

Status: Needs review » Needs work

The last submitted patch, 49: allow_multiple_local-2257033-49.patch, failed testing.

The last submitted patch, 49: allow_multiple_local-2257033-49.patch, failed testing.

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new15.37 KB
new1.88 KB

Fixed those failing tests.

miro_dietiker’s picture

Status: Needs review » Reviewed & tested by the community

I think this is how it should be.

The last submitted patch, 46: allow_multiple_local-2257033-46.patch, failed testing.

The last submitted patch, 46: allow_multiple_local-2257033-46.patch, failed testing.

berdir’s picture

Status: Reviewed & tested by the community » Needs work
  1. +++ b/src/Form/TranslatorForm.php
    @@ -232,14 +232,16 @@ class TranslatorForm extends EntityForm {
        * Overrides Drupal\Core\Entity\EntityForm::save().
        */
       public function save(array $form, FormStateInterface $form_state) {
    +    //$this->entity->set('remote_languages_mappings', $form_state->getValue('remote_languages_mappings'));
         $entity = $this->entity;
         $status = $entity->save();
    +    //$status = parent::save($form, $form_state);
    

    This should be removed.

  2. +++ b/src/Form/TranslatorForm.php
    @@ -232,14 +232,16 @@ class TranslatorForm extends EntityForm {
    -      drupal_set_message(format_string('%label configuration has been updated.', array('%label' => $entity->label())));
    +      drupal_set_message(format_string('%label configuration has been updated.', array('%label' => $this->entity->label())));
         }
         else {
    -      drupal_set_message(format_string('%label configuration has been created.', array('%label' => $entity->label())));
    +      drupal_set_message(format_string('%label configuration has been created.', array('%label' => $this->entity->label())));
    

    unnecessary change?

  3. +++ b/src/Tests/TMGMTTestBase.php
    @@ -158,6 +158,7 @@ abstract class TMGMTTestBase extends WebTestBase {
           'label' => $this->randomMachineName(),
           'plugin' => 'test_translator',
    +      'remote_languages_mappings' => [],
    

    This should *not* be necessary. If it's not then it means that the property on Translator is not defined. Define it there as an empty array instead.

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new13.62 KB
new1.75 KB

As I said, that was a patch work in progres, so I had some things that I tried commented just to get feedback about what was better.
Now should be everything ok.

Status: Needs review » Needs work

The last submitted patch, 57: allow_multiple_local-2257033-57.patch, failed testing.

The last submitted patch, 57: allow_multiple_local-2257033-57.patch, failed testing.

miro_dietiker’s picture

Hmm, some mappings now seem to be null:
"Schema errors for tmgmt.translator.tw9mzwfb with the following errors: tmgmt.translator.tw9mzwfb:remote_languages_mappings variable type is NULL"

edurenye’s picture

Status: Needs work » Needs review
StatusFileSize
new14.18 KB

Rebased

+++ b/src/Tests/TMGMTTestBase.php
@@ -158,7 +158,6 @@ abstract class TMGMTTestBase extends WebTestBase {
       'plugin' => 'test_translator',
-      'remote_languages_mappings' => [],
       'settings' => empty($values['plugin']) ? [

Added again this line, is defined but still don't work without this.

miro_dietiker’s picture

Status: Needs review » Needs work

Committing this intermediate to make the mapping work as expected.

Keeping open to discuss how we resolve the empty mapping more clean as a second step in this issue. Could also be moved into a followup.

  • miro_dietiker committed 2b9d3d0 on 8.x-1.x authored by edurenye
    Issue #2257033 by edurenye, miro_dietiker: Allow multiple local language...
edurenye’s picture

Status: Needs work » Fixed

We discussed and there is an issue where we will try to fix it #2654958: Default remote language mappings are not recreated
So I'm closing it.

edurenye’s picture

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.