SourceValueItem field synchronized translations

Feedbacks

that stores language-specific payloads as JSON, keyed by a stable identifier.

Why not using a serialized blob like we already do?

      'columns' => [
        ...
        'source' => [
          'type' => 'blob',
          'size' => 'big',
          'serialize' => TRUE,
        ],
      ],

Introduce a new sibling key source_uuid alongside existing source_id in the canonical tree:

We have it already _node_id in Display Builder, so let's harmonize it:

    $data = [
      '_node_id' => \uniqid(),
      'source_id' => $source_id,
      'source' => $data,
    ];

https://git.drupalcode.org/project/display_builder/-/blob/1.0.x/src/Enti...

We picked "Node ID" to fit with the expected vocabulary in data tree structure.

We picked uniqid() instead of UUID because it stays in must be unique only in the scope of the stored tree, and it is not shared between environment.

We added an underscore at the start of the key because it was something custom from Display Builder, if it became official at the UIP2 level, we can drop it.

Which values are translatable is decided by the configuration schema.

Exactly, configuration schema of each source plugin.

Notes

Talking with Christian:

• Config translations already work well in synchronous (expected the usual annoyance of display translation in Druapl core) because config translation are by design always synchronous
• However, config translations have an issue: they identify each translatable string by the path (like a JSON path), so it is messed up when we move a source around. Let's not fix this config translation issue, but let's adopt a better mechanism for content translation: the "Node ID".
• No need to do a MergeDeep of the translated tree to the original langue tree, because of this new mechanism.

Add a second storage column (translations)

Just asking: why do we need an other storage column dedicated to translations?

Both columns are never used at the same time:

And it is odd to have a field item dedicated to translations when there is already a translation mechanism at the Field API level.

Would it be better to keep 2 columns:

Then, the job is to take the sources of the translation, loop on the flat list and replace the source in the tree according to the node_id.

We will still face two challenges:

• source property is currently not accepting a list of source but a single source. What do we do? A special "translation bag" source? That sounds interesting if it eases the translatibility of config: #3548640: [trans] Sources should support configuration translation
• what happen if a source is moved in original language from a field item to another? Do we move the translation storage from other field item?

Also, discussed in #3584856: Source field model and storage, let's move the ui_patterns_field out of the experimental status in the scope of this issue

It is better to merge #3584856: Source field model and storage first because it will already ship some changes in SourceValueItem and will make this MR a bit simpler.

I will squash and review

I have rebased and squashed the linting commits.

The MR is adding 439 LOC (tests excluded) which is OK for such a change.

in ui_patterns_field:

• /src/Field/SourceValueList.php: where the magic happens, i guess
• /src/Plugin/Field/FieldFormatter/SourceFormatter.php : only cache update (and linting)
• /src/Plugin/Field/FieldType/SourceValueItem.php: let's merge #3584856: Source field model and storage first
• /src/Plugin/Field/FieldWidget/SourceComponentWidget.php : only linting
• /src/Plugin/Field/FieldWidget/SourceWidget.php: ComponentSlotForm::buildSourceForm() update

5 classes in newly added src/ComponentTree/ directory: looks not very drupally but I guess it may be necessary

• src/Element/ComponentFormBase.php: new parameter in ::getSelectedSource()
• src/Element/ComponentPropForm.php: add node_id in props. That's cool, but a big subject, let's check how it fits with #3574820: Unexpected properties in ComponentSource data and the plan for the new Source plugin system
• src/Element/ComponentSlotForm.php: new parameter in ::buildSourceForm()
• src/Plugin/UiPatterns/Source/FieldFormatterSource.php: just linting

Still reviewing.

(removing third_party_settings in all examples for clarity)

Thanks you Christian for this amazing work. It is impressive and already 90% OK in my opinion.

Symmetric translation

1. Just translating without messing with the source tree

1. I have 2 languages: English (default) and French.
2. I add a Source field with multiple values and symmetric translation (settings: { synchronized_translation: true }, am I right?) in article content type
3. I create a node in English, with 2 items in the source field:
   

   - source_id: component
     source:
       component:
         component_id: ui_suite_daisyui:button
         slots:
           label:
             sources:
             - source_id: textfield
               source:
                 value: 🏴󠁧󠁢󠁥󠁮󠁧󠁿🍺🍲 1
               node_id: 6a0f441718130
     node_id: 6a0f4417177f5
   
   - source_id: textfield
     source:
       value: 🏴󠁧󠁢󠁥󠁮󠁧󠁿🍺🍲 2
     node_id: 6a0f441718b50
   

4. I create the French translation with "🇫🇷🥐🍷 1" instead of "🏴󠁧󠁢󠁥󠁮󠁧󠁿🍺🍲 1" and "🇫🇷🥐🍷 2" instead of "🏴󠁧󠁢󠁥󠁮󠁧󠁿🍺🍲 2", without moving anything around for now

Results:

• I see English language in /node/1: ❌ KO, it is French
• I see French language in /fr/node/1: ✅ OK

Then:

1. I edit again /node/2/edit and I change back to English language

Results:

• I see English language in /node/1: ✅, OK it is now English
• I see French language in /fr/node/1: ✅ OK it still French

2. Moving stuff around in the translated language

I simply move the second field item before the first, only in the French translation:

Results:

• The textfield has moved at the first position in both languages: ✅ OK
• In English, we are still in English: ❌ KO, it is French
• in French, we are still in Frencht: ✅ OK it still French

Here is the current stored content:

The French strings (🇫🇷🥐🍷) are stored in the English translation.

3. Changing a non translatable value from original language

Before anything, I fix the previous translation issue by manually updating the strings in the English node. Then, I change the component variant in the English translation:

Results:

• The componet variant has changed in both languages: ✅ OK
• In English, we are still in English: ✅ OK
• in French, we are still in Frencht: ✅ OK it still French

Technical analysis

In the translated field, each field item as a /source/translations property:

- source_id: component
  source:
    translations:
      6a0f441718130:source.value: 🇫🇷🥐🍷 1
  node_id: 6a0f4417177f5

- source_id: textfield
  source:
    translations:
      6a0f441718b50:source.value: 🇫🇷🥐🍷 2
  node_id: 6a0f441718b50

This complexity may be the source of the errors because we are hiding a translation system inside a property data instead of working at the Field API level.

I know we have already talked about that, but is it possible to do this instead?

• All the translated nodes in a flat structure
• So we have a field item by translated node, the number of field item in a translation can sometimes be different from the one in the source language, this is not an issue for Drupal Field API
• Each node is using the same symmetric translation strategy as the Config API: just a skeleton with the translated values
• source and third_party_settings host the translations. source_id is missing because never translatable.

- source:
    value: 🇫🇷🥐🍷 1
  node_id: 6a0f441718130

- source:
    value: 🇫🇷🥐🍷 2
  node_id: 6a0f441718b50

And then, in FieldIitemInterface::getValue() and FieldIListitemInterface::getValue() we rebuild the full tree by merging the content of each translated node into the node positioned in the original language tree (because there are only the translated values to merge, no risk of overriding an unexpected value):

- source_id: component
  source:
    component:
      component_id: ui_suite_daisyui:button
      slots:
        label:
          sources:
          - source_id: textfield
            source:
              value: 🇫🇷🥐🍷 1
            node_id: 6a0f441718130
  node_id: 6a0f4417177f5

- source_id: textfield
  source:
    value: 🇫🇷🥐🍷 2
  node_id: 6a0f441718b50

Asymmetric translations

Same test with same data:

• I see English language in /node/2: ✅ OK
• I see French language in /fr/node/2: ✅ OK

I move stuff around in the translation:

• The textfield has moved at the first position in French: ✅ OK
• The component is still at the first position in English: OK
• In English, we are still in English: ✅ OK it still English
• in French, we are still in Frencht: ✅ OK it still French

I change a not translatable value in the original language:

• English has been updated: OK
• French is the same: ✅ OK
• In English, we are still in English: ✅ OK it still English
• in French, we are still in Frencht: ✅ OK it still French

Misc

To discuss:

• No node_id in asymmetric translations. I know there are not useful in this situation, but is it the opportunity of promote the node_id as a standard, expected, property for every stored source?
• Field settings UI: Is "Synchronized Translation" clearer than "Symmetric translation" ? Just asking... (and let's swicth to sentence case)
• SourceValueItem extends MapItem is wrong, MapDataDefinition::create('map') is confusing and not useful. Proposal: MapDataDefinition::create() and some empty sources are created at the root level, but I hope #3584856: Source field model and storage is already addressing all this
• We need documentation ;)

Also, to discuss with Jean (@mogtofu33) once the rest is OK: How src/ComponentTree/SourceTree.php could be related to\Drupal\display_builder\SourceTree?

Thanks. I will review.

Before starting the test, i want to share i really like your new idea. Content translation is asymmetric by nature, that means:

• each translation can have a totally different field data (ex: a translated Tags field can have totally different tags than the default language, not necessary the same tags but translated)
• and also, as a consequence, the translation storage is "complete". It can be loaded and used as it is (display, checks...), without being merged to the default language data

By adding symmetric translation mechanism, we alter some (and only some) load & save of the of the content, but we don't mess with the expected storage.

It can be surprising at first to store the full identical source tree in all translations when doing symmetric translations, but it starts to make sense to me.

I hope we will adopt the same mindset to fix the Display Builder part: #3555110: Symmetric translation where we will store in config in the way translated config is stored and managed, and only do alteration in our own load/publish proxy layer.

Setup

I have 2 languages: English (default) and French. I add a Source field with multiple values and symmetric translation (settings: { synchronized_translation: true }) in article content type.

Feedback about the field settings admin pages:

• We need to use sentence case: "Synchronized translation"
• It is currently visible even when the field is not translatable
• When the field is translatable, what is happening if "Synchronized Translation" is checked when "Users may translate this field" when is not checked?

Like last time, I create a node in English, with 2 items in the source field:

- source_id: component
  source:
    component:
      component_id: ui_suite_daisyui:button
      slots:
        label:
          sources:
          - source_id: textfield
            source:
              value: 🏴󠁧󠁢󠁥󠁮󠁧󠁿🍺🍲 1
            node_id: 6a0f441718130
  node_id: 6a0f4417177f5

- source_id: textfield
  source:
    value: 🏴󠁧󠁢󠁥󠁮󠁧󠁿🍺🍲 2
  node_id: 6a0f441718b50

(removing third_party_settings in all examples for clarity)

Symmetric translation

1. Just translating without messing with the source tree

I create the French translation with "🇫🇷🥐🍷 1" instead of "🏴󠁧󠁢󠁥󠁮󠁧󠁿🍺🍲 1" and "🇫🇷🥐🍷 2" instead of "🏴󠁧󠁢󠁥󠁮󠁧󠁿🍺🍲 2", without moving anything around for now

Results:

• I see English language in /node/1: ✅ OK
• I see French language in /fr/node/1: ✅ OK

Then I edit again default language /node/2/edit and I save without changing anything
Results:

• I see English language in /node/1: ✅, OK it is still English
• I see French language in /fr/node/1: ✅ OK it still French

2. Moving stuff around in the translated language

I simply move the second field item before the first, only in the French translation:

Results:

• The textfield has moved at the first position in both languages: ✅ OK
• In English, we are still in English: ✅ OK
• in French, we are still in Frencht: ✅ OK

3. Changing a non translatable value from original language

I change the component variant in the English version.

Results:

• The component variant has changed in both languages: ❌ KO, it changed only in English
• In English, we are still in English: ✅ OK
• in French, we are still in Frencht: ✅ OK

Asymmetric translations

Not tested again. I believe it is still OK :)

Follow-up

Also, to discuss with Jean (@mogtofu33) once the rest is OK: How src/ComponentTree/SourceTree.php could be related to\Drupal\display_builder\SourceTree?

As asked by Christian, we will do a follow-up issue for this task.

Only one little KO :)

Hi Christian,

a few things to change before sending to review for the last time.

1. SourceValueItem::mainPropertyName() must not return NULL

It breaks the entity queries (DatabaseExceptionWrapper). We need to pick one of the 4 properties: source_id, source, node_id or third_party_settings. I don't know which one.

--- a/modules/ui_patterns_field/src/Plugin/Field/FieldType/SourceValueItem.php
+++ b/modules/ui_patterns_field/src/Plugin/Field/FieldType/SourceValueItem.php
@@ -33,7 +33,7 @@ class SourceValueItem extends FieldItemBase {
    */
   public static function mainPropertyName() {
-     // A source item has no main property.
-    return NULL;
+    return 'source';
   }

2. The pipeline is red

cspell:

• andere
• artefacts
• deutscher
• erste
• fassung
• sprache
• unsynchronized
• zweite

phpunit: Twice Drupal\Tests\ui_patterns\Kernel\SchemaManagerTest::testResolve(), I don't know if it is related to the MR:

-    1 => 'ui-patterns://enum-list'
+    1 => 'ui-patterns://enum_list'
-    3 => 'ui-patterns://enum-set'
+    3 => 'ui-patterns://enum_set'

3. Module status

We decided a few weeks ago this ticket was the opportunity to move the module out of experimental status, didn't we?

I will do the 3 changes

Pipeline is green. Ready to be reviewed.

Careful: in the last commit, there is a a change in phpunit fixtures which is not related to the MR, but to a Drupal Core dependency upgrade:

Upgrading justinrainbow/json-schema (6.8.2 => 6.9.0)

You may encounter the same issue in other MR. I don't know if this change is "safe" or not.

Back to Christian just to rename /src/ComponentTree/* to /src/SourceTree/*

i will rebase, squash, rename ComponentTree and send to review

Done. Excluding the tests, the MR is only adding 361 lines of PHP code.