Updated: Comment #2

Problem/Motivation

Currently the configuration translations are saved with prefix language.config.[langcode] which is causing issues with

  1. Writing configuration schemas

Proposed resolution

  1. Contextual override config lives in subdirectories of the config directory.
  2. The maximum nesting level is 1 → sub-sub-directories are not allowed/supported.
  3. For non-file-based configuration storage implementations, each subdirectory denotes a collection name (cf. KeyValueStore).
  4. Each subdirectory/collection specifies the context(s) to which it applies in HTTP query string syntax → ./langcode=de/system.site.yml
  5. Context namespaces (e.g. langcode) are not owned by any particular module/provider; they use common names → ./domain=example.com/, ./subdomain=support/, etc.
  6. Multiple contexts may be combined → ./subdomain=support&langcode=de/
  7. Within each subdirectory/collection, configuration objects use the same name as in the main/root directory/collection → ./system.site.yml ↔ ./langcode=de/system.site.yml

Remaining tasks

User interface changes

No

API changes

No

Files: 
CommentFileSizeAuthor
#21 2168609-diff-18-21.txt1.24 KBvijaycs85
#21 2168609-config-translation-schema-21.patch6.26 KBvijaycs85
PASSED: [[SimpleTest]]: [MySQL] 64,417 pass(es). View
#18 2168609-diff-16-18.txt794 bytesvijaycs85
#18 2168609-config-translation-schema-18.patch5.68 KBvijaycs85
PASSED: [[SimpleTest]]: [MySQL] 64,383 pass(es). View
#18 2168609-config-translation-schema-18-test-only.patch936 bytesvijaycs85
FAILED: [[SimpleTest]]: [MySQL] 64,371 pass(es), 12 fail(s), and 0 exception(s). View
#16 2168609-diff-15-16.txt4.38 KBvijaycs85
#16 2168609-config-translation-schema-16.patch5.68 KBvijaycs85
PASSED: [[SimpleTest]]: [MySQL] 64,363 pass(es). View
#15 schema-to-language-overrides.patch2.32 KBGábor Hojtsy
PASSED: [[SimpleTest]]: [MySQL] 64,315 pass(es). View
#14 schema-to-language-overrides.patch2.32 KBGábor Hojtsy
PASSED: [[SimpleTest]]: [MySQL] 64,314 pass(es). View
#9 schema-to-language-overrides.patch2.31 KBGábor Hojtsy
PASSED: [[SimpleTest]]: [MySQL] 63,795 pass(es). View

Comments

vijaycs85’s picture

vijaycs85’s picture

Title: Move config translations (locale.config.[langcode]) in to new location to keep the config file name same » Move config translations (locale.config.[langcode]) in to new location
sun’s picture

I agree the files should be moved into a subdirectory. However, not necessarily into nested subdirectories à la ./translations/$langcode/.

To recap, this is what we originally discussed for storing configuration context overrides:

  1. Contextual override config lives in subdirectories of the config directory.
  2. The maximum nesting level is 1 → sub-sub-directories are not allowed/supported.
  3. For non-file-based configuration storage implementations, each subdirectory denotes a collection name (cf. KeyValueStore).
  4. Each subdirectory/collection specifies the context(s) to which it applies in HTTP query string syntax → ./langcode=de/system.site.yml
  5. Context namespaces (e.g. langcode) are not owned by any particular module/provider; they use common names → ./domain=example.com/, ./subdomain=support/, etc.
  6. Multiple contexts may be combined → ./subdomain=support&langcode=de/
  7. Within each subdirectory/collection, configuration objects use the same name as in the main/root directory/collection → ./system.site.yml./langcode=de/system.site.yml
vijaycs85’s picture

Issue summary: View changes

Thanks @sun. It is very useful information. Updating issue summary from #2

catch’s picture

Status: Postponed » Active
Gábor Hojtsy’s picture

Title: Move config translations (locale.config.[langcode]) in to new location » Move config translations (language.config.[langcode]) in to new location
Issue summary: View changes

Updated to current file naming pattern. I'm not sure I agree with the HTTP naming scheme for the directories, I can see how = and & in directory names could become a nightmare.

xjm’s picture

Issue tags: -CMI +Configuration system
Gábor Hojtsy’s picture

Assigned: Unassigned » Gábor Hojtsy
Issue tags: +sprint, +language-config

WIll take a stab at this tomorrow.

Gábor Hojtsy’s picture

Status: Active » Needs review
FileSize
2.31 KB
PASSED: [[SimpleTest]]: [MySQL] 63,795 pass(es). View

I've been looking at this at various places, and I think this is indeed more complex than what I expected it to be :/ I spent a few hours thinking about this and here is my conclusion.

So all the API in config is centered around the fact that config is accessible by a single string key (and is located in one place). If we introduce / as a possible component in the string (and storage to interpret that as a directory separator), then the string key would still not be applicable to the config schema. So to be directly applicable for the schema, we would need to keep the string key as-is. Then we would need to add additional arguments to look at subdirectories. Especially with @sun's proposal a full on 'context' key support for subdirectories. That sounds like would trickle down all over the API especially on all methods in Drupal\Core\Config\StorageInterface. That complication just so we can apply config schema to language overrides sounds to me like a problem...

So I am thinking if we can extend the config schema system in a way so that it can understand optional prefixes on filenames that if found may be removed from the file to then take the reduced file name for schema lookups. Then we keep the complexity within the schema system and not making the whole config system that much more complex. Also the schema system is designed so that we do actually need to look up the schema and then associate with the data, so we can look up the schema independently of the data, which is to our great advantage here.

Another issue with the all-config-in-one-dir situation as-is now was raised earlier that there may be too many files in the directory, if you have numerous blocks, menus, etc. in 3-10-20 languages. The schema solution would not solve that obviously. I'm not sure if this is a real problem currently given the limitations of number of files in directories are not that bad anymore :) Even FAT32 used to allow 65k files.

Totally untested but theoretically "complete" (functionality-wise) patch applied. I think an extension method for other prefix-based solutions would be good, but otherwise this is what maps into the simple key based solution of config without complicating with alternate buckets of storage for different things on the main config API.

sun’s picture

Thanks for kick-starting this, Gábor.

I can see how mixing a subdirectory name into the key name is problematic. Technically, the subdirectory represents a different "collection" of the storage. → I'm fairly sure that the proposal in #2190723: Add a KeyValueStore\FileStorage to replace e.g. ConfigStorage would help us to implement this more cleanly.

Gábor Hojtsy’s picture

I'm not sure if complicating the base API is worth it?

Also: *If* that would block, this that one cannot be less than a critical beta blocker.

vijaycs85’s picture

+++ b/core/lib/Drupal/Core/Config/Config.php
@@ -183,7 +190,12 @@ public function getName() {
+      $this->schemaName = preg_replace('!^'. preg_quote($prefix, '!') . '(.+)!', '\1', $name);

Getting 'fr.system.settings' for language.config.fr.system.settings. Not sure if this is what we want here: http://3v4l.org/jJFut

Gábor Hojtsy’s picture

@vijaycs85: Yeah that is true, we would need to remove one more part after this until the next dot.

Gábor Hojtsy’s picture

FileSize
2.32 KB
PASSED: [[SimpleTest]]: [MySQL] 64,314 pass(es). View

Here is a quick edited version of the above patch. It moves the preg escaping to the array construction and adds provisions for the langcode that I did not have before. This is untested, just hand-edited. @vijaycs85: can you maybe look at testing this? :)

@alexpott: any thinking about extensibility? I think it would maybe a problem to call out to module hooks in config factory, sounds like recipe for circular dependencies.

Gábor Hojtsy’s picture

FileSize
2.32 KB
PASSED: [[SimpleTest]]: [MySQL] 64,315 pass(es). View

Should of course not escape the actual pattern part.

vijaycs85’s picture

FileSize
5.68 KB
PASSED: [[SimpleTest]]: [MySQL] 64,363 pass(es). View
4.38 KB

Here is some more updates:

1. Moved schemaName part to a static method as test needs it
2. Updated preg to match language pattern
3. Removed exception for config translation in defaultConfig test
4. Added missing element for test config translation (language.config.it.tour.tour.tour-test.yml) to stop test fail.

Gábor Hojtsy’s picture

Status: Needs review » Needs work

So I think my regex was not good :D '\.[^\.+]\.' because the + should be right after ]. Your regex is better but not perfect either :) '\.[^\.]*\.' will match .. (two dots on a row), so as if the langcode is empty. I don't think that would be practically a problem, but strictly it is :)

Also, wow, the tests were already failing thanks to the language file not matching? Is this code also run from the test? Can we do a test with a broke file to catch the exception and prove it works for language files?

vijaycs85’s picture

Status: Needs work » Needs review
FileSize
936 bytes
FAILED: [[SimpleTest]]: [MySQL] 64,371 pass(es), 12 fail(s), and 0 exception(s). View
5.68 KB
PASSED: [[SimpleTest]]: [MySQL] 64,383 pass(es). View
794 bytes

#17:

'\.[^\.]*\.' will match .. (two dots on a row), so as if the langcode is empty.

- Updated.

the language file not matching?

- Yes, wasn't matching language file.

Is this code also run from the test?

- Yes, in defaultConfigTest calling this same static method.

Can we do a test with a broke file to catch the exception and prove it works for language files?

- Yep, removing the condition that skip translation would prove?

The last submitted patch, 18: 2168609-config-translation-schema-18-test-only.patch, failed testing.

Gábor Hojtsy’s picture

Yeah that proves it works, but it would be amazing to have permanent tests, ie. a test with language.config.xx.... that would throw and catch an exception and pass if it got that exception :) Otherwise looks amazing :)

vijaycs85’s picture

FileSize
6.26 KB
PASSED: [[SimpleTest]]: [MySQL] 64,417 pass(es). View
1.24 KB

Let's try this as we might need to change the defaultTest anyway to exclude this file...

Gábor Hojtsy’s picture

I think this is adequate coverage. The passes show this works:

Schema found for language.config.en.config_test.system and values comply with schema. ...
Schema found for language.config.fr.config_test.system and values comply with schema. ...
language.config.xx... is not valid configuration translation ...

Now we only need to figure out how to make this extensible and probably get other CMI reviewers on board.

Gábor Hojtsy’s picture

Assigned: Gábor Hojtsy » Unassigned
alexpott’s picture

+++ b/core/modules/config/lib/Drupal/config/Tests/DefaultConfigTest.php
@@ -91,17 +93,22 @@ public function testDefaultConfig() {
+        // Make sure invalid translation excluded.
+        if ($config_name == 'language.config.xx...') {
+          $this->pass(String::format('!config_name is not valid configuration translation', array('!config_name' => $config_name)));
+          continue;
+        }

+++ b/core/modules/config/tests/config_test/config/language.config.xx....yml
@@ -0,0 +1 @@
+foo: fr bar

This seems a weird test. What are we actually testing here?

Gábor Hojtsy’s picture

@alexpott: that tests that 'language.config.xx...' does not have schema. It indeed does not ensure that 'language.config.xx...' would appear among the ones which don't have schema, that would need a fail() later if this condition was never matched. All the other language.config... files are not specifically exempt in the test and therefore will fail if no schema found for them.

Gábor Hojtsy’s picture

Assigned: Unassigned » alexpott
Status: Needs review » Needs work

Anyway, based on discussions with @alexpott he is not happy either with the language baked into the factory and that we should also solve the use case here of config files being aggressively overflown by lots of translation files, if there are many languages / lots of config, so the subdirectory solution would be preferred. He has code in the making which supposedly does not affect the config() core API badly. Marking needs work for that, and assigning to him.

alexpott’s picture

Just opened #2201437: [META-1] Config overrides and language to as the solution is far beyond the scope of this patch.

Gábor Hojtsy’s picture

Status: Needs work » Postponed

Sounds like to me we need to postpone this on #2201437: [META-1] Config overrides and language or maybe even mark as a duplicate, since that supersedes this (would include a solution for this). Marking postponed for now.

Gábor Hojtsy’s picture

Status: Postponed » Closed (duplicate)

More accurate to say this is a duplicate of #2201437: [META-1] Config overrides and language.

Gábor Hojtsy’s picture

Issue tags: -sprint