commit message (message needs work. Not sure if we are at the point for suggesting one yet.):
Issue #1648930 by Jose Reyero, YesCT, tstoeckler, spearhead93, alexpott, yched, Gábor Hojtsy, fago, effulgentsia, dawehner: Introduce configuration metadata and use it for translation.


As a site builder I want Drupal core (and module/theme) shipped configuration, such as default views, email text settings, the site maintenance message, the default contact form, and so on, translatable just as the software it ships with. Also I want to let translators who might not have access to change user email text, views or contact form settings to be able to modify and update the translations as needed.

With the Drupal 8 configuration system, multilingual sites loose two key features which need to be restored and the ultimate goal in Drupal 8 is to make configuration generally translatable. All three require metadata, about the configuration stored, to be properly restored.

  1. When default configuration is saved into the database, the code setting up the default configuration used to use t() on the strings that were translatable. So configuration was saved initially in the site's default language. (Think content types, forums vocabulary name, etc.) This is lost with Drupal 8 because we have no information as to which parts of configuration are to be translated or are translatable.
  2. For variable_set()/get() based settings, unmodified menu items and many other things in Drupal 7, unless you modify the original values, translation updates are applied to your configuration translation. This lets you get translation updates for shipped configuration so long as you don't customize it. The configuration system does not currently have granular modification tracking and again no translatability metadata.
  3. Finally, other pieces of configuration have one-off translation implementations but should be generalized. For configuration translation, core only has two implementations: path alias language support and date format language support. Both have highly custom code and highly custom user interfaces to operate. First to reproduce these (at least date formats as ported to Drupal 8's configuration system), we need metadata about translatability. Also, we want to generalize this feature to other types of configuration. We cannot say Drupal is a multilingual CMS in 2013 onwards when it cannot even translate the site name you enter!

Proposed solution

Design goals

The following design goals have been set out for this system:

  1. Separate metadata from the data itself to avoid repetition and keep .yml configuration files simple.
  2. Avoid repetition in the metadata files themselves and allow for representation of metadata for pluggable systems (image effects, views, etc).
  3. The metadata must not be needed for regular pages (that just retrieve configuration values).

Metadata file format and placement

Introduce configuration metadata with basic information about each element in the configuration structure. Using YML files for the description of the metainformation, we can use the same format as applied to configuration itself.
This metadata will match configuration data with data types defined by the TypedData API so a) we'll be able to access configuration data as typed data and b) We can identify data types in the configuration data that are to be localized.
Metainformation is proposed to be placed in a sibling directory of the shipped configuration files using the same machine name. For example, core/modules/system/config/system.maintenance.yml looks like this:

enabled: '0'
message: @site is currently under maintenance. We should be back shortly. Thank you for your patience.

The metafile for this YML is at core/modules/system/meta/system.maintenance.yml (note the meta subdirectory in place of config):

   .label: 'Put site into maintenance mode'
   .type: boolean
  .label: 'Message to display when in maintenance mode'
  .type: text

The metadata file follows the same structure as the original YML file but uses keywords prefixed with . (dot) to add more information to the data. This is to differentiate the child elements from the metadata elements, it serves the same purpose as # in Form API to differentiate properties from children. The keywords used here are (two groups):

Keyword group I. data type definition

For the data type definition of the element as specified by the TypedData API:

- .label: contains the 'label' for the data type definition for the element.
- .type: contains the 'type' for the data type definition for the element.
- .definition: contains an array with any other keys of the data type definition for the element

** While we could stick all the data type definition into the '.definition' array, more than label and type is barely needed. More definitions (than label and type) is just for a) rare cases when we need to use the full TypedData API properties and b) also for future extensions allowing contributed modules to define their own TypedData data types which may need more complex definitions. Having 'label' and 'type' will be enough for almost all cases. Leaving .definition out of the file for those makes the metadata files more readable and easier to write (they save one level of indentation). For a side by side comparison of how this would look using only the '.definition' array see more discussion in the follow-up #1851498-7: Polish file format of configuration metadata (for translation)

keyword group II. Nesting and Includes

For defining the structure of the configuration data matching nested elements / plugins in the configuration data with their metadata:

- .include: Name of other configuration metadata element to be included (and merged with the current metadata). This allows reusing metadata from components for different containers. An example is the '' metadata using multiple instances of 'image effects'.
- .list: Contains an array of metadata to be reused for all of the elements in a list. An example is the user.mail metadata that defines mail text and body once to be used for all the emails in the user.mail configuration file.

Metadata applied to multiple configuration files (% placeholder)

When metadata need to apply to multiple configuration files, the metadata file name should use a % placeholder where variable values are possible. For example, contact form categories are in files contact.category.{category name}.yml, such as contact.category.webmaster.yml, contact.category.sales.yml, but they follow the same structure, so the metadata should apply to all of them. A % placeholder is used (same as with tpl.php template file wildcards) to account for this, so core/modules/contact/meta/contact.category.%.yml applies to all contact categories. The structure of the metadata file is the same as above, it just applies to a set of configuration files.

Metadata for nested data structures

In many cases configuration contains nested data structures provided by plugins or other sources where the nested data structure in itself has a pattern and need configuration metadata. In this case, nested levels of metadata can be provided by referring to nesting patterns. Take image effects. A sample configuration file for that (core/modules/image/config/ looks like the following:

name: medium
label: Medium (220x220)
    name: image_scale
      width: '220'
      height: '220'
      upscale: '1'
    weight: '0'
    ieid: bddf0d06-42f9-4c75-a700-a33cafa25ea0

Note the main properties name and label as well as the property "effects" which embeds configuration that has its own structure, name, etc. A corresponding metadata file for image styles uses the above explained wildcard matching for all image effects core/modules/image/meta/

  .label: 'Machine name'
  .label: 'Label'
  .type: text
  .label: 'Style effects'
    .label: 'Image effect'
      .label: 'Weight'
      .type: integer
      .label: 'IEID'
      .label: 'Machine name'
      .include: 'image.effect.[]'
      .label: 'Data'

This defines metadata for name and label as usual and then defines 'effects' as a list of nested objects which under the '.list' key have metainformation to be reused for all the effects in the list, that are indexed in the configuration data by UUID.
The first part of this (.list) metadata are the common properties for all image effects that are dependent on the container object (image style), thus they belong to the container, not to the plugins themselves.

The second part, inside 'data' is an include that will include for each of the image effects the metadata defined by 'image.effect.[ ]', [ ] being the value of 'name' property from each of the nested objects in the configuration data. ( %parent refers to the parent element of the one the include is in. So this refers to the parent elment's 'name' property)

In the example, this include will add the metadata from 'image.effect.image_scale' (name = image_scale).

Note that for each of the nested effects we are merging two arrays of metadata to get the final result:
1. Metadata for image effects specific of image styles (weight, uuid), this is defined in
2. Specific metadata for a given effect, in this case 'image_scale', defined in image.effect.image_scale.yml

While the first one (1) is specific of image styles, the other (2) can be reused for image effects in the case they are used by a different object (that is not an image style).
This accounts for the case of a component (typically a plugin) being used in several, independent systems it has no knowledge of. Example: the configuration for a field formatter can be included in a view, a block, a regular entity display... The metadata provided by the field formatter about its own settings is destination-agnostic and can thus be included by all these independant systems.

The image_scale specific metadata file explains width, height and upscale properties from the image_scale specific embedded structure:

Metadata in core/modules/image/meta/image.effect.image_scale.yml

  .label: 'Width'
  .type: integer
  .label: 'Height'
  .type: integer
  .label: 'Upscale'
  .type: boolean

These two metadata files merged serve as description of this nested object.

Implementation details

The metadata system is based on existing subsystems implemented by the TypedData API and the configuration APIs. The .yml files are read and handled by the existing configuration API readers, and the data types are defined and handled using TypedData. A new config_element type is defined to handle elements that have structured data (lists or have children).

The metadata objects are built recursively so the system supports any level of nested objects that contain other nested properties. The system is extensible; modules can define their own TypedData configuration wrappers. However we may not need to extend it since the complexities required by Views are already supported.

Lets use site name as an example.
Files related to this example:
contains in part:

name: Drupal

and core/modules/system/meta/
contains in part:

  .label: 'Site name'
  .type: text

To get type wrapped data based on a config element:

// Get a TypedData wrapper for this configuration. This returns a Drupal\Core\Config\Metadata\TypedConfig object.
$wrapper = config_wrapper('');

// This returns not a string but a Drupal\Core\TypedData\Type\String object.
$name_property = $wrapper->get('name');

$name_property->getType() // will return 'text'.
$name_property->getValue() // will return the site name ('Drupal' by default).
$name_property->getElementLabel() // will return 'Site name'. Not part of this patch. See: ...

Locale module extends the TypedConfig by providing a LocaleTypedConfig that implements TranslatableInterface, actually allowing to get translated configuration data using the localization system. This is used to build the language-specific override data that will be loaded on top of each configuration object.

// Wrap system site information with Locale config accessors.
$data = config('')->get();
$wrapper = new LocaleTypedConfig('', $data, locale_storage());
// Get a config wrapper that works with the Spanish overriden version of this config.
$translation = $wrapper->getTranslation('es');
// Get a a nested array with the translated strings.
$strings = $translation->getData();

For a key piece of the patch, see locale_config_update_multiple() where the LocaleTypedConfig is used to retrieve translations of pieces of configuration from the Locale system and saved as overrides with the configuration system for later retrieval.

The full class/interface structure of the patch is as follows:

Some great side benefits

Though the primary use of this is getting the default imported configuration translated, it can also be used for building a data browser for any configuration object and also for auto generating settings forms.
The data browser is demonstrated by an example module, which also provides metadata for Views, not yet included in the patch, see:
- Config Demo module (browse or translate any configuration),
- Views configuration, generated by the demo module out of the metadata, screenshot,

Translation API

The code rebuilds configuration translations every time string translations are updated (or new modules/themes are added). Using the string location data structure for source strings, we can track which strings belong to each configuration object so we only need to update affected ones when updating the translation.

Translatable data types are identified using hook_data_type_info(), see 'text' in system_data_type_info() in the patch. This data type is meant to represent 'human readable (aka translatable) text'. There are two small additions to the definition of these elements that can be added to the '.definition' array of the metadata if needed:
- 'translatable' => FALSE, to opt out from translation.
- 'locale context', to define locale context for translation,

In short, a configuration value is translatable (by core locale module) if:

  • It is a 'text' data type, and doesn't have a 'translatable' => FALSE flag
  • It remains with its default value from initial module installation.

There is no support yet for translating these strings once customized, since the localization system doesn't support user defined strings nor translating from source languages other than English. Module default configuration files, the ones provided with the module, are assumed to be English language unless they contain a different high level 'language' property with a different language code.

Test drive the patch!

An example of using these with specific details in comment #198

  1. Apply the patch, install a fresh Drupal 8
  2. Under Extend, Enable the Locale module (Language module is a dependency so it gets enabled too.)
  3. Under Configuration » Regional and languages » Languages add a language
  4. Go to the string translation page at Configuration » Regional and languages » User interface translation
  5. Search for strings like "Medium" (should find Medium (220x220)) or "Account cancellation request" (should find the email subject). If it does not find that souce string, might have to visit the page at Configuration » Media » Image styles and click on Medium (220x220).
  6. Translate either the image style name or the email subject. For example translate Medium (220x220) to Middle (220x220)
  7. Observe change applied by .yml files in active storage:
    • in the ui Configuration » Media » Image styles (switch to the language you translated it for, for example: es/admin/config/media/image-styles)
    • appearing with the translation overrides in Configuration » Regional and languages » User interface translation (admin/config/regional/translate)
    • in the active storage in sites/default/files/config_XXXXXXXXXXXXXXXXXXXXXXX/active/
    • similar data structures appearing in the database as config overrides. [searching in phpmyadmin for Middle]

Remaining Tasks

  • (done)#250 7.b make this issue summary exactly accurate to match the most recent patch
  • (done) verify the test drive the patch instructions are accurate.
  • (done) make follow-ups for namespace docs style only file changes that were removed from the patch


Unless critical to getting this in before feature freeze please do not start new discussions on this issue.
Please open a follow-up for items related to this issue. List them here.

Code standard follow-ups:

#298 config_metadata-1648930-298.patch94.64 KBJose Reyero
FAILED: [[SimpleTest]]: [MySQL] 49,563 pass(es), 1 fail(s), and 0 exception(s). View
#284 config_metadata-1648930.252-284.txt1.08 KBpenyaskito
#284 config_metadata-1648930-284.patch89.82 KBpenyaskito
PASSED: [[SimpleTest]]: [MySQL] 49,242 pass(es). View
#252 config_metadata-1648930-252.patch89.82 KBJose Reyero
FAILED: [[SimpleTest]]: [MySQL] 49,298 pass(es), 1 fail(s), and 0 exception(s). View
#252 config_metadata-1648930-252-diff.txt4.64 KBJose Reyero
#247 config_metadata-1648930-247.patch89.85 KBYesCT
FAILED: [[SimpleTest]]: [MySQL] 48,840 pass(es), 15 fail(s), and 24 exception(s). View
#247 interdiff-244-247.txt52.68 KBYesCT
#246 New CMI Metadata architecture (2).png98.57 KBGábor Hojtsy
#244 config_metadata-1648930-244.patch92.2 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 49,299 pass(es). View
#244 config_metadata-1648930-244-diff.txt16.93 KBJose Reyero
#236 config_metadata-1648930-236-diff.txt12.9 KBJose Reyero
#236 config_metadata-1648930-236.patch94.64 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 49,172 pass(es). View
#227 config_metadata-1648930-228.patch93.5 KBJose Reyero
FAILED: [[SimpleTest]]: [MySQL] 48,962 pass(es), 4 fail(s), and 2 exception(s). View
#227 config_metadata-1648930-228-diff.txt11.64 KBJose Reyero
#226 config_metadata-1648930-226.patch90.79 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 48,789 pass(es). View
#226 config_metadata-1648930-226-diff.txt23.49 KBJose Reyero
#214 1648930_214.patch89.08 KBchx
PASSED: [[SimpleTest]]: [MySQL] 48,694 pass(es). View
#214 interdiff.txt5.49 KBchx
#206 interdiff-chx_E_STRICT_currenttextarea-194-206.txt772 bytesYesCT
#206 interdiff-justDocs-194-206.txt11.98 KBYesCT
#206 config_metadata-1648930-206.patch88.55 KBYesCT
PASSED: [[SimpleTest]]: [MySQL] 48,622 pass(es). View
#204 config_metadata-1648930-204-diff.txt996 bytesJose Reyero
#200 notes-1648930-194.txt11.36 KBYesCT
#198 configmeta-s01-dbsearch_a-2012-11-16_0753.png154.3 KBYesCT
#198 configmeta-s02-dbsearch_b-2012-11-16_0756.png156.03 KBYesCT
#194 1648930_194.patch87.24 KBchx
FAILED: [[SimpleTest]]: [MySQL] 48,278 pass(es), 0 fail(s), and 2 exception(s). View
#194 interdiff.txt665 byteschx
#193 1648930_191.patch87.26 KBchx
FAILED: [[SimpleTest]]: [MySQL] 48,291 pass(es), 0 fail(s), and 566 exception(s). View
#189 1648930_189.patch86.9 KBchx
FAILED: [[SimpleTest]]: [MySQL] 48,271 pass(es), 0 fail(s), and 566 exception(s). View
#189 interdiff.txt29.55 KBchx
#187 config_metadata-1648930-187.patch81.21 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 48,312 pass(es). View
#187 config_metadata-1648930-178-187-diff.txt47.19 KBJose Reyero
#179 1648390_178-diff.txt7.06 KBJose Reyero
#178 1648390_178.patch81.78 KBchx
PASSED: [[SimpleTest]]: [MySQL] 48,303 pass(es). View
#154 interdiff-meta-1.txt5.16 KBalexpott
#154 evasive-interdiff.txt12.74 KBalexpott
#154 config_metadata-1648930-154.patch79.53 KBalexpott
PASSED: [[SimpleTest]]: [MySQL] 48,237 pass(es). View
#139 New CMI Metadata architecture (1).png93.23 KBGábor Hojtsy
#138 New CMI Metadata architecture.png97.57 KBGábor Hojtsy
#136 config_metadata-1648930-136.patch77 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 46,944 pass(es). View
#136 config_metadata-1648930-136-interdiff-119.txt83.28 KBJose Reyero
#133 config_metadata-1648930-133.patch94.53 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 46,975 pass(es). View
#133 config_metadata-1648930-133-interdiff.txt56.71 KBJose Reyero
#131 CMI Metadata architecture.png83.9 KBGábor Hojtsy
#127 config_metadata-1648930-127.patch96.76 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 46,331 pass(es). View
#127 config_metadata-1648930-127-interdiff.txt24.47 KBJose Reyero
#125 config_metadata-1648930-125.patch96.71 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 46,328 pass(es). View
#119 config_metadata-1648930-120.patch101.8 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 46,324 pass(es). View
#114 config_metadata-1648930-114.patch60.61 KBYesCT
FAILED: [[SimpleTest]]: [MySQL] 42,621 pass(es), 2 fail(s), and 1 exception(s). View
#114 interdiff-109-114.txt6.83 KBYesCT
#113 interdiff-101-109.txt61.21 KBYesCT
#109 config_metadata-1648930-109.patch60.59 KBJose Reyero
FAILED: [[SimpleTest]]: [MySQL] 42,394 pass(es), 3 fail(s), and 1 exception(s). View
#101 1648930-config_metadata-101.patch49.19 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 42,322 pass(es). View
#101 views_config_metadata_example.png146.59 KBJose Reyero
#91 1648930-config_metadata-91.patch57.13 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 42,109 pass(es). View
#88 1648930-config_metadata-88.patch48.9 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 42,007 pass(es). View
#85 1648930-config_metadata-85.patch47.65 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 41,923 pass(es). View
#81 1648930-config_metadata-81.patch47.65 KBJose Reyero
FAILED: [[SimpleTest]]: [MySQL] 41,902 pass(es), 1 fail(s), and 0 exception(s). View
#80 1648930-config_metadata-full-80.patch45.14 KBspearhead93
FAILED: [[SimpleTest]]: [MySQL] 41,904 pass(es), 1 fail(s), and 0 exception(s). View
#78 1648930-config_metadata-full-78.patch89.05 KBJose Reyero
FAILED: [[SimpleTest]]: [MySQL] 41,770 pass(es), 1 fail(s), and 0 exception(s). View
#78 1648930-config_metadata-diff-78.txt45.14 KBJose Reyero
#76 1648930-config_metadata-full-76.patch85.9 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 41,522 pass(es). View
#76 1648930-config_metadata-diff-76.txt42.38 KBJose Reyero
#68 1648930-typed_data-config_metadata-full-68.patch73.17 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 41,390 pass(es). View
#68 1648930-typed_data-config_metadata-68.diff.txt31.72 KBJose Reyero
#67 1648930-config_metadata-67.patch25.74 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 41,252 pass(es). View
#65 1648930-config_metadata-65.patch21.13 KBJose Reyero
FAILED: [[SimpleTest]]: [MySQL] 41,227 pass(es), 33 fail(s), and 2 exception(s). View
#48 config-data-type-info-hook-do-not-test.patch4.01 KBtstoeckler
#47 1648930-39-config_metadata_3.patch17.47 KBtstoeckler
PASSED: [[SimpleTest]]: [MySQL] 40,631 pass(es). View
#47 config-metadata-interdiff-1-3.txt4.32 KBtstoeckler
#45 1648930-39-config_metadata_1.patch16.32 KBtstoeckler
PASSED: [[SimpleTest]]: [MySQL] 40,629 pass(es). View
#45 config-metadata-interdiff.txt4.91 KBtstoeckler
#42 config-metadata-minor-cleanup.txt1.88 KBtstoeckler
#41 1648930-39-config_metadata.patch15.99 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 40,639 pass(es). View
#39 1648930-39-config_metadata.patch0 bytesJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 40,629 pass(es). View
#15 1648930-14-config_metadata.patch17.66 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 37,055 pass(es). View
#5 config_metadata-04.patch12.85 KBJose Reyero
FAILED: [[SimpleTest]]: [MySQL] 36,937 pass(es), 72 fail(s), and 108 exception(s). View
#4 config_metadata-03.patch8.42 KBJose Reyero
PASSED: [[SimpleTest]]: [MySQL] 37,019 pass(es). View
config_meta_form.png20.01 KBJose Reyero
Members fund testing for the Drupal project. Drupal Association Learn more


beejeebus’s picture

i've discussed this patch with catch and chx over the last few days. i don't like that we're introducing metadata for all the things instead of just a narrowly focussed feature based on locale's needs.

i'd really like us to consider something that simply ships files identifying the config paths (config path == some.config.file:some.config.key) that are targets for translation, with a label, and that's it. there may be more, but that should be driven solely by the needs of the locale system, not by a metadata system for all the things that allows modules to do all the things with it.

while talking to chx about this today, he came up with some ideas about how we might implement this and created #1861640: Provide config metadata solely for translation.

i hope it's not too late for us to take one step back here and consider a simpler, more targeted implementation that meets locale's needs. i've posted something similar to #1763640-59: Introduce config context to make original config and different overrides accessible.

Gábor Hojtsy’s picture

Some metadata yes, but I am more and more worried about how we do it. Like, why is it not in code if only developers need to do this?

Once again it could be in code IF it is statically parseable without actually running the code *and* can be related to the "exported" .yml data itself, so a static code/data analyser that is used on can extract all the translatable pieces. Given these constraints, even if it is embedded in PHP code for example, it would need to follow very strict structural rules to be readily parseable (eg. strictly no call-out to API functions or plugin methods), so it would equally need to be declarative, just encoded in an executable-looking format.

Jose Reyero’s picture

94.64 KB
FAILED: [[SimpleTest]]: [MySQL] 49,563 pass(es), 1 fail(s), and 0 exception(s). View

Rerolled for changes into the locale bulk / update api in #1804688: Download and import interface translations
The only bits changed are locale backend api integration and 1 test update for a new property in

@chx #295,

What's here is not consensus but teeth gritting admittance that we do not have a better idea and noone had the time to come up with something better. However, that was the case with the December 1 deadline. With feature squeeze in, I am less and less convinced every time I look at it.

Well, that's what happens with deadlines. However this was not a last minute thing and I don't know whether we are producing anything much better in another 2 or 3 months.
(And certainly not me, as I had scheduled enough of my time around the previous deadline, but other people may have the time for it...)

Anyway, we have that follow up issues to make the format better, but no one is likely to invest much time on them unless we are sure we really have some metadata to improve.


... instead of just a narrowly focussed feature based on locale's needs.

Because if we do so, any contrib module trying to implement any extended translation with some decent UI would need to completely ditch this and add it's own hook / metadata / whatever and then we'll be back at square 1, or Drupal 7.
The problem with narrowly focused features is that they are just that, "narrowly focused features", and that is not how you get great frameworks to build upon, but more often just more lines of code.

sun’s picture

There are other configuration system issues for which metadata might be the only viable solution.

One of them is #1653026: [META] Use properly typed values in module configuration — which would essentially mean that we'd leverage the metadata for every Config::save() operation; not only to switch from the enforced casting to strings to an enforced type-casting to the expected/defined data type, but potentially also to add a [pre-]storage-level validation of values. (Configuration is rarely written, so this is feasible.)

I additionally want to point out that the combined purpose of #1648930: Introduce configuration schema and use for translation and #1763640: Introduce config context to make original config and different overrides accessible is a base building block for multilingual (!= translatable) configuration; i.e., allowing more than translatable strings to diverge per language.

beejeebus’s picture

re #299 - data types: well, then lets use XML. when we chose yml, the CMI team was explicitly rejecting schema for our config. if we're going to do this, then lets use the right tools for the job (xml), and not invent our own more complex, less powerful schema handling. just to be clear, that's not my preferred option. however, i'd prefer that to yml plus our own, unique to Drupal, less powerful than xml-schema thing.

re #299 - multilingual (!= translatable) configuration: how does that relate to schema for our config? (not a rhetorical question.) other than having different values for config keys based on language, what does that mean, and how does it require a schema for our config data?

Gábor Hojtsy’s picture

@beejeebus: re multilingual != translatable, the subtle differentiation we make there is where most of the discussion on #1851498: Polish file format of configuration metadata (for translation) happens. We consider translatable are the things that are *translations* of each other. Such as how Montag is a German translation of Monday. Multilingual is where you use different values per language but they are not translations of each other, such as when you use a single "work in progress" node as a landing page for your Japanese site but a full blown panel with numerous panes as your French front page. These two are not translations of each other, just different things used for different languages. Both of them are the setting value for your default front page path but apply to different languages.

The translatables we need to identify first and foremost through the .yml files because they live in "exported configuration" that works as part of the software. Eg. Views and user notification email subjects and bodies shipped with Drupal core should be translatable as part of the software on That gives us the requirement to have a format that lets us statically parse the .yml file for all the translatable pieces. This has no need for types or labels, it is just binary information about cherry-picked pieces of config.

Then to support the multilingual use case (and more importantly to be able to generate translation forms for anything that is not shipped configuration, such as a view you create on your site), we need to reproduce all the pieces of the configuration that could be multilingual in a sensible settings form. Eg. if you look at the account settings screen on your site, you want to translate the Anonymous user name and all the email subjects and bodies. BUT you don't want to let translators change the admin role or change signup security settings for your site. That would be disastrous. So we need to be able to build a highly simplified version of the form that does not take its data from the original config, does not save to the original config, does not fire the regular hooks for the forms, and so on. In essence we need some kind of representation as to how a translation form would look like for this data (view, account settings, etc). That needs type info, labels, etc. This is the most contentious part. For example, here @catch argued we should support the full force of Form API and @chx argues we should support as little as possible, while others suggest we should use XML (Schema), although it is not clear how would XML Schema provide us with features such as types, labels (descriptions?) that we can map to Form API forms.

Damien Tournoud’s picture

Title: Introduce configuration metadata and use for translation » Introduce configuration schema and use for translation
Status: Reviewed & tested by the community » Needs review

My point all the way back in #13 is still valid. We are talking about introducing a schema for configuration.

The format itself seems limiting and inconsistent. Some examples:

  • It doesn't support marking properties as required
  • It doesn't support arrays/list
  • It calls "list" what is actually a "map"
  • It relies on magic prefixes ("."), which are a pile of pain (see the Form API use of #)
  • It doesn't seem to support complex data types properly (the handling of that seems at odd with the rest of the Typed Data API)

Also, the schema is currently defined in two places (at least for configuration entities):

  • In the schema
  • In the entity class

Fago #117 was never properly replied to. And really, it all boils down to: why would we re-write the Typed Data API instead of just using it? We already have Typed Data information for configurable entities, we really just need to add a way to declare the same for other CMI data, and there is really no need to do that in YAML, we can and should do that directly in PHP.

Gábor Hojtsy’s picture

@Damien Tournoud: how do you imagine that being defined directly in PHP and still processable statically (without actually running the code). Yes, we can put that in PHP if all coders of that PHP code follow strict guidelines, so we can parse said PHP code statically without running it, which means it needs to be in fact a declarative description, which is better done in a declarative format to aid that poor developer AFAIS.

Gábor Hojtsy’s picture

Status: Needs review » Reviewed & tested by the community

Based on an IRC discussion with @DamZ, his proposal is we adopt the typedData properties/structure as-is (as in to describe config structures (eg. types would be like 'string_field' and 'integer_field' and 'translatable' would be its own property). The plugin based inclusion system would still be needed that is introduced in the patch and DamZ understood we need this in a statically processable format (so we can actually extract translatable strings without running the code), so it all ends up in a YML file and would need the separations and inclusions introduced as well as the file naming, locale mapping and other subsystems introduced by this patch.

Given that, his concern seems to be tightly related to reworking the file format with yet another proposal that is not yet brought up in other followups. So his concerns would be resolvable fully in #1851498: Polish file format of configuration metadata (for translation).

Moving back to RTBC with that.

Jose Reyero’s picture

Status: Reviewed & tested by the community » Needs work

Congratulations! We got the issue paged on the issue tracker, which makes it still harder to follow :-(

I can understand people not liking the format but the last ones.... So first we trim it down hardly (we don't need forms, validation, etc, bla, bla) to let it at the very minimum we need for translation, and now the complain is it doesn't provide enough information, it's not a real schema, etc.. ?

This is what I think we can do:
- Drop this one. I won't cry, I promise.
- Add real schema data:
(That would be a minimum api patch)
- Add in next steps, small patches: validation, form generation, translation, plugin support, etc, etc..

This is what I can do with Kwalify:
Content of contact.category.%.schema.yml

type: map
class: ContactCategory
    type: str
    type: str
    type: seq
      - type: str
    type: str
    type: int

Generated class:

kwalify -a genclass-php -tf contact.category.%.schema.yml > ContactCategory.php

class ContactCategory {
    var $id;         // str
    var $label;      // str
    var $recipients; // seq
    var $reply;      // str
    var $weight;     // int
    function __construct($arr=NULL) {
        if (! $arr) return;
        $this->id         = @$arr['id'];
        $this->label      = @$arr['label'];
        $this->recipients = @$arr['recipients'];
        $this->reply      = @$arr['reply'];
        $this->weight     = @$arr['weight'];

Now for anyone concerned, the DX for this is harder to learn, but at least we would be having some real schema for yaml, that can be used with external tools (parsers, validators, class generation for a few languages not only PHP, etc...)

We don't even need a patch to get started, just a simple statement and documentation about where it should be placed: "We use Kwalify schema definitions for configuration files and they need to be placed in ... "

Kwalify exists as a debian package btw.

Gábor Hojtsy’s picture

Erm, how would that make it possible to generate forms? Damien pointed out above that we szould mirror the TypedData available to entities in general, so having a separate type/schema system here was a major drawback in terms of DX. He indicates building the format pure and simple on TypedData would be better DX. I am sure we keep could keep discussing this for as long as we want, months on end.

Gábor Hojtsy’s picture

Issue tags: +Avoid commit conflicts

Adding tag since this keeps not being applyable all the time...

Dries’s picture

I spent some more time on this patch. It's really hard and have been breaking my head on it for a while. After having spent 3+ hours on it, I still don't feel that I fully grok this approach. As a result, I'm struggling to provide good direction for the time being.

Here is my current stream of thoughts: I understand that this is important but I'm nervous about introducing a new custom file format that is pretty hard to grok. I still struggle with the fact that it only marginally improves the translator experience and that it is still an incomplete solution. I'm also not interested in having core automatically generate simple forms from the meta files; I don't want to introduce another form API, at least not in Drupal 8. It's too late for that.

It feels like we should have some time to continue to figure this out. I encourage people to explore simpler solution similar to what chx is working on (just fixing the translation regression) as well as effulgentsia (simplified file format). I think both are interesting evolvements that give us a point of comparison.

I'm not giving up on this issue. I'll continue to study this issue and will think about it more. In the mean time, I encourage people to continue to work on this issue and the other issues listed above.

Jose Reyero’s picture

@Gábor #309,
Config entities don't have any type schema at all they're just adhoc php code. So they don't use the TypedData API AFAIK.
Field types in the TypedData API are multidimensional arrays of things. They're not reusable at all for anything that are not our multlingual-multi-everything-things entities.
Overall the TypedData API has caused more trouble than benefit -I mean for the purposes of this patch-. And if anyone pretends to use Entities for all this I'd dare them to write up the code for any simplest case that has a translatable string. (Which just means I am clueless about it, I don't buy the "entities are good for everything" and I'm not going to understand the explaination either, so please just show me the code)

Oh, about how to generate forms from schema data (=data type + validation rules), if you add a label to it, I think that's pretty trivial, see initial versions of this very same patch :-)

Jose Reyero’s picture

Status: Needs work » Postponed

Thanks for reviewing this one.

Now, not replying to anyone, just some thoughts on how to move on.

I'm not sure anyone will want to continue with this one at least without some important buy-in on the idea of adding metadata, certainly not me. So I'm marking this as potponed at least until we see what happens with the other patches trying to address similar issues.

Or if ayone wants to reroll it, just in case, there are things that could be better / simpler / cleaner:
- The TypedData API has been more trouble than benefit, we really don't need it here, I'd drop it if I ever do a reroll (which doesn't mean we cannot use the type names like 'string' or 'integer' from TypedData, just trying to map the whole thing, as config_elements are typed data themselves, has been exhausting)
- Bundling the translation feature into this only made sense with the timing of the feature freeze, so for any future move my advice would be splitting it in two patches.
- Forget about plugins for now, since modules using plugins don't seem to have any clean pattern for adding plugin data into configuration yet, nor figuring out dependencies, so we should wait at least until this exists. (Or at least wait for node types to be converted to configuration to see how they look like)

Gábor Hojtsy’s picture

I'm also not interested in having core automatically generate simple forms from the meta files; I don't want to introduce another form API, at least not in Drupal 8. It's too late for that.

All right, we have been working on this issue since the summer (and had g.d.o discussion pieces and drupalcon/camp meetings of the underlying ideas since 2011 summer), but have indeed had a hard time to get reviewers and people interested to support it. Looks like the form generation task will be fully on contrib then. With the form generation use case discarded though, I don't think D8MI has any reason to push for a metadata of any richness (eg. labels, types), because we will need to do it all over in contrib (either by duplicating Form API structures as-is or introducing some kind of metadata). Then contrib modules will either choose to not be multilingual with their config or support that contrib system. (In short, if this is an ugly system but we don't have better ideas, contribs will need to cope with this either way. If we have better ideas then we'll be able to experiment in contrib. The goal of making Drupal 8 more multilingual for configuration is hardly achieved.)

So the core use case is back down to the "simple" problem of identifying strings for, so "exported configuration" as part of the software can be translated and translations can be applied to installations of Drupal and its distributions. A huge chunk of this patch deals with that (importing those strings to the locale system, exporting translations from the locale system to .yml files, etc), so that will need to be carried over to whatever format patch we have for identifying those strings. Also, if we keep the assumption that we need to identify those strings statically, then we also need the inclusion and plugin support in the metadata format. So in fact all we loose if the type and label from the metafile (as thus the TypedData connection altogether), the rest of the complexity for describing nested structures with plugins to identify their pieces to extract is needed. So we will need to salvage that piece of this code as well.

I think the conclusion is that core will be simpler, contrib (wanting to support config translation) will be much more complex but the UX will benefit from it. Contrib wanting to implement multilingual for config was not an easy ride in Drupal 7 either, we'll not have a huge step forward.

Gábor Hojtsy’s picture

Untagging with things not relevant anymore.

plach’s picture

Disclaimer: this is no personal attack to anyone, just my thoughts in this moment.

I think D8 cannot ship without a solid API to support multilingual configuration. It's a hard task but this is the time to achieve it. Jose and Gabor have done a tremendous work here, which is going to bring us many advantages besides what is strictly required by the D8MI.

A scenario where we provide a similar API in contrib is simply desolating: we will end-up like with the Variable API which, despite being a sensible solution to many problems we've been having for years now, is leveraged by few projects besides i18n.

Moreover, I'd be tempted to say that if we fail to provide a system allowing us to support multilingual configuration, the D8MI cannot be considered successful. And we should stop considering this a minor problem: our usage statistics tell us that i18n in installed on more or less the 10% of the Drupal installations out there, and this is not counting very simple multilingual websites not needing it. In addition we should consider that doing multilingual websites in Drupal is still a very hard task, which is potentially steering away many potential Drupal adopters. With a full-fledged i18n system in core and a cohesive API allowing contrib to implement i18n easily (DX is crucial here), we will see the above percentage raise upper and upper. Let's stop thinking about multilingual sites as a minority that can be safely ignored, please: at the BadCamp keynote @Dries defined i18n a key factor for Dupal's success in Europe. Key factor.

I'm not diving into implementations details, I didn't follow this issue closely enough to say something relevant, besides the fact that the raised DX concerns seem to be addressable to my eyes. I just want to point out that, from a project management perspective, rescoping this issue to just being able to parse translatable strings in configuration is pure madness.

I can be biased but IMO it's time to put an end to this: we cannot ship with yet another version of Drupal which can do amazing things by just installing modules and clicking through their UIs and, as soon you have to add a language, you have to revisit most of your site-building process, apply tens of patches, write lots of LOC and hope for the best.

That said, I think we should close this issue and open a new one, copy the issue summary and update it with any relevant change needed, because starting again with this after 300 comments will just not work.

plach’s picture


Gábor Hojtsy’s picture

@plach: well, the reality is that multilingual needs various things:

1. it needs a collection of all translatable things to be exported on
2. it needs a collection of all things translatable on a site (includes config that was not shipped with the software), so translation status can be calculated, translation can be QA-ed, etc., this is wider than 1.
3. it need translation UIs that make sense to each piece of configuration and is not contextual only (ie. serious sites will not allow people to go edit a view or edit account settings, so they find a translation tab there, we need to be able to generate translation experiences that are separate from the original forms)

This patch wanted to attack all three and it did an admittedly sub-par job at 3 with the intention to experiment with it and *expand* on it. Now reviewers actually found it already very extensive and dangerous for DX, so no possibility in sight to even expand on it later. (If you go to your site account settings or go edit a view's header text, you'll see lots of supporting UX, descriptions, input formats, tokens, etc. that this patch did not support but an *ideal* UI would). So Dries is right this patch does not further the cause in terms of actually providing a better UX. This was not supposed to be a be all and end all, it was supposed to be a patch to set up the base system which we can experiment and work with. Based on reviews, people don't want to do that :)

I don't think we can skip solving (1) for core in Drupal 8, given that most useful things will be in CMI and if Drupal 8 translators cannot translate shipped content type names, view titles, field labels, and so on, that is going to be a huge problem. Then you could try installing in a foreign language but never be successful with it :) Solving 2 and 3 in core looks like requires such complexity that whoever came in to review or think about it got their "head hurt" and neither did they support the solutions proposed or came up with better ones that others would have supported. The matter of fact is the problem does not go away, it is that it will need to be solved in contrib, and as you rightly pointed out, only those contribs who care about it will work with it (and will likely have a harder time doing it). There is only so much one can do in one core release.

It is also not like problems for CMI/multilingual are at an end in Drupal 8, we still nee to solve (1) and make config accessible in different languages: #1861640: Provide config metadata solely for translation and #1763640: Introduce config context to make original config and different overrides accessible. And then we have problems with the existing override system in #1859110: Circular dependencies and hidden bugs with configuration overrides. Those issues lack people collaborating as well :) Contributions welcome.

plach’s picture

I perfectly understand the scope and context of this patch, thanks, I read though all of its comments, albeit I didn't have the time to look at the code. I realize it's a really hard task, probably far more complex than the ones we have to face while working on the entity stuff: just the fact that we are dealing with arbitrarily complex data-structures instead of predictable ones makes me want to cry.

And I understand the use-cases we want to support here, but it seems to me that by rescoping this we are losing a formidable opportunity to solve in core a number of problems that in contrib will simply explode. And none will have the brainpower, the dedication or simply the time to fix them all, then.

This is why I'm talking about PM and not about solutions: we need to change our mindset here, IMHO this really should be a critical release blocker. No deadlines. We just need to do this. Personally I've not been able to help here because I've never been particularly involved with the CMI from the very beginning. I've tons of stuff that have a deadline to work on. But this one IMO is one of the most important changes in D8, and the fact that very few people actually worked on it is what makes me think we need a change in our mindset. If my help can be useful in any way, I'll just complete just the stuff that strictly needs to be done before feature completion on the entity side and then focus on this.

I'd be totally ok with having a UI providing both multilingual config forms and stripped-down translation forms in contrib (I have some experience with this approach ;). This is exactly what we are trying to achieve in #1807776: Support both simple and editorial workflows for translating entities, and having core support the same for config is a unique chance to unify or make more consistent at least our patterns if not our APIs. But the plumbing must be in place and it has to be complete.

Heck, we need this metadata stuff :)

effulgentsia’s picture

@plach: per my comment in #1861640-14: Provide config metadata solely for translation, I agree that extensible metadata, and not just translatable string identification, would be really awesome plumbing to have, and a sad missed opportunity to not have. That's why I'm happy this issue is still prioritized as critical, even if it's also postponed for the time being. I disagree about it being release blocking: it is a feature request after all, so IMO, should be subject to the Feb. 18th feature completion deadline. I think we can still solve it by then.

I think some more tightly scoped exploration in #1861640: Provide config metadata solely for translation is a good thing. I'm curious to see what patches show up there.

I continue to be interested in pursuing this full issue though. I agree with most of Jose's suggestions in #312 and his reason for postponing this issue, along with your suggestion in #315 to start new issues for doing the things Jose recommends in smaller chunks.

However, I disagree with Jose's claim that the TypedData integration should be abandoned. I'm currently delving into that to see if I can come up with a patch that demonstrates my understanding and reasonings, and maybe that will help me convince others why TypedData integration is worth continuing, or will help others convince me why it isn't. It'll take a little time for me to do that though, so if others in the meantime want to pursue some other ideas, please do. Some parallel explorations would not be a bad thing here: it'll get more people understanding the problem space from different perspectives, and then maybe we can converge on something that works for all those perspectives.

And #317 lists other very important related issues too, so there's no shortage of things that people can help on right now without feeling like everything is stuck on this issue.

fago’s picture

Coming back to this issue after a while.

I'm also not interested in having core automatically generate simple forms from the meta files; I don't want to introduce another form API, at least not in Drupal 8. It's too late for that.

I'm not sure it would be the right solution UX wise, but I would not agree this re-invents form API. Field API widgets do not re-invent form API either, it's a layer on top of it.

But given the issue summary states multiple good reasons for having the metadata, this does not seem to be the real problem here. I guess it's more the new file format:

I understand that this is important but I'm nervous about introducing a new custom file format that is pretty hard to grok. I still struggle with the fact that it only marginally improves the translator experience and that it is still an incomplete solution.

I'm not sure why we went away from a straight php2yaml typed data defintion + an easy way to nest things to mixing .type and .definition. Maybe, we could re-consider a slightly simplified approach that takes typed data definitions from php really 1:1 to yaml? That would mean it would be more work to integrated nested structures, as you'd have to expose a separate "type" for each level. But it would work and once we've more experience with nested structures we know how it could be simplified best.

1. it needs a collection of all translatable things to be exported on

After diving more into translations with validation violations translations I must say I'm not sure that our current approach of extracting translations just magically with potx is right. I'm not sure it's reasonable to think about alternatives that late for d8, but what about making that more explicit?
In addition to supporting potx for t(), we could support an optional file of to-be-translated strings. So if you need to pass something dynamically to t(), you can still register it there. The config system could embrance the metadata to determine translatable strings and write it to the file, once config gets updated. Yes this duplicates some strings, but is that really a problem?

Once we'd have that file written, we do not need a way to statically parse config schema any more. Thus we could embrace the regular typed data API way of adding metadata. Right now, this uses hook discovery, but once switched to class based discovery this should be reasonable simple - dx++

Given #1845546: Implement validation for the TypedData API I really think we should embrace the type data API for config - either from PHP or from yaml. In addition to the must have of config-string-translations, having config being properly validated *independently from form submissions would be a big plus to me.

effulgentsia’s picture

As promised in #319, I posted my initial work in #1865300: Introduce configuration definition file format and API for reading it. Early feedback there would be welcome and appreciated.

I'm not sure why we went away from a straight php2yaml typed data defintion

That issue brings us back to that.

Gábor Hojtsy’s picture

The config system could embrance the metadata to determine translatable strings and write it to the file, once config gets updated. Yes this duplicates some strings, but is that really a problem?

That is funny you are getting us back to there, and I've been discussing "going back to the Barcelona discussions" with heyrocker yesterday, and in fact this very idea (again) with chx as well. One of the options early on (eg. in the Barcelona discussions and earlier) indeed was to either have syntax *in the data yml file* that would identify the translatables, some creative thing like:

key: String
tobetranslated%: Some other string

tobetranslated: t(Some other string)

tobetranslated: Some other string #t

Where obviously %= or t() or #t (or any other kind of syntax) would be syntactical markers in the file to mark items to translate and would not end up as part of the value. Then another one of our ideas was to just do what Views does in Drupal 7 and export a list of translatables at the end of the file, something like:

key: String
tobetranslated: Some other string
thistoo: One more strings
 - Some other string
 - One more string

Both of these proposals were deemed against the simplicity goal of CMI (as well as making the files hardly hand-editable and result in magic/broken side effects). Obviously both of these would be limited to *the use case only* and would not let us display translation forms or figure out if an element needs an input format / text editor or should be validated as a number or email address.

It is hard to argue the resulting metadata system that was developed in light of the various times repeated arguments to *keeping the CMI simplicity* and thus not touching the actual data files might in fact be more complex, although encoding the identification of the translatable pieces in the runtime PHP code might not in fact be simpler, and even more importantly it would need the CMI storage system to know which items are translatable (or each implementation in the application layer to have custom logic for it), which also goes against the design goal of the metadata system that it would only be needed if you actually need to identify the translatables, and therefore not put a baggage on regular use of CMI.

So coupling the CMI requirements with the D8MI requirements, we need a system that does not affect the regular CMI runtime operations and/or complicate the simplicity of the CMI files (both requirements from the CMI team) but still let us statically identify the translatable pieces in the files (requirement from D8MI), which all result in a big fat pointer to have a declarative external description format.

Since we need this anyway, and its use or weight does not affect normal CMI operation, we wanted to couple it with a solution for the form generation problem which needs a similar description of the data that lets us generate translation editing forms. That needs a *lot* more information to generate useful forms, and the patch proposed here would not be enough to have useful forms (eg. for a simple example: Views header text which both has an input format associated (and could use WYSIWYG editors) and has tokens offered). So we would need to expand on the original format based on further experience. We cannot really solve everything right away, especially if a simpler format/system needed 300 comments to figure out.

plach’s picture

Can at least some power user delete the system messages so that we get under the 300 threshold?

alexpott’s picture

My 2 cents on this issue.

Q: Why don't people like this patch?
A: This is a complex problem.
A: People don't like the need for a schema and metadata format.
A: XML does it better than YAML.
A: TypedData has proved difficult to use.

So given that we need this complexity - and Gabor has outlined why many times...

Q: Do you we need a schema?
A: Yes. Why? Gabor's comment #322

Q: Do we need all the metadata keys or can this be simplified?
A: If we want to provide a sensible interface with which to translate the values then yes.

Q: XML vs YAML? And is this important?
A: I think not and I think XML will buy us more pain than it worth. And I'm not alone... see but we could swap this out at a later date anyway.

Q: Should we use TypedData?
A: Well we have a TypedData interface and this is typed data so we need a good reason why not to use it. So... Yes see @effulgentsia's work #1865300: Introduce configuration definition file format and API for reading it

So will #1861640: Provide config metadata solely for translation actually lead to a simpler patch? I think not.

The solution in this patch managed to maintain the simplicity of CMI interface. Even with this patch a module does not have to declare metadata and the essential config api is untouched.

A way forward?

  1. Get the metadata format implementation solved in #1865300: Introduce configuration definition file format and API for reading it
  2. Extract the config batch translation from this patch and implement in a new issue.
  3. Continue the context work in #1763640: Introduce config context to make original config and different overrides accessible


  1. Commit this patch
  2. Finish #1851498: Polish file format of configuration metadata (for translation) with input from #1865300: Introduce configuration definition file format and API for reading it
  3. Continue the context work in #1763640: Introduce config context to make original config and different overrides accessible
heyrocker’s picture

It has been interesting watching this issue go round and round for the last two years (no matter what we do I suspect it will circle back around again in the D9 cycle.) I was actually never actually 100% against including the metadata in the configuration files themselves, I just don't think we ever found a good way to do it. I remember talking this over a lot with various people in Barcelona a year ago. Other than that I largely agree with Gabor's summary of CMI's goals and priorities.

I have long been a major proponent of making the metadata files simpler from a DX perspective, because module devs are going to have to create them by hand. However, my concern has always been around the wildcard nesting, which is by far the most complicated piece of this. Since none of the counterproposals remove this need, I don't think we're gaining much simplicity out of them in the end. We are making the DX very slightly easier, but we are removing the potential for a very powerful new feature for D8MI that will offer users a greatly enhanced experience. That doesn't seem like sensible ROI to me.

I'm sort of torn on the TypedData question, I haven't reviewed effulgentsia's new patch and this is also the part of the current patch I am least familiar with. I do know that there is a huge ton of abstraction going on there and I'd be interested in what the patch would look like without it. I suspect it would be a lot easier to grok.

We are not switching back to XML. The issues around it are well documented and still wouldn't have solutions, and we absolutely do not have to time to re-re-convert everything. Everyone stop talking about it.

So in summary I guess I largely agree with what alex and gabor are saying above, with the caveat that I'm more open on the TypedData question and need to research it a bit more to understand the pros/cons.

Jose Reyero’s picture

Surprisingly, there's some wider and stronger support for the "we need schema/metadata" idea than I thought before :-)

However, I have to agree current patch and specially the schema format is not beautiful, not to say it's ugly, so I can understand @Dries concerns about it. But also, on the other follow up issues, I don't think it is looking much better and I'm really concerned that after one more month we get stuck at the same point... :-(

So this is what I think: Let's find a good powerful real one. The bad news though is that it's not going to be too simple because we do need to describe complex data. But really I'd rather settle for something powerful than for something simple. So if it's not going to be simple, let's make it powerful!!

And this is (yet one more) schema format based on Kwalify (which is the only viable option I've found out of here, Rx looking too limited to me), and extended to meet Drupal needs.
Also I have to step back from my previous statements about TypedData not being any help here, though I tried to build without it, ended up (once again) reusing TypedDataInterface (thought not the whole thing).

So this builds on both TypedData and Kwalify and really, the schema itself is contained in the schema, so it at least starts simple... :-)

This is it #1866610: Introduce Kwalify-inspired schema format for configuration

Gábor Hojtsy’s picture

All right, we have three options now with three people working in parallel on different schema approaches. If that does not say there is an agreement we need some kind of schema, I don't know what would :)

  1. Using kwalify and typed data: #1866610: Introduce Kwalify-inspired schema format for configuration
  2. Using typed data only: #1865300: Introduce configuration definition file format and API for reading it
  3. Not even using typed data: #1861640: Provide config metadata solely for translation

Follow these and provide review feedback please as appropriate! (The last one has the representative priority of the task, given that Dries postponed this, so not all three show up as critical).

fago’s picture

So coupling the CMI requirements with the D8MI requirements, we need a system that does not affect the regular CMI runtime operations and/or complicate the simplicity of the CMI files (both requirements from the CMI team) but still let us statically identify the translatable pieces in the files (requirement from D8MI), which all result in a big fat pointer to have a declarative external description format.

Yes, but if we export all translatable strings to a separate file on save we do not need any extern description format - thus that would save us from inventing a new file format! On the PHP side we could just re-use what we have (typed data) and make sure that's simple. Writing that separate translation file could happen on config-entity save, for non-entity config a simple helper could do it. Wouldn't that simplify things and make it easier to grok what can be translated? Just look up the file?

Thus, we could keep it out from CMI also. Yes, if you are editing the config file from hand the translation file would become out of sync. But honestly, I don't see a big problem with that. The manual config update would need a CMI reload anyway, what probably a drush command or so would take care of. It can take care of updating the translations file also.

Gábor Hojtsy’s picture

@fago: my understanding was that there is resistance in the CMI team to tie this into the storage layer, like you suggest, but for D8MI, exporting the items on config save (and re-export after hand edit) works as well. That would mean there is a dynamic description layer of the data somehow and nesting of things like image effects with styles is somehow resolved by either plugins or the application layer. Currently the base CMI system attempts to limit itself to being a generic key-value store, your suggestion would essentially dress up the storage layer with types, nesting, inheritance and other metadata pieces in my understanding.

Gábor Hojtsy’s picture

Had a fruitful discussion with @fago about this in IRC which looks like resulted in a very good argument against a declarative metadata format. Think of the inclusions and data wrapping that we want to model properly. As @yched pointed out, parts of config cannot be sure how they are wrapped in other config, while the wrapper cannot be sure what it wraps. The resulting solution of cross-referencing CMI meta-definition files solves the issue however it means that the metadata can only be evaluated with the context of all dependent modules involved (some of which might be contrib), and there is also a versioning component where this metadata can change, so even if could pull in the metadata from the dependent modules (downloading and considering those as well), it would not know the version of the module/metadata to use for sure. In short the declarative metadata does not in fact solve the identification problem of the strings because of cross-dependencies and versioning issues. This sounds like a very good argument for actually running the metadata processing in the Drupal runtime and statically store the list of strings identified in files, which then entirely nullifies the need for the metadata itself to be declarative/static. It does put baggage on all CMI save operations though, so direct feedback from CMI leads would be important.

(Clarified: it does not matter if the metadata is static or not, because we need to evaluate it in the context of the .yml file being created on the actual Drupal site. If we don't like static metadata, we can move it into the runtime.)

The full discussion:

[5:09pm] fago: GaborHojtsy, hey! What do you think about exporting translatable strings in a separate file?
[5:09pm] GaborHojtsy: fago: *I* would absolutely be fine with that it makes the identification of them way simpler for multilingual
[5:09pm] GaborHojtsy: fago: my understanding was it was an absolute no-go from CMI
[5:09pm] GaborHojtsy: fago: pinged alexpott and heyrocker about it
[5:10pm] fago: GaborHojtsy, but we could do that on top of CMI, couldn't we?
[5:10pm] GaborHojtsy: fago: it would need the full type, nesting, etc. understanding in CMI when the data is saved
[5:11pm] fago: GaborHojtsy, well, assuming we'd have type data metadata we could use that to find translatable strings and export them to a file - beside of saving the config to CMI. That way, cmi would not have to know?
[5:11pm] GaborHojtsy: fago: well, anytime the data is saved, the data structure would need to be figured out, and it would need to be part of the base system
[5:12pm] GaborHojtsy: fago: eg. if you create drupal commerce, a distro in English, it would still need to have these extracted strings
[5:12pm] GaborHojtsy: fago: so it would need to maintain the functionality on an English only site
[5:12pm] GaborHojtsy: fago: which is effectively adding a type/nesting/inheritance discovery and application step on all config saved
[5:12pm] GaborHojtsy: saves
[5:12pm] GaborHojtsy: fago: on all sites
[5:13pm] fago: GaborHojtsy, but if it's a separate operation we could simply skip that on english sites?
[5:13pm] fago: GaborHojtsy, ah I see your point is we need that even for English sites.
[5:15pm] fago: GaborHojtsy, I guess it could be functionality that you can turn on somehow, so it could be disabled for English sites by default. But if you do a distro and want it, you'd have to enable it
[5:15pm] GaborHojtsy: fago: so even if it is not in CMI per say, it would need to run all the time
[5:15pm] GaborHojtsy: fago: meh
[5:15pm] fago: GaborHojtsy, yes
[5:15pm] GaborHojtsy: fago: so it would then be on users to submit an issue for each module "can you please export your translatable strings in this module"
[5:15pm] fago: GaborHojtsy, writing configuration is not performance critical, who cares whether it writes one or two files?
[5:15pm] GaborHojtsy: fago: because the module maintainer included 2 views exports, but no exported strings
[5:16pm] GaborHojtsy: fago: *i* don't care
[5:16pm] fago: GaborHojtsy, true, that sucks. So it would have to be always on - yep
[5:16pm] GaborHojtsy: fago: I care for finding a solution within the constraints, and I took this as a constraint
[5:17pm] GaborHojtsy: fago: if CMI folks say this is not a constraint, then we opened up for new opportunities
[5:17pm] fago: GaborHojtsy, yes, but it make things a lot harder. So I'm not sure it's the right approach.
[5:17pm] GaborHojtsy: fago: yeah, well
[5:18pm] GaborHojtsy: fago: we have seen where we can get here
[5:18pm] GaborHojtsy: fago: for some reason neither chx, not effulgentsia nor reyero broke out of this assumption, so I think we need feedback from "up"
[5:18pm] GaborHojtsy: as in CMI die-hards
[5:18pm] GaborHojtsy: alexpott might be reading us here
[5:18pm] fago: GaborHojtsy, you know for the translation includes to work out, you'd have to statically find other modules to source the right include? That sounds like a nightmare to do with multiple drupal versions, module versions, etc. I'd even say it's impossible due to versioning issues.
[5:19pm] GaborHojtsy: fago: well, we assume config structure is according to the current module descriptions
[5:19pm] alexpott: GaborHojtsy fago: I'm reading… making mental notes to think about later :)… I'm unfortunately busy.
[5:19pm] GaborHojtsy: fago: if it is defined via code in modules, it will be the same situation
[5:20pm] GaborHojtsy: fago: I assume when modules are updated, the exported strings would need to be regenerated, because they might be affected
[5:20pm] GaborHojtsy: fago: or at least when module updates cause CMI updates
[5:20pm] fago: GaborHojtsy, no it's not, because does not know the module versions you were running when exporting the config.
[5:20pm] GaborHojtsy: fago: yeah well THAT is a very good point, it will not be able to take the other contrib modules possibility involved and merge their metadata
[5:21pm] GaborHojtsy: damn
[5:21pm] fago: GaborHojtsy, yeah, but this is what it would have to do for "includes" to work out
[5:21pm] fago: -> better do it in PHP
[5:22pm] GaborHojtsy: fago: eh
[5:22pm] GaborHojtsy: #facepalm
[5:22pm] • GaborHojtsy bangs head to desk
[5:22pm] GaborHojtsy: yeah
[5:22pm] fago: GaborHojtsy, doing it in PHP would be reliable. you'D have a simple file of translatable strings. you can look into it to verify it has found something, so you know it's translatable. no magic involved -> dx++
[5:23pm] GaborHojtsy: fago: yeah I know
[5:23pm] GaborHojtsy: fago: it definitely seems to go against the CMI requirement of keeping the base functionality simple
[5:23pm] GaborHojtsy: fago: however, the inclusionn problem, well, I don't know how I missed that
[5:24pm] fago: GaborHojtsy, keeping which base functionality simple? I think this "simplicity" requirement creates lots of complexity, so it's wrong.
[5:24pm] GaborHojtsy: fago: yeah, well

Jose Reyero’s picture

@fago, @Gábor Hojtsy,
Yeah, ok, so the translatable strings need to be identified by the runtime system exporting the configuration.

However, as I see it, it only shifts requirements a little bit for the schema system. We still need need a way to identify that translatable strings / elements. And it may be the schema very well.

The option, as I understand is "everything in PHP", for which I fail to see how introducing yet another hook is better DX than adding a yml file. Actually each complex hook we add is one more "file format" to learn.

Still, please, don't dare tell me the 'pure' PHP solution is better until such solution exists.

"Existence" is a pre-requisite for "being better".

yched’s picture

I don't think I grasp why is a problem with the "declarative metadata files" approach ?
How is involved here ? Could you clarify that ? (sorry to ask, I'm not too familiar with that area)

Jose Reyero’s picture

@yched, uses pre-extracted lists of strings and it gets them from the module's code using potx extractor. That's done on a package by package basis.
Now if module A depends on module B and A contains configuration for a plugin provided by B that contains translatable strings we need both modules (with the right versions) to be able to tell which are the strings.
So though theoretically possible (you would need to checkout all the dependencies with the right versions for generating the strings for each package), it looks like a hardly workable approach.

heyrocker’s picture

my understanding was that there is resistance in the CMI team to tie this into the storage layer

I actually don't remember expressing this, although it is possible I did back around Barcelona, when I was fighting against the entire concept of metadata to begin with. However we've come a long way since then, and I would be willing to consider all options that can move us forward at this point. Yesterday I talked some with effulgentsia, xjm, beejeebus, and chx about this. An idea I've been toying with is defining the metadata in PHP (either in annotations, or in TypedData, or whatever) and then actually writing that data out to metadata files on import/install/update. We would need a way for new plugins being installed to trigger a rewrite of the metdata they affect (or possibly just all of it, this is a rare operation and not really a choke point.) This could then get to a world where we have one metadata file per config file. If a plugin changes, you trigger a rewrite and it all gets redone. No more wildcard nesting.

While it does add metadata-specific code into the cmi storage engines, I think the tradeoff could be worth it in terms of DX. I don't have a good picture in my head of how the metadata would look or be declared or anything, as I said before my knowledge of a lot of these new subsystems is pretty limited. Note this is just based on like a day of thoughts rattling around in my head, so there's likely lots of things I'm not thinking through. I'm going to try and find time this weekend to see if I can't work through a proof of concept of some sort.

yched’s picture

@reyero: thanks, makes sense.

Agreed that regenerating the metadata tree associated to a given config item without being on the system that generated the config item itself sounds like asking for trouble. Generating the associated metadata while writing the config file itself does seem much more robust.

Gábor Hojtsy’s picture

All right, so seems like we'd either need to compile a CMI .yml-file specific metadata file in the Drupal runtime (in storage) or actually execute that metadata on the data and extract the translatable strings. My understanding is @heyrocker is pursuing the former route and will hopefully come back with a patch.

Jose Reyero’s picture

While this may mean some workflow change, we still have the need to identify these translatable strings in the configuration for which having schema data is still a solution. Actually there's no other solution anywhere so far.

So I really hope people advocating "other better solutions" are busy working on them. Otherwise I'm afraid Drupal 8 will ship without configuration strings being translatable at all.

yched’s picture

Hm. Possible issue with generating metadata files on write of the config item : config items written during hook_update_N.
We're not supposed to call api funcs in there, just work on raw data. config() read & write have been deemed ok so far (they are the equivalent of the direct db read & write on config tables in D7), but the mechanisms to fetch metadata snippets from various mudules / plugins are probably not kosher in hook_update.

yched’s picture

#338 is the "on config write" part of the problem.
Not strictly part of this exact issue, but we probably have another problem on config read: if the lowest level API we have, config()->get(), already bakes in switching values to the ones assigned to the current language, then how do we access the very-very-raw data in hook_update_N ?
hook_update_N is the place where you'd modify the values or structure of existing config items.
It needs to be able to do so on the raw, untranslated values ?

Gábor Hojtsy’s picture

@yched: the loading of originals problem had a solution in previous patches in #1763640: Introduce config context to make original config and different overrides accessible, although beejeebus did not cover that in his current proposals. We hoped to fix it via the same mechanism, where you say what kind of override you want to have working or ignored.

yched’s picture

@Gabor: Ack.

So, "just" the issue that we're not sure we'll be able to compute metadata trees for config() writes in hook_update_N()...

Gábor Hojtsy’s picture

@yched: @beejeebus made sure we have an issue to track (the raw data inaccessibility bug) at #1868028: Raw (original) config data is not accessible.

fago’s picture

So, "just" the issue that we're not sure we'll be able to compute metadata trees for config() writes in hook_update_N()...

We could have an upper-level, metadata enabled config writing routine which takes advantage of the metadata to update translation also + an lower level config writing routine that writes to the config files only. Writing the translation file should have a similar lower level API then, such that the update functions could leverage both lower level APIs without depending on any metadata or system state.

I'd assume the lower level config API is what CMI is now, but I'm not sure how the upper-level API could like. We could do it as part of writing config-entities, but that won't cut it as we need to take care of single-config forms also. We could add it as another function call there.

Or, as an alternative we could add TranslationFileManager in Config and invoke it from save(), but move it to an injectable dependency. Then, in update functions we inject a null TranslationFileManager and care about translation file writing on our own (if necessary)?

Gábor Hojtsy’s picture

Gosh, if only there would be a way to get people review the forks of metadata formats, and somehow narrow the extent that we get people occupied here (and make some progress):

  1. Using kwalify and typed data: #1866610: Introduce Kwalify-inspired schema format for configuration
  2. Using typed data only: #1865300: Introduce configuration definition file format and API for reading it
  3. Using typed data as well (see patches above)
  4. Not even using typed data: #1861640: Provide config metadata solely for translation

So my understanding is that @heyrocker (and maybe @fago?) is advocating possibly not sourcing the metadata from .yml files but generating them out into .yml files, so there would be two metadata formats, one in annotations and PHP and one in .yml files, and the .yml would be generated. *The two largely different metadata formats are proposed to improve DX*. Sounds like the generated metadata format needs a system as well (one of the above?).

Everybody else seems to assume the metadata would be sourced from .yml files (see @effulgentsia's recent comment at There is still some uncertainty there that maybe their inclusion/merging rules would be sourced from PHP somehow (see @effulgentsia again).

Then there is the above patch, the kwalify patch and @chx's patch where the inclusion is assumed to be happening in the metadata .yml level.

Are we clarifying anything?! (I'm personally amazed that this many people care so much yet cannot get to a common ground).

Gábor Hojtsy’s picture

Category: feature » task
Issue tags: +regression

Refiling as a task. While we might not agree yet on the implementation path, it would be a serious regression if this is not implemented. "Hey, we added views! But, well, you cannot translate views to other languages. Well, you could do that in Drupal 7? Yes, not anymore." is not a discussion point we'd want to use.

#1861640: Provide config metadata solely for translation was revoked, so there are two (or three) proposals for metadata formats (see above), depending on whether we count the patches above in this issue or only the two remaining forks. There are no issues or alternate proposals for anything against the above patch but the format and how the metadata files are located and discovered.

heyrocker’s picture

I would like to establish a deadline for deciding how we're going to proceed with this. If we don't, we are going to go back and forth on these decisions in a circle until feature freeze comes and then we're screwed. My feeling is we will probably need about a month to implement any new proposal and get it to RTBC, so I'd like to lay down January 15 as our deadline or deciding what to do. If we haven't agreed on any new proposal, then we reroll the original RTBC patch from #285 and get it in. We have to get this done, and right now I don't see any light at the end of the tunnel which scares the crap out of me.

So there is just under a month to work out the new proposals and establish consensus. Lets do it or admit its not getting done and go with what we've got.

Gábor Hojtsy’s picture

@heyrocker: sounds like a good plan! Looking forward to continued feedback and patches :)

svettes’s picture

Fabulous plan @heyrocker, @gabor seems on board too -- but no movement since :) Hols are keeping everyone busy I think! Anyone care to chime in?
Happy New Year everybodeh!

Gábor Hojtsy’s picture

Posted this call for feedback on the Core group (also going to Drupal Planet):

Gábor Hojtsy’s picture

Update! The action is now in #1866610: Introduce Kwalify-inspired schema format for configuration.

@chx revoked #1861640: Provide config metadata solely for translation in favor of @effulgentsia's approach and @effulgentsia revoked #1865300: Introduce configuration definition file format and API for reading it in favor of @reyero's proposal in #1866610: Introduce Kwalify-inspired schema format for configuration.

It is the most verbose format of all of the proposals, however it is based on (and extends) a pre-existing format.

Although we now have a "replacement" issue for this, I'm NOT marking this a duplicate, since there are a *lot* of integration pieces in the proposed patches in this issue that are not covered by just introducing the format and will need extraction and rerolling to integrate with the other format.

Gábor Hojtsy’s picture

Issue summary: View changes

fix typo

heyrocker’s picture

So here we are at January 15. After several attempts and some long conversations with people I haven't been able to find a way to make the declarative method work, so I guess it is tabled.

In the meantime, we are waiting for feedback from Dries on the Kwalify file format. Both Gabor and I agree that we want to make sure Dries is on board with the plan before proceeding, as we don't want to spend a bunch of time unnecessarily with so little time to go. Hopefully that will happen this week. If not we will re-evaluate again on Friday.

Gábor Hojtsy’s picture

Need reviews on the kwalify-inspired schema that is the only proposal now and is supported by Dries: (ignore comment #40, that was an interdiff).

Gábor Hojtsy’s picture

#1866610: Introduce Kwalify-inspired schema format for configuration just got committed. Woot! Instead of reopening this for the locale integration (that we need to resurrect from the latest patches here), I think opening a new issue for that would be beneficial. I'll look into that tomorrow if nobody beats me to it :)

Gábor Hojtsy’s picture

Assigned: Dries » Unassigned
Status: Postponed » Closed (duplicate)

All right, opened #1905152: Integrate config schema with locale, so shipped configuration is translated to resurrect all the missing functionality that was not added in #1866610: Introduce Kwalify-inspired schema format for configuration but was part of the above patches. Since that functionality had no debates in this issue either, I think its fine to leave these 320+ comments behind.

That said, marking duplicate of #1866610: Introduce Kwalify-inspired schema format for configuration.

Gábor Hojtsy’s picture

Gábor Hojtsy’s picture

Also opened #1952394: Add configuration translation user interface module in core just for the heck of it, given that Dries opened the possibility of features getting in later ( Although given the number of critical/major bugs and Dries' negative reviews above on even the concept, I don't have a lot of hope for it to land in core for Drupal 8. (The contrib module is progressing nicely btw).

yched’s picture

Hand waving for #1975112: Config schema mechanisms cannot reflect alterable config trees - "the elephant has officially (re-)entered the room"

Gábor Hojtsy’s picture

Re #330 - the versioning problem, I just opened #1977498: Add version tracking for configuration schema and data where we can discuss that now that the locale module integration landed. @fago, et. al. your feedback is very welcome there. We need to resolve that to support module updates in Drupal 8 going forward as well as to make it possible for to figure out data/schema versions as they relate to each other.

Gábor Hojtsy’s picture

Issue summary: View changes

Updated issue summary.

Leeteq’s picture