commit message (message needs work. Not sure if we are at the point for suggesting one yet.):
Issue #1648930 by Jose Reyero, YesCT, tstoeckler, spearhead93, alexpott, yched, Gábor Hojtsy, fago, effulgentsia, dawehner: Introduce configuration metadata and use it for translation.

Problem/Motivation

As a site builder I want Drupal core (and module/theme) shipped configuration, such as default views, email text settings, the site maintenance message, the default contact form, and so on, translatable just as the software it ships with. Also I want to let translators who might not have access to change user email text, views or contact form settings to be able to modify and update the translations as needed.

With the Drupal 8 configuration system, multilingual sites loose two key features which need to be restored and the ultimate goal in Drupal 8 is to make configuration generally translatable. All three require metadata, about the configuration stored, to be properly restored.

  1. When default configuration is saved into the database, the code setting up the default configuration used to use t() on the strings that were translatable. So configuration was saved initially in the site's default language. (Think content types, forums vocabulary name, etc.) This is lost with Drupal 8 because we have no information as to which parts of configuration are to be translated or are translatable.
  2. For variable_set()/get() based settings, unmodified menu items and many other things in Drupal 7, unless you modify the original values, translation updates are applied to your configuration translation. This lets you get translation updates for shipped configuration so long as you don't customize it. The configuration system does not currently have granular modification tracking and again no translatability metadata.
  3. Finally, other pieces of configuration have one-off translation implementations but should be generalized. For configuration translation, core only has two implementations: path alias language support and date format language support. Both have highly custom code and highly custom user interfaces to operate. First to reproduce these (at least date formats as ported to Drupal 8's configuration system), we need metadata about translatability. Also, we want to generalize this feature to other types of configuration. We cannot say Drupal is a multilingual CMS in 2013 onwards when it cannot even translate the site name you enter!

Proposed solution

Design goals

The following design goals have been set out for this system:

  1. Separate metadata from the data itself to avoid repetition and keep .yml configuration files simple.
  2. Avoid repetition in the metadata files themselves and allow for representation of metadata for pluggable systems (image effects, views, etc).
  3. The metadata must not be needed for regular pages (that just retrieve configuration values).

Metadata file format and placement

Introduce configuration metadata with basic information about each element in the configuration structure. Using YML files for the description of the metainformation, we can use the same format as applied to configuration itself.
This metadata will match configuration data with data types defined by the TypedData API so a) we'll be able to access configuration data as typed data and b) We can identify data types in the configuration data that are to be localized.
Metainformation is proposed to be placed in a sibling directory of the shipped configuration files using the same machine name. For example, core/modules/system/config/system.maintenance.yml looks like this:

enabled: '0'
message: @site is currently under maintenance. We should be back shortly. Thank you for your patience.

The metafile for this YML is at core/modules/system/meta/system.maintenance.yml (note the meta subdirectory in place of config):

enabled:
   .label: 'Put site into maintenance mode'
   .type: boolean
message:
  .label: 'Message to display when in maintenance mode'
  .type: text

The metadata file follows the same structure as the original YML file but uses keywords prefixed with . (dot) to add more information to the data. This is to differentiate the child elements from the metadata elements, it serves the same purpose as # in Form API to differentiate properties from children. The keywords used here are (two groups):

Keyword group I. data type definition

For the data type definition of the element as specified by the TypedData API:

- .label: contains the 'label' for the data type definition for the element.
- .type: contains the 'type' for the data type definition for the element.
- .definition: contains an array with any other keys of the data type definition for the element

** While we could stick all the data type definition into the '.definition' array, more than label and type is barely needed. More definitions (than label and type) is just for a) rare cases when we need to use the full TypedData API properties and b) also for future extensions allowing contributed modules to define their own TypedData data types which may need more complex definitions. Having 'label' and 'type' will be enough for almost all cases. Leaving .definition out of the file for those makes the metadata files more readable and easier to write (they save one level of indentation). For a side by side comparison of how this would look using only the '.definition' array see more discussion in the follow-up #1851498-7: Polish file format of configuration metadata (for translation)

keyword group II. Nesting and Includes

For defining the structure of the configuration data matching nested elements / plugins in the configuration data with their metadata:

- .include: Name of other configuration metadata element to be included (and merged with the current metadata). This allows reusing metadata from components for different containers. An example is the 'image.style.%' metadata using multiple instances of 'image effects'.
- .list: Contains an array of metadata to be reused for all of the elements in a list. An example is the user.mail metadata that defines mail text and body once to be used for all the emails in the user.mail configuration file.

Metadata applied to multiple configuration files (% placeholder)

When metadata need to apply to multiple configuration files, the metadata file name should use a % placeholder where variable values are possible. For example, contact form categories are in files contact.category.{category name}.yml, such as contact.category.webmaster.yml, contact.category.sales.yml, but they follow the same structure, so the metadata should apply to all of them. A % placeholder is used (same as with tpl.php template file wildcards) to account for this, so core/modules/contact/meta/contact.category.%.yml applies to all contact categories. The structure of the metadata file is the same as above, it just applies to a set of configuration files.

Metadata for nested data structures

In many cases configuration contains nested data structures provided by plugins or other sources where the nested data structure in itself has a pattern and need configuration metadata. In this case, nested levels of metadata can be provided by referring to nesting patterns. Take image effects. A sample configuration file for that (core/modules/image/config/image.style.medium.yml) looks like the following:

name: medium
label: Medium (220x220)
effects:
  bddf0d06-42f9-4c75-a700-a33cafa25ea0:
    name: image_scale
    data:
      width: '220'
      height: '220'
      upscale: '1'
    weight: '0'
    ieid: bddf0d06-42f9-4c75-a700-a33cafa25ea0

Note the main properties name and label as well as the property "effects" which embeds configuration that has its own structure, name, etc. A corresponding metadata file for image styles uses the above explained wildcard matching for all image effects core/modules/image/meta/image.style.%.yml:

name:
  .label: 'Machine name'
label:
  .label: 'Label'
  .type: text
effects:
  .label: 'Style effects'
  .list:
    .label: 'Image effect'
    weight:
      .label: 'Weight'
      .type: integer
    ieid:
      .label: 'IEID'
    name:
      .label: 'Machine name'
    data:
      .include: 'image.effect.[ %parent.name]'
      .label: 'Data'

This defines metadata for name and label as usual and then defines 'effects' as a list of nested objects which under the '.list' key have metainformation to be reused for all the effects in the list, that are indexed in the configuration data by UUID.
The first part of this (.list) metadata are the common properties for all image effects that are dependent on the container object (image style), thus they belong to the container, not to the plugins themselves.

The second part, inside 'data' is an include that will include for each of the image effects the metadata defined by 'image.effect.[ %parent.name ]', [ %parent.name ] being the value of 'name' property from each of the nested objects in the configuration data. ( %parent refers to the parent element of the one the include is in. So this refers to the parent elment's 'name' property)

In the example, this include will add the metadata from 'image.effect.image_scale' (name = image_scale).

Note that for each of the nested effects we are merging two arrays of metadata to get the final result:
1. Metadata for image effects specific of image styles (weight, uuid), this is defined in image.style.%.yml
2. Specific metadata for a given effect, in this case 'image_scale', defined in image.effect.image_scale.yml

While the first one (1) is specific of image styles, the other (2) can be reused for image effects in the case they are used by a different object (that is not an image style).
This accounts for the case of a component (typically a plugin) being used in several, independent systems it has no knowledge of. Example: the configuration for a field formatter can be included in a view, a block, a regular entity display... The metadata provided by the field formatter about its own settings is destination-agnostic and can thus be included by all these independant systems.

The image_scale specific metadata file explains width, height and upscale properties from the image_scale specific embedded structure:

Metadata in core/modules/image/meta/image.effect.image_scale.yml

width:
  .label: 'Width'
  .type: integer
height:
  .label: 'Height'
  .type: integer
upscale:
  .label: 'Upscale'
  .type: boolean

These two metadata files merged serve as description of this nested object.

Implementation details

The metadata system is based on existing subsystems implemented by the TypedData API and the configuration APIs. The .yml files are read and handled by the existing configuration API readers, and the data types are defined and handled using TypedData. A new config_element type is defined to handle elements that have structured data (lists or have children).

The metadata objects are built recursively so the system supports any level of nested objects that contain other nested properties. The system is extensible; modules can define their own TypedData configuration wrappers. However we may not need to extend it since the complexities required by Views are already supported.

Lets use site name as an example.
Files related to this example:
core/modules/system/config/system.site.yml
contains in part:

name: Drupal

and core/modules/system/meta/system.site.yml
contains in part:

name:
  .label: 'Site name'
  .type: text

To get type wrapped data based on a config element:

// Get a TypedData wrapper for this configuration. This returns a Drupal\Core\Config\Metadata\TypedConfig object.
$wrapper = config_wrapper('system.site');

// This returns not a string but a Drupal\Core\TypedData\Type\String object.
$name_property = $wrapper->get('name');

$name_property->getType() // will return 'text'.
$name_property->getValue() // will return the site name ('Drupal' by default).
$name_property->getElementLabel() // will return 'Site name'. Not part of this patch. See: ...

Locale module extends the TypedConfig by providing a LocaleTypedConfig that implements TranslatableInterface, actually allowing to get translated configuration data using the localization system. This is used to build the language-specific override data that will be loaded on top of each configuration object.

// Wrap system site information with Locale config accessors.
$data = config('system.site')->get();
$wrapper = new LocaleTypedConfig('system.site', $data, locale_storage());
// Get a config wrapper that works with the Spanish overriden version of this config.
$translation = $wrapper->getTranslation('es');
// Get a a nested array with the translated strings.
$strings = $translation->getData();

For a key piece of the patch, see locale_config_update_multiple() where the LocaleTypedConfig is used to retrieve translations of pieces of configuration from the Locale system and saved as overrides with the configuration system for later retrieval.

The full class/interface structure of the patch is as follows:

Some great side benefits

Though the primary use of this is getting the default imported configuration translated, it can also be used for building a data browser for any configuration object and also for auto generating settings forms.
The data browser is demonstrated by an example module, which also provides metadata for Views, not yet included in the patch, see:
- Config Demo module (browse or translate any configuration), http://drupal.org/sandbox/reyero/1635230
- Views configuration, generated by the demo module out of the metadata, screenshot, https://drupal.org/files/views_config_metadata_example.png

Translation API

The code rebuilds configuration translations every time string translations are updated (or new modules/themes are added). Using the string location data structure for source strings, we can track which strings belong to each configuration object so we only need to update affected ones when updating the translation.

Translatable data types are identified using hook_data_type_info(), see 'text' in system_data_type_info() in the patch. This data type is meant to represent 'human readable (aka translatable) text'. There are two small additions to the definition of these elements that can be added to the '.definition' array of the metadata if needed:
- 'translatable' => FALSE, to opt out from translation.
- 'locale context', to define locale context for translation,

In short, a configuration value is translatable (by core locale module) if:

  • It is a 'text' data type, and doesn't have a 'translatable' => FALSE flag
  • It remains with its default value from initial module installation.

There is no support yet for translating these strings once customized, since the localization system doesn't support user defined strings nor translating from source languages other than English. Module default configuration files, the ones provided with the module, are assumed to be English language unless they contain a different high level 'language' property with a different language code.

Test drive the patch!

An example of using these with specific details in comment #198

  1. Apply the patch, install a fresh Drupal 8
  2. Under Extend, Enable the Locale module (Language module is a dependency so it gets enabled too.)
  3. Under Configuration » Regional and languages » Languages add a language
  4. Go to the string translation page at Configuration » Regional and languages » User interface translation
  5. Search for strings like "Medium" (should find Medium (220x220)) or "Account cancellation request" (should find the email subject). If it does not find that souce string, might have to visit the page at Configuration » Media » Image styles and click on Medium (220x220).
  6. Translate either the image style name or the email subject. For example translate Medium (220x220) to Middle (220x220)
  7. Observe change applied by .yml files in active storage:
    • in the ui Configuration » Media » Image styles (switch to the language you translated it for, for example: es/admin/config/media/image-styles)
    • appearing with the translation overrides in Configuration » Regional and languages » User interface translation (admin/config/regional/translate)
    • in the active storage in sites/default/files/config_XXXXXXXXXXXXXXXXXXXXXXX/active/locale.config.yy.image.style
    • similar data structures appearing in the database as config overrides. [searching in phpmyadmin for Middle]

Remaining Tasks

  • (done)#250 7.b make this issue summary exactly accurate to match the most recent patch
  • (done) verify the test drive the patch instructions are accurate.
  • (done) make follow-ups for namespace docs style only file changes that were removed from the patch

Follow-ups

Unless critical to getting this in before feature freeze please do not start new discussions on this issue.
Please open a follow-up for items related to this issue. List them here.

Code standard follow-ups:

CommentFileSizeAuthor
#298 config_metadata-1648930-298.patch94.64 KBJose Reyero
#284 config_metadata-1648930.252-284.txt1.08 KBpenyaskito
#284 config_metadata-1648930-284.patch89.82 KBpenyaskito
#252 config_metadata-1648930-252.patch89.82 KBJose Reyero
#252 config_metadata-1648930-252-diff.txt4.64 KBJose Reyero
#247 config_metadata-1648930-247.patch89.85 KBYesCT
#247 interdiff-244-247.txt52.68 KBYesCT
#246 New CMI Metadata architecture (2).png98.57 KBGábor Hojtsy
#244 config_metadata-1648930-244.patch92.2 KBJose Reyero
#244 config_metadata-1648930-244-diff.txt16.93 KBJose Reyero
#236 config_metadata-1648930-236-diff.txt12.9 KBJose Reyero
#236 config_metadata-1648930-236.patch94.64 KBJose Reyero
#227 config_metadata-1648930-228.patch93.5 KBJose Reyero
#227 config_metadata-1648930-228-diff.txt11.64 KBJose Reyero
#226 config_metadata-1648930-226.patch90.79 KBJose Reyero
#226 config_metadata-1648930-226-diff.txt23.49 KBJose Reyero
#214 1648930_214.patch89.08 KBchx
#214 interdiff.txt5.49 KBchx
#206 interdiff-chx_E_STRICT_currenttextarea-194-206.txt772 bytesYesCT
#206 interdiff-justDocs-194-206.txt11.98 KBYesCT
#206 config_metadata-1648930-206.patch88.55 KBYesCT
#204 config_metadata-1648930-204-diff.txt996 bytesJose Reyero
#200 notes-1648930-194.txt11.36 KBYesCT
#198 configmeta-s01-dbsearch_a-2012-11-16_0753.png154.3 KBYesCT
#198 configmeta-s02-dbsearch_b-2012-11-16_0756.png156.03 KBYesCT
#194 1648930_194.patch87.24 KBchx
#194 interdiff.txt665 byteschx
#193 1648930_191.patch87.26 KBchx
#189 1648930_189.patch86.9 KBchx
#189 interdiff.txt29.55 KBchx
#187 config_metadata-1648930-187.patch81.21 KBJose Reyero
#187 config_metadata-1648930-178-187-diff.txt47.19 KBJose Reyero
#179 1648390_178-diff.txt7.06 KBJose Reyero
#178 1648390_178.patch81.78 KBchx
#154 interdiff-meta-1.txt5.16 KBalexpott
#154 evasive-interdiff.txt12.74 KBalexpott
#154 config_metadata-1648930-154.patch79.53 KBalexpott
#139 New CMI Metadata architecture (1).png93.23 KBGábor Hojtsy
#138 New CMI Metadata architecture.png97.57 KBGábor Hojtsy
#136 config_metadata-1648930-136.patch77 KBJose Reyero
#136 config_metadata-1648930-136-interdiff-119.txt83.28 KBJose Reyero
#133 config_metadata-1648930-133.patch94.53 KBJose Reyero
#133 config_metadata-1648930-133-interdiff.txt56.71 KBJose Reyero
#131 CMI Metadata architecture.png83.9 KBGábor Hojtsy
#127 config_metadata-1648930-127.patch96.76 KBJose Reyero
#127 config_metadata-1648930-127-interdiff.txt24.47 KBJose Reyero
#125 config_metadata-1648930-125.patch96.71 KBJose Reyero
#119 config_metadata-1648930-120.patch101.8 KBJose Reyero
#114 config_metadata-1648930-114.patch60.61 KBYesCT
#114 interdiff-109-114.txt6.83 KBYesCT
#113 interdiff-101-109.txt61.21 KBYesCT
#109 config_metadata-1648930-109.patch60.59 KBJose Reyero
#101 1648930-config_metadata-101.patch49.19 KBJose Reyero
#101 views_config_metadata_example.png146.59 KBJose Reyero
#91 1648930-config_metadata-91.patch57.13 KBJose Reyero
#88 1648930-config_metadata-88.patch48.9 KBJose Reyero
#85 1648930-config_metadata-85.patch47.65 KBJose Reyero
#81 1648930-config_metadata-81.patch47.65 KBJose Reyero
#80 1648930-config_metadata-full-80.patch45.14 KBspearhead93
#78 1648930-config_metadata-full-78.patch89.05 KBJose Reyero
#78 1648930-config_metadata-diff-78.txt45.14 KBJose Reyero
#76 1648930-config_metadata-full-76.patch85.9 KBJose Reyero
#76 1648930-config_metadata-diff-76.txt42.38 KBJose Reyero
#68 1648930-typed_data-config_metadata-full-68.patch73.17 KBJose Reyero
#68 1648930-typed_data-config_metadata-68.diff.txt31.72 KBJose Reyero
#67 1648930-config_metadata-67.patch25.74 KBJose Reyero
#65 1648930-config_metadata-65.patch21.13 KBJose Reyero
#48 config-data-type-info-hook-do-not-test.patch4.01 KBtstoeckler
#47 1648930-39-config_metadata_3.patch17.47 KBtstoeckler
#47 config-metadata-interdiff-1-3.txt4.32 KBtstoeckler
#45 1648930-39-config_metadata_1.patch16.32 KBtstoeckler
#45 config-metadata-interdiff.txt4.91 KBtstoeckler
#42 config-metadata-minor-cleanup.txt1.88 KBtstoeckler
#41 1648930-39-config_metadata.patch15.99 KBJose Reyero
#39 1648930-39-config_metadata.patch0 bytesJose Reyero
#15 1648930-14-config_metadata.patch17.66 KBJose Reyero
#5 config_metadata-04.patch12.85 KBJose Reyero
#4 config_metadata-03.patch8.42 KBJose Reyero
config_meta_form.png20.01 KBJose Reyero
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Gábor Hojtsy’s picture

Issue tags: +D8MI, +sprint

Tagging for D8MI sprint to discuss this in relation of a multilingual config UI.

yched’s picture

Not sure I fully get the idea behind generic metadata files, but I fail to see how there can be such a thing as the examplified "image.style.effects.*.yml", since the list of settings in there is arbitrary, and entirely up to the specific image effect ('resize' has width, height, upscale; 'crop' only has width, height; 'greyscale' has none; 'random_foobar_effect' will have any arbitrary list of settings).

This is of course not specific to image effects/styles. Field types, widgets, formatters behave similarly, and in fact any "pluggable" component (views handlers, text filters...)

Looks like what is needed here is an enhanced version of the info available in hook_field_info() ('settings' / 'instance_settings' entries, which currently only list setting names with their default value).

sun’s picture

mmm, why did we create a duplicate of #1498270: Meta data for configuration ?

Jose Reyero’s picture

FileSize
8.42 KB

Improved the previous patch, also addressing comments in #2:
- Defined merge/fallback for metadata files. Example: meta for 'image.style.large' will include 'image.style.*' too.
- Mapping metadata types to form element types and properties, see _config_meta_form_element()
- Better support for nested objects (example of multiple image styles).

Example: for a image style with name = image scale, we'll load these two (the second one is merged into the first one which defines common properties for all image effects):

core/modules/image/meta/image.style.effects.*.yml

name:
  title: 'Style name'
  type: 'item'
data:
  title: 'Data'
  type: 'fieldset'
  subkeys: '1'
weight:
  title: 'Weight'
  type: 'weight'
ieid:
  title: 'IEID'
  type: 'ieid'
core/modules/image/meta/image.style.effects.image_scale.yml

data.width:
  title: 'Width'
  type: 'number'
data.height:
  title: 'Height'
  type: 'number'
data.upscale:
  title: 'Upscale'
  type: 'checkbox'

For this complex mapping, we define a 'nested' property in the main image.style metadata file which tells us which property of the nested objects is used to get the metadata name for each (in this case the 'name') though maybe we don't need this and we could use the nested object's 'name' property always (?).

core/modules/image/meta/image.style.*.yml

name:
  title: 'Machine name'
  type: 'machine_name'
effects:
  title: 'Style effects'
  type: 'fieldset'
  nested: 'name'
  meta: 'image.style.effects.*'
Jose Reyero’s picture

FileSize
12.85 KB

The previous patch (besides a missing file) had some issue with nested objects that couldn't be provided by a different module. All it takes to fix it is adding some namespace in nested object's name (demonstrated for image style effects). The effects definition would look like (note the longer namespaced name):

function image_image_effect_info() {
  $effects = array(
    'image.style.effects.resize' => array(
      'label' => t('Resize'),
     ....
    ),
...
}

This way any module could provide image effects and have the metadata on its own folder, which should be usefule for any type of pluggable objects / configuration. If we looke at the image style config file we can see how this can be mapped to any other configuration object. Note the 'name' property under effects.image_scale:

name: large
effects:
    image_scale_480_480_1:
        name: image.style.effects.scale
        data:
            width: '480'
            height: '480'
            upscale: '1'
        weight: '0'
        ieid: image_scale_480_480_1

Now a module could define a new effect named 'mymodule.image.style.effects.paint' and we would be able to automatically do the mapping and find the metadata on mymodule/meta.

Btw, a helper module to see all this in action (builds forms using metadata) is here, http://drupal.org/sandbox/reyero/1635230

Jose Reyero’s picture

Status: Active » Needs review
plach’s picture

@Jose Reyero:

Should this be kept in sync with #1324618: Implement automated config saving for simple settings forms?

(Sorry if this sounds dumb but I've been away for a while and I'm feeling a bit lost)

Jose Reyero’s picture

Title: Add metadata for configuration in YML files, first try at generating settings forms. » Add metadata for configuration in YML files
Status: Needs review » Needs work

@plach,
Maybe though at this point all issues are waiting for all the other issues.

So that 'settings forms' was more a long term goal, if this gets in they can pull data from here to add some automation, though it is not really the same feature.

Let's keep this focused on metadata storage/handling only, changed title, no longer mentioning settings forms.

Gábor Hojtsy’s picture

Is your nesting technique explained in #4 enough for covering as complex use cases as Views, where the structure could be pretty arbitrary with plugins and displays and basically everything being very dynamic in the config structure?

Jose Reyero’s picture

@Gábor Hojtsy,

I think yes, since it can take any number of nested arbitrary objects and nesting levels and the whole metadata building process is recursive, which means objects can be nested inside other nested objects.

But we cannot be sure until we know how a real view stored in the config system is going to look like, whether it's going to be stored in a single file or not, etc..

However I think the general principle can be addapted and maybe the most important idea here is in #5, properly namespacing object types. I.e. 'image.style.effects.scale' instead of 'scale' but also 'node.type.story' instead of 'story'.

So this or similar should work, though we cannot really know about 'metadata' till we actually have 'data' which is what I'm missing most atm in the config system.

Gábor Hojtsy’s picture

@sun, @yched: what do you think of the current approach?

Damien Tournoud’s picture

This "metadata" sounds quite a lot like a... schema. At this point, we are just reinventing the wheel. Because we picked YAML, we only have a couple of pre-existing options (there would be a lot richer options in XML and JSON):

  • Rx: an extensible schema language for JSON and YAML with existing implementations in Perl, JavaScript, Ruby, Python, and PHP
  • Kwalify: a validation / data mapping tool first and a schema language second. Seems less extensible then Rx. Implementations in Ruby and Java.

Let's not reinvent our own schema language.

Jose Reyero’s picture

Updated the patch for latest changes in the config system and fixed tests with new image style names.

@Damien Tournoud,
Yes, it sounds a lot like a schema, though it is a different thing, it's a mapping into Drupal's objects and administration forms. It is not a mapping into database/storage types nor validation rules. (For which I'm afraid we've already reinvented the wheel, so we need to map our configuration into our already reinvented wheel).
Besides, I don't see any of that options to provide any kind of support for nested objects, nor user interface names (though these UI strings we could drop from this patch anyay..)

The thing is while moving configuration from tables into data structures, we are already losing all schema information associated with it. Example, for images, we are dropping two tables (image_styles, image_effects) and all their schema information. Not that we used that too much but the useful data in that schema and some more (readable names for fields, kind of data types that we can map into forms, etc..) would be back here.

Jose Reyero’s picture

Status: Needs work » Needs review
FileSize
17.66 KB

Forgot the patch

merlinofchaos’s picture

In clicking through the Rx documentation, it's difficult for me to tell in a short time but it doesn't look like Rx would be suitable for the dynamic needs of Views which will not have a 'stable' schema but instead provide a whole bunch of fields which are valid only for a particular object based upon the configuration of that object at that point in time. Such a 'schema' would be very dynamic, and thus not really a schema at all.

merlinofchaos’s picture

In reviewing the patch, it feels like most of this code should actually either be on the config object or in a secondary object that the config object uses, not separate procedural functions. It's difficult for me to read where this code is actually fitting in, but for the most part I thought metadata was supposed to be something that's more or less invisible. This feels like it's kind of bolted on after the fact, and I can't tell from the patch how I'll use it or how the metadata will affect me when I'm loading config.

Jose Reyero’s picture

@merlinofchaos,
The idea of this metadata is that is is not needed at runtime, only when editing / updating the configuration (Otherwise yes I would agree it needed to be inside the config object itself).

So this is mostly a generic description of config objects that will be used (on the side I'm working on) for being able to tell which are translatable properties and for naming such properties, but also it could be used to generate config / edit forms on the fly.

Though the metadata itself I think is quite straight forward (but for the plugins support) maybe the confusing part is where we merge the metadata with actual config data to build an array containing both, this is just an intermediate step for being able to process it easily like in the patch, when generating a config form on the fly that possibly won't fit all cases but maybe we can save a good number of configuration forms.

Actually, current configuration forms contain a good amount of metadata. We are just trying to move that into some format that can be a) reusable b) easily parsed c) (why not?) possible to generate and move around without moving php code.

chx’s picture

I already burned a lot of karma and bridges over trying to protect my vision of CMI which was a simple system. That's of course now has been ruled out and I have been very badly burned over trying to get even a simplification patch in. But, seriously, file names with asterisks in their names which is a wildcard in every filesystem? Recursive metadata files? What the issue summary imo lacks is the actual problem we try to solve. It lists why this would be good but why is this needed? If we need metadata, is that part of the config or is that like an information hook? This whole things looks like an ugly hack to me but that can be just me.

Gábor Hojtsy’s picture

@chx: With CMI, multilingual sites loose two features and we wanted to introduce one more key feature, all three of which need metadata about configuration to function.

Features lost with CMI (regressions from Drupal 7):

1. When default configuration is saved into the database, the code settings up the default config uses t() on strings that are translatable. Think content types, forums vocabulary name, etc. That is, if you are installing a foreign language site, you get your config set up in your language. This is lost with CMI because we have no information as to which parts of config are to be translated on setup.

2. For variable_set()/get() based settings in Drupal 7, unless you saved the setting to your database, translation updates are effective to your configuration translation. So if you don't save your user email text templates, you are getting translation updates normally for the strings as they are available. Again, this is because the runtime default code for variable_get() would use t() to translate the default text (and save the translated text to the DB in your language, but also use the right runtime translation until you save it). Just as above, we'd need metadata about the strings that need translation to be able to keep track of this.

And finally, a feature that has one-off implementations but would ideally be generalized:

3. For configuration translation, core only has two implementations: path alias language support and date format language support. Both have highly custom code and highly custom user interfaces to operate. When porting things to CMI (even if we don't port path aliases, we'll most likely port date formats, right?), we can either keep the custom built translation code or migrate to a more general translation solution. To be able to regenerate settings forms for translation, again, we need metadata about the configuration, we need to know which ones should be stored translated, which ones will end up changing for all languages when changed, etc.

(Copying this to the issue summary as well).

Helpful?

Gábor Hojtsy’s picture

Issue summary: View changes

Including screenshot

chx’s picture

1. Option A/C provided this with great ease and little code. It didn't need recursive metadata. Noone is going to understand this and noone will care.

2. That's a hack "if you do not save to the database". I do not think preserving a hack is something we should be worried about.

I would urge considering what are the priorities of CMI are. Is it going to be same as with field API where everyone suffers for multilanguage even when a fraction of users use it and noone knows what to do with it otherwise?

chx’s picture

Issue summary: View changes

Add problem statement

plach’s picture

[Sorry for being OT]

@chx:

Is it going to be same as with field API where everyone suffers for multilanguage even when a fraction of users use it and noone knows what to do with it otherwise?

Just for the record I have been proposing improvements for months now in #1260640: Improve field language API DX.

And the driver of this activity was mainly your feedback about field language DX. Nonetheless the issue has seen very little feedback and recently has been demoted from critical to major, since actually noone cares about it.

chx’s picture

To continue, when you have type: 'ieid' it's clear this metadata is bordering meaningless, it's completely arbitrary and haphazard. Are we going to implement an arbitrary type for every sort of primary key?? Why do you need more than 'translatable' perhaps implicit in an 'original language' key? Auto generating forms is not something that generally works so I would suggest not even considering that.

Gábor Hojtsy’s picture

1. Option A/C provided this with great ease and little code. It didn't need recursive metadata. Noone is going to understand this and noone will care.

Option A/C used a lookup function to look up translations in the config file. How did those translations get into the file for your default content types when you enabled the node module? (Regardless of whether you considered option A, B or C?) As far as I've seen this is a problem that was not considered by either before, because it is about initializing your default config on installation with data from translations, that was not considered in any of the options, and all of them would need some kind of metadata/description/information as to which pieces to translate when initializing the default config for your site.

Gábor Hojtsy’s picture

2. That's a hack "if you do not save to the database". I do not think preserving a hack is something we should be worried about.

I think it is part of the CMI goals to be able to update your config as long as it is still the default. At least that is a basic features module feature. Updating translations is just a subquestion of that. If the default config can be updated but translations cannot, then translations will get outdated and out of sync.

chx’s picture

As for autogenerated forms, just look at the example, surely you would want an arrangement of width x height, perhaps a preview, ieid machine generated etc. Many frameworks tried the "let's autogenerate forms!" and it looks good in a tutorial... and that's it.

Jose Reyero’s picture

@chx

I would urge considering what are the priorities of CMI are. Is it going to be same as with field API where everyone suffers for multilanguage even when a fraction of users use it and noone knows what to do with it otherwise?

Nope. The idea we are working with for CMI is building a localization system that has zero impact and complications when disabled. That's why we are working on plug-ins and overrides with separate files per language.

If you look at this metadata system too, the idea again is that it sits there unless needed, and it won't be needed for runtime configuration.
And another requirement we have (for localization) is that it can be parsed by an external tool (think of po extractor) to extract translatable strings.

To continue, when you have type: 'ieid' it's clear this metadata is bordering meaningless, it's completely arbitrary and haphazard. Are we going to implement an arbitrary type for every sort of primary key?? Why do you need more than 'translatable'...

You are right about data types not being well defined / thought of here and this is where this patch may really need improvement, because these are just examples of how it may work.
About 'ieid', well, it depends on whether it ends up being reused on more objects or not (we are waiting for CMI on that). If not, we could map it simply like 'varchar'.
About 'translatable' the idea we are working with is that a module doesn't 'decide' whether a piece of data is translatable/localizable or not. It is the localization system that decides that depending on what the data is. For the default (core) localization system, likely only human readable strings will be localizable, and that's why we need to mark them somehow. But another contrib module may decide later that email, currencies, strings, measures, paths, etc are localizable.

So I agree about the data types not being very well thought of yet. Really we are needing more data (content types, contact forms, user emails, etc...) moved into the config system to see how it may look like.
The initial idea is, lacking a better one yet, mapping into Forms API field types so most settings forms can be generated on the fly (Since we need it for translation UI, why not using it for most settings forms saving maybe a ton of PHP code?).

About other issues like the wildcard character, well, it's just a character, if it creates issues with any filesystem we can just change it.

So, the basic ideas (which I think this system fits with):
1. Modules don't need to deal with it unless they actually use it for anything.
2. We don't add tons of lines of PHP, these files are loaded just for administrative operations.
3. Really reusable data, easily parseable so it can be used by other tools. No need to parse PHP to extract data that is not PHP.
4. Support for pluggable objects, as needed (likely) by Views but also can be used for simpler things like image styles, which is demonstrated in the path.

What I agree it still lacks:
1. Better definition of data types . For what we need to see more real data in CMI and (2)
2. Cleaner target, what we are going to use it for. I'm working on this for the localization system but, would we want it for validating config data? For generating settings forms? For development tools that handle config data?....

One example: Say I am importing a config file (yml) that is a View. Would I want to be warned if it is using any plug-in provided by a module that I don't have available? That would be possible with this kind of metadata.

Gábor Hojtsy’s picture

@chx: yeah, I've had that discussion with @webchick, @Dries et. al., and while I was/am on the side of not doing crappy forms, they were on the side of having crappy forms are better than no forms.

For a config translation UI, there are basically two options: (a) you show the same form to translators thaw as used to config the original thing (b) you show a translation form with just translatable pieces. Unfortunately (a) is only possible if you can grant permissions to edit the actual config, or all config needs to move to a "field" based permission model instead of protecting the whole config form altogether. Also for (a) we need to know the translatable/multilingual metainfo foe each "config field", so we can somehow display on the UI if your change affects all languages or only one language. For (b) however, we need to regenerate part of the config form for only the translatable/multilingual pieces.

An example: It is very hard to imagine that Views in core would allow translation of views with the base views UI itself. You likely don't want to grant people access to your views config form just to translate your views, so we'll need to somehow generate a UI for your views, and this level of data would need to live in core or contrib on behalf of core. By not having it in core, other contrib modules will not adapt it, and i18n module would need to implement it on behalf of select other contrib modules, which is both painful and will limit the multilingual feature set a great deal.

This is our main motivation to have this metadata in core to a level that is at least useful to generate some crappy forms. Yes, ideally, we'd refactor all config UIs to be "fields" based, make their form widgets and validation independent and componentized. This is not even in the goals of CMI, and have not found sufficient support for that, so we'll need to have some kind of metadata to be able to generate at least some crappy forms. We'll need this metadata in contrib at least, but that limits the spread of it, so we are trying to get it in core.

Hope that helps enlighten the background.

chx’s picture

Modules don't need to deal with it unless they actually use it for anything.

Well, that's the problem. As a contrib author I likely need to write metadata for my contrib to be translateable and as with the field API, I predict this will be something noone will know how. As a core developer, i need to write meta data for every piece of config we convert or create new.

It seems we are collating two issues into one: autogenerating forms and multilanguage just because we have a particular implementation that would allow for both. However, that particularly implementation is a huge DX WTF. If we do this, then the point of picking a human read-writeable config is lost. And no I do not have a particular suggestion on what else to do because every comment is moving the goalposts and I am completely lost on what is needed.

Gábor Hojtsy’s picture

@chx: what is needed is a solution for this simple story: "As a site builder I want the Drupal core shipped Views to be translated to my site's language when Drupal is installed and then let translation of my Views to other languages for people with or without permission to modify my Views". Succinct enough?

Update:

[12:39pm] chx: GaborHojtsy: previously that was solved by the translatables in views exports, wrapping the translateable strings in t(), right?
[12:39pm] GaborHojtsy: chx: not for editable views
[12:40pm] GaborHojtsy: chx: i18n_views module has solutions for those
[12:40pm] chx: ah ha because if you imported them then views wrote the strings in the db and no longer were they t()'d ?

Gábor Hojtsy’s picture

Issue summary: View changes

"Lose" means the opposite of "gain." "Loose" means your mother, Trebek.

chx’s picture

Sure it is. Copied to the issue summary. This is indeed something we can now think on, thanks.

Jose Reyero’s picture

So, about the questions we are discussing here:

1. Do we need metadata? (Yes)
I think at this point we mostly agree that we do need metadata so other system like localization can do their work without the modules defining the configuration needing to bother about it, right?
(Because the other option is each module translating its own stuff)

2. How to implement it?
For which this patch is a proposal which as explained above has some advantates. This is mostly what we should discuss here IMHO about this implementation details or whether are there other options (other patches).

3. What data / data types do we need in the metadata?
For the purpose we are bulding it for, we need just some basic information about data type and some mapping into forms (so we can build proper translation forms). However we can be more ambitions here and that comes to the next question:

4. What else could we use that metadata for?
And this is really open. The initial idea here was being able to build some edit forms out of the metadata, that may work for a big % of current settings forms. For that we'd need the full Forms API data here (title, description, etc).

chx’s picture

I am not sure I agree on even 1. I will ponder on the Views user story posted during the weekend.

Is the following true: some information is translateable and some information is multilingual and these are not the same and likely need different UIs and we are trying to solve only the translation case here?

I am thinking on using the StorageDispatcher to have a conf override and to have a translation override as well. If we go down that route, which override wins?

Jose Reyero’s picture

> Is the following true: some information is translateable and some information is multilingual and these are not the same and likely need different UIs and we are trying to solve only the translation case here?

This is just some kind of consensus about terminology. UI strings are 'translatable' while other pieces of data that may be different per language (like a logo or an email address) we call them 'multilingual'.
While our priority is to handle just strings in core (translatable) what to do with other kinds of data can be easily fixed in contrib *as long as we have metadata*.

I am thinking on using the StorageDispatcher to have a conf override and to have a translation override as well. If we go down that route, which override wins?
I guess $conf should win, but you tell me, it's just a question of 'weights' if we ever get some plug-ins/helpers/listeners into the config system.... See #1646580: Implement Config Events and Listeners, and storage realms for localized configuration

chx’s picture

OK, so let's suppose we need metadata. I refuse to think that automated forms will do anything good, it complicates the system and will have zero avails IMO because you can create automated forms with just machine names and they are equally non-useful. So, if all we need metadata for is to indicate what parts need to be l10n-able then what's the simplest we can come up with?

chx’s picture

I withdraw my concerns. There can't be anything better than core/modules/image/meta/image.style.*.yml . Arbitrary types are great, let's use them for validation, that gives them meaning.

Jose Reyero’s picture

Assigned: Unassigned » Jose Reyero
Status: Needs review » Needs work

Since it seems, from the phone call yesterday that we mostly agree on:
1. We need metadata for configuration.
2. YML files for that look good (so let's keep working on that direction).

But we are still not sure about the last part:
3. What do we want in / from the medata / which data types we should use.

I will be trimming down this patch, forget about config forms for now (though that's an option to let open for the future) and sticking to the very basics we need to build some translation UI: Being able to tell which config properties are translatable strings, have a title for that properties (so it can be shown to the translator).

So this initial patch is going to get a bit simpler and in the meanwhile we can discuss other concerns, like what other uses we want for this (besides localization), what other data types we need, etc..

Jose Reyero’s picture

New simplified version:
- No forms, moved that to the test module linked below.
- Assuming the 'name' property to find the right nested configuration metadata
- Added metadata for first translatable string I've found in Core!! (Site maintenance message).
- Only two data types for now: 'string' and 'text' (these will be the translatable ones), the other are there as an example and to fill the blanks in metadata files.

I'll post another patch with the locale plug-in for using this metadata.

In the meanwhile you can take a look at 'real' data and metadata, included form generation, using http://drupal.org/sandbox/reyero/1635230

tstoeckler’s picture

:-( The patch is 0 bytes.
Could you re-upload that?

Jose Reyero’s picture

Status: Needs work » Needs review
FileSize
15.99 KB

I'm sorry, here's the patch

tstoeckler’s picture

I found a bunch of minor docs/clean-up stuff, but didn't want to bloat this issue, so I'm attaching a txt file with (that part of) my Dreditor review inside. It probably makes to completely ignore that for now, but I didn't want it to get lost.

+++ b/core/includes/config.inc
@@ -105,6 +106,142 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+          $meta[$name] = drupal_array_merge_deep($meta[$name], $data);

This should be NestedArray::mergeDeep($meta[$name], $data);

The fact that the patch passes means that this code isn't run anywhere, currently. That's probably OK for now, but just saying.

+++ b/core/includes/config.inc
@@ -105,6 +106,142 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+function _config_meta_match($meta, $data, $name = '', $prefix = '', $group = '') {

Like a bunch of the other functions, this is missing @param and @return documentation for now.
I'm mentioning this here, because I found for this function, it was the hardest to grok what was going on. In general the functions are, IMO, not exactly easy to follow, but then again, it's also not exactly an easy topic. But this particular function I found hard to grok. I don't know if we can do something about that codewise (again, I realize that this does some pretty cool stuff!), but providing some docs here would be really cool.

+++ b/core/includes/config.inc
@@ -105,6 +106,142 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+    // Nested elements, fetch metadata for each element using two elements.
+    // - $meta['meta'] will contain the base metadata name.
+    // - 'name' property will contain the full configuration name for that element

It would be cool to document these two different things in more detail. It took me a while to grok this. Maybe something like:
"For nested elements, the parent metadata file declares the child element using the 'meta' key. Actual configuration (value) files directly specify the nested configuration in one file and denote the child element with the 'name' key."

+++ b/core/includes/config.inc
@@ -105,6 +106,142 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+/**
+ * Helper function to build metadata name.
+ */
+function _config_meta_name() {
+  $args = func_get_args();
+  return implode('.', array_filter($args));
+}

I do not like the introduction of this function. In my opinion, it makes things harder to read. I find

$name = "$base.$index";

much more grokable than

$name = _config_meta_name($base, $index);

Thoughts?

+++ b/core/modules/image/meta/image.style.*.yml
@@ -0,0 +1,7 @@
+  type: 'fieldset'
+  meta: 'image.style.effects.*'

We discussed shortly that this notation is not really ideal, but did not really come up with something ideal. I proposed something like:
type: 'config'
child_name: 'image.style.effects.*'
Not really sure, though. (Mentioning here for completeness' sake.)

+++ b/core/modules/image/meta/image.style.effects.*.yml
@@ -0,0 +1,13 @@
+  type: 'ieid'

Is there a reason this is not simply "type: 'id'"?

+++ b/core/modules/system/meta/system.maintenance.yml
@@ -0,0 +1,7 @@
+message:
+  title: 'Message to display when in maintenance mode'
+  type: 'text'

+++ b/core/modules/system/meta/system.site.yml
@@ -0,0 +1,18 @@
+slogan:
+  title: 'Site slogan'
+  type: 'string'

IIRC, Jose Reyero explained that the difference between 'string' and 'text' is that 'text' can contain HTML. I don't think that is very clear with the current names, though.

vasi1186’s picture

One thing that in my opinion is not so nice for the metadata files (actually also for the config files) is the name of the files. A module like views can have some really ugly metadata names. I agree with having the name of the module in the filename, but I think the full path (like image.style.effects.scale) could be actually kept in the file itself.

Probably this thing is not fully related, or not only related to this issue, but I posted this here because I just reviewed this patch now.

gdd’s picture

For the config files themselves, we keep that data in the filename for easier searchability without having to load all the files for a module. For instance, when I need a listing of all image styles, I can just get all the files that begin 'image.style.*' rather than loading every file in the directory and introspecting it. I am just starting to review this patch but I wanted to clarify why we did it in the original files.

tstoeckler’s picture

Here's a new version. Interdiff is attached. This patch changes the following (apart from trivial stuff):
1. changes _config_meta_name() to config_build_name()
The function really isn't specific to the metadata system even if it is introduced here. I tried replacing the function with inline code, but most of the strings used can be empty strings, so that became very verbose.

2. Changes the syntax for nesting of config files (image styles > image effects) into:
type: 'nested_config'
config_name: 'foo.bar.baz.*'
Maybe not ideal, but it makes sense that this is actually a type. I thought about introducing some sort of processing for data types, so that the 'nested_config' type could handle the nesting on its own. That would simplify the code in config.inc. For now we're going with the simplest solution, though.

I tested this with the test module, so things *should* work...

tstoeckler’s picture

I'm now working on improving the in-code documentation, and I have three proposals. I will not add them to the patch I will post later, because that would break @Jose Reyeros test module in the sandbox, so you should probably to that yourself (if you agree).

1. Rename config_meta_data() to config_metadata()

2. Moving the actual code in config_metadata() into Config::getMetadata()
Since we always need $name to get the metadata, we might as well instantiate the Config object. There shouldn't be much overhead with that.

3. Making config_meta_build either take just $name (preferred) or both $metadata and $data. Currently you pass in $data, but we get $metadata for you from $name. That's inconsistent IMO.

tstoeckler’s picture

This is what I have so far. Go @heyrocker! :-)

tstoeckler’s picture

So in the mean time I worked on a hook_config_data_type_info().
The only data we collect is whether a data type is translatable or not. That way the LocaleConfigOverrideSubscriber (or whatever it is called) does not need to hardcode the 'text' data type.
I put a bunch of the questions I had while doing this in code. In general, I'm still torn on this, as currently it doesn't really buy us anything. I think that in general / in the long run it is a good idea, but...

Anyway, I attached only the interdiff in order not to mess up the master patch. Should probably just apply on top of that.

tstoeckler’s picture

What I just noticed trying this (the hook_config_data_type_info()) and is super strange is that when we get to config_meta_build(), every single key (!) is of type 'value'. I don't even know what 'value' means in this context, but I find it very odd that that information just gets lost. I don't grasp the code enough yet to know whether that is intended or a bug, and if the latter how to fix it.

gdd’s picture

One thing I realized today is this won't work with metadata provided by other modules. For instance, say that my module provides a new image effect. The current system only will only look in the image/meta folder, and thus won't find the metadata for my new custom image effect, residing in my own module directory.

I think this brings us back to copying metadata into a central location like config/meta at installtime.

yched’s picture

@heyrocker:
Field API, or any other system that is based on pluggable components, will have the same issue: we can provide some generic, hardcoded metadata about the structure of field.[field_name].yml, but some parts (the 'settings' part) are specific to the actual field type of field_foo, and are thus dynamic. Notably, it's the job of each field type to specify which of its settings are translatable strings (same goes for widget and formatter settings within the yml file for an $instance definition).

I'm not sure what the plan is to deal with this, but it seems that figuring out the metadata for the actual content of a specific field.field_foo.yml would require some runtime logic to merge the pieces that are dependent on a specific plugin implementation.

Jose Reyero’s picture

About @heyrocker #50

this won't work with metadata provided by other modules. For instance, say that my module provides a new image effect. The current system only will only look in the image/meta folder, and thus won't find the metadata for my new custom image effect, residing in my own module directory.

The idea, that is included in the patch, is changing these 'plugin names' so the have a full namespace. So if you look at image style effects they've moved from names like 'resize_image' to 'image.style.effect.resize'. The first part of the full name is the module name so this is the way we can map in plug-ins or add-ons from other modules.

That feature is included in the patch, just not put to work yet so what we are missing is only some code to test that.

(For the latest updates of the patch, I've just got back home and haven't been able to review it yet.)

Jose Reyero’s picture

About data types, it seems there's something in progress, not committed yet, from Entity API side that could be useful here. See:
- hook_data_type_info() in this patch #1696640: Implement API to unify entity properties and fields
- List of data types at the top of #1525958: revisit primitive data types
I think we could use this list as a basic set of data types for this patch.

Then about pluggable configurations, while current system works, I am thinking it could make more sense to add the base type in the metadata for each configuration name (so we could save that config_name property for nested configuration). Example:

mymodule/meta/mymodule.image.style.effects.xxxxxx.yml

config_type: image.style.effects.*
......
   other metadata here
......

This would give use some more straight forward mapping from any config name to the full metadata.

Gábor Hojtsy’s picture

Issue tags: +VDC

Adding VDC tag since this is a cornerstone of porting language support from (exported) Views to core.

Gábor Hojtsy’s picture

Priority: Normal » Major

Elevating to major as it is a cornerstone to have any CMI language support in core.

KarenS’s picture

@Jose, in #52, are you saying that image module would provide 'image.style.effect.resize' and my custom module might provide 'mymodule.style.effect.squeeze' and the system would pick that up and know it's part of image styles? Or are you saying my custom module would provide 'image.style.effect.squeeze', using the 'image' prefix? I'm struggling with the way that module-provided date formats would be implemented, which brings me to the same issue. I don't see how that will work.

KarenS’s picture

OK, missed the comment in #53, it would be 'mymodule.image.style.effects.squeeze.yml'. So the intention is that will get picked up I guess, both for the config and the meta ('mymodule/config/mymodule.image.style.effects.squeeze.yml' and 'mymodule/meta/mymodule.image.style.effects.squeeze.yml'). I'm just trying to understand if the system is intended to be used that way.

sun’s picture

First of all, I don't think this applies to date formats/types, since all of them are following a single, defined schema.

Second, the three-dimensional module dependencies issue is indeed concerning and somewhat questioning the concept here.

Given:
- An Image module that supports pluggable image effects for image styles.
- Image style configurations in image.style.* config objects, also containing the configured image effects.
- No default schema/metadata for image effects provided by Image module, since each effect config is different.
- A Unicorn module that provides a 'Unicornifier' effect, which can be configured.

When:
- Unicorn module ships with a default "unicorn" image style as modules/unicorn/config/image.style.unicorn.yml
- Unicorn module ships with a schema/metadata definition for its 'unicornifier' effect in modules/unicorn/meta/image.style.*:effects.unicornifier.yml

Then:
- The non-Drupal parser won't be able to properly parse/extract the schema/metadata definition for unicornifier effects, as long as the meta definition for image styles does not exist (i.e., Image module's schema has to exist in parallel and must be known already, in order to process the Unicorn module schema).

(possibly further problems/consequences)

This is a bit concerning.


Btw, two further critical issues:

1) We cannot use * (asterisk) in filenames. Windows doesn't support that. It's going to get extremely hard to find a sane wildcard character that does not clash with any other valid or reserved characters in config object names.

2) A schema file of image.style.effect.quality.yml is structurally wrong. We need to follow the exact structure in the configuration objects somehow and need to separate the config object name from the targeted config object keys within; i.e., image.style.*:effects.quality.yml, because the former would be ambiguous otherwise.

yched’s picture

Somewhat related (also, shameless plug) : #1764380: Merge PluginSettingsInterface into ConfigurableInterface
Having a standard way to deal with those 'properties that are specific to a given plugin implementation' (e.g. a given image effect) and their metadata seems like it would help here.

gdd’s picture

I've reviewed this patch twice and I see two problems. One is find it really hard to follow. Part of that is just a lack of documentation, but the other is that the recursive logic it goes through to assemble the final result from least-to-most-specific-file is fairly confusing. I was often reminded of FAPI as I stepped through it in the debugger (the only way I could really manage to get it in any meaningful way.) So I really think we need to figure out a way to simplify things, especially since as sun mentions above in #58 we're going to have a hard time figuring out a wildcard that works cross-platform.

Another problem is that it still does way more than we need for the one single use case we have that absolutely has to go in before feature freeze: identifying translatable strings and providing a label for them.

So what I'd like to propose is that we do something that is very simple and provides the bare minimum necessary functionality. If we want to go beyond that we certainly can, but this guarantees we get in what we need before feature freeze and will help me sleep at night.

One thing I think makes sense is just do one metadata file per config file. While this might result in some repetition of data across files, it would also make the system a lot clearer. Additionally, since our only real use case at the moment is identifying translatable configuration, couldn't we just simplify this down to the items that are actually translatable? Then we wouldn't even need type, and we could just make the value the label. So for system.site:

name: 'Site name'
slogan: 'Site slogan'

These easily map back to the original configuration so it would be really easy for a tool to be built that grabs them all and lists them along with their current values. It would also reduce this patch down to almost nothing, making it easy to commit and getting one more huge essential feature off our plate. I realize this isn't really a 'metadata system' anymore, but a 'translatable identification system'. In the meantime, if people want to keep working on a greater metadata system, go nuts. I'm just getting really worried this issue is going to drag on forever and I don't have a lot of confidence right now in a fully featured system getting to RTBC.

Yes? No? Flame?

Jose Reyero’s picture

About the need to make the code cleaner and more understandable I agree, though due to the recursive structure (data may contain plugins that may contain other plugins), it is not that easy. Anyway, taking note for working into that.

But about making metadata elements a single string I think we would be missing a great chance to build something really extendable. If it is an array you can just add any other property there. If it is just a title and we are limiting this to translatable strings for the future then I guess we don't really need such a complex subsystem.

fago’s picture

For a metadata system, I think this could built on the TypedData API as built at #1696640: Implement API to unify entity properties and fields. The work there now has a complete set of defined and documented primitives and hook docs now. So it can be considered complete, although it might move to using the plugin system.

+ * - title: The human-readable name for this configuration item.
+ * - type: The data type of this configuration item. Valid types include number,
+ * text, string, machine_name, ...

That would have to be 'label' and types must be one of the defined-types. Types are modular extensible though, so you might want to restrict allowed-types for your use-cases further, e.g. to the pre-defined primitives - such that metadata stays easily parse-able by other systems.

About the need to make the code cleaner and more understandable I agree, though due to the recursive structure (data may contain plugins that may contain other plugins), it is not that easy. Anyway, taking note for working into that.

The typed data API offers wrapper objects you could use for combining metadata+data values. So it doesn't do the nested parsing for you, but you could easily pass on the nested-metadata with the wrapper objects such that each wrapper just receives the according metadata and passes on relevant parts to any derived property wrappers. That way, you would not have to do any complex, up-front recursive merges any more.

Gábor Hojtsy’s picture

Issue tags: +language-config

Tagging for config.

effulgentsia’s picture

I spun off #1777268: Add a TypedData API for allowing metadata to be associated with data from #1696640: Implement API to unify entity properties and fields in the hopes that it can be more easily reviewed by the people here to make sure it satisfies the needs of this issue in addition to the entity property one.

Jose Reyero’s picture

Fully reworked patch:
- All the code encapsulated in ConfigMetadata class.
- Cleaner logic, elements are processed one at a time, etc...
- Changed 'title' to 'label' and most data types for being closer to TypedData API (thanks @effulgentsia #64)
- Changed some element types looking for more consistency (All config specific types are config_ )
- Changed file names so parent metadata uses '__' (double underscore) instead of '*' which should be Windoz compatible.

Now config_metadata($name) returns a full ConfigMetadata object.
Data can be built using $metadata->build($data);

Some elements in the metadata look different now:

// image.style.__.yml

name:
  label: 'Machine name'
  type: 'machine_name'
effects:
  label: 'Style effects'
  type: 'config_nested'
  config_name: 'image.style.effects.*'

// image.style.effects.__.yml

name:
  label: 'Style name'
data:
  label: 'Data'
  type: 'config_subkeys'
...

Jose Reyero’s picture

Improved latest patch:
- Fixed image style effect names in failing image test.
- Made some methods static. Added comments and built data examples.
- Keep built data into the Metadata object so it can be retrieved later.
- Automatically build with current configuration if no other config data is passed.

Jose Reyero’s picture

And this is another one using TypedData API that:
- Returns TypedData properties instead of plain arrays.
- Repurposes ConfigMetadata to be a cached array to retrieve metadata from name.
- Adds new class ConfigWrapper which basically wraps a configuration object with metadata and allows retrieving full typed properties instead of plain values.

Important: Includes: #1777268: Add a TypedData API for allowing metadata to be associated with data
- The full patch (first) includes both patches.
- The second one (diff) requires the TypedData to be applied first.

// Usage
  $name = 'system.site';
  $meta = config_metadata($name);
  $data = config($name)->get();
  $data_wrapper new Drupal\Core\Config\ConfigWrapper($name, $meta, $data);
  // To get a property from the wrapper.
  $data_wrapper->get('name');
  // This will return a Drupal\Core\TypedData\Type\String Object
gdd’s picture

Status: Needs review » Needs work

I spent some time with this today, and overall I have to say I like it a lot better than the last approach. The code is a LOT easier to follow, and I agree that if the TypedDataAPI is going into core anyways then leveraging it like this makes a ton of sense. I did encounter a couple smallish bugs, but I think that rather than just post a reroll here I'm just going to start building the tests to start really shaking that stuff out. I should have time to start doing that in the next day or two. The docs and comments could probably use some cleanup and stuff but overall I'm feeling a lot better about this now.

I see that the wildcard has been changed to a +, are we sure this is going to be safe across the systems we support? We should research this some.

Thanks a lot for all the work on this Jose!

Gábor Hojtsy’s picture

Status: Needs work » Needs review

The template suggestion system uses % as a wildcard character *in filenames*, that I guess worked so far, read more at http://drupal.org/node/1089656

fago’s picture

Status: Needs review » Needs work

It's great to see you are exploring the typed data api for this :-)

+++ b/core/lib/Drupal/Core/Config/ConfigWrapper.php
@@ -0,0 +1,426 @@
+   *   The metadata property that was requested or the full metadata array.
+   */
+  public function getMetadata($key = '') {

Hm, where is the difference between getDefinition() and getMetadata(). Is there a difference between the data definitions and the metadata? I'd assume it would have to be *the same*?

+++ b/core/lib/Drupal/Core/Config/ConfigWrapper.php
@@ -0,0 +1,426 @@
+   * The result will be a nested array merging the full configuration data and
+   * the metadata. The array keys will be the property names
+   * and the array values will be data properties.

With the typedata API you shouldn't have the use case of that, as you generally have the wrappers for that. I.e. you have the data wrapped into typed data object which allow you to access the metadata (=defintions) at any layer.

I'm not sure whether you'd need something like that internally. I'd assume it to work that way:

Root level: Config wrapper object. Retrieves metadata about the config object.
Then to get a property wrapper:
- a) If its a primitive, pass on its definition and create the wrapper.
- b) If its a structure, create a ConfigStructureWrapper object, i.e. of type 'config_structure' or so. Pass on its data value and the subtree of metadata so it knows about any contained properties (potentially nested).
Level X: ConfigStructureWrapper. Gets data and a subtree of metadata passed from the wrapper object.
To get a property wrapper it can follow the process as above.

With that approach no recursive merging of data and metadata or building nested structures should be required, it just creates the right wrapper object on each level an passes on the metadata. Does that make sense or am I overlooking something?

+++ b/core/lib/Drupal/Core/Config/ConfigWrapper.php
@@ -0,0 +1,426 @@
+    if (!isset($meta['type'])) {
+      $meta['type'] = 'variant';
+    }

So variant really means "unknown" or "undefined" right?

fago’s picture

Status: Needs work » Needs review

Not meant to change status, more reviews please ;-)

Jose Reyero’s picture

Hm, where is the difference between getDefinition() and getMetadata(). Is there a difference between the data definitions and the metadata? I'd assume it would have to be *the same*?

getMetadata() just gives us the raw data from the yml file and is used internally so maybe I just change the visibility of this method or rename it to getConfigMetadata().

About the ConfigStructureWrapper this is something that should be provided by de TypedData api itself, if it pretends to support any kind of nested data structures so it shouldn't be config-specific. But I'm looking forward to seeing some example of that cleanly implemented in the code.

So our main issue at this point is TypedData API *failing* to support nested data structures, then if needing our specific implementation of that for Config it will be easier not using the StructureInterface at all.

This means, if we pretend to get anything in time, we may need to look for alternates unless all these issues are fixed in the API itself and it gets committed sometime soon. See other related comments here, https://drupal.org/node/1777268#comment-6510732

Jose Reyero’s picture

New patch simplifying the code -still more- and dropping StructureInterface that was not any help for us at all, see my previous comment. Notes:

- Added tests. Please take a look at the ConfigMetadataTest to see full capabilities of this.
- Simplified constructor now it only needs config name, will figure out the rest.
- Moved some methods from Config to NestedArray so they can be reused.
- Now the wrapper is really a wrapper around the configuration data and it can really be updated using the wrapper.
- Also it can be used to get properly typed values out of the configuration data, that would be useful for admin/validation/etc. (!)

Really missing here more properly designed and more flexible Config objects, then we could really build some wrapper for them. So far this is just a wrapper around 'config data', not around 'config object'.

Important: Includes: #1777268: Add a TypedData API for allowing metadata to be associated with data
- The full patch (first) includes both patches.
- The second one (diff) requires the TypedData to be applied first.

Usage, still easier and more powerful:

  $wrapper new Drupal\Core\Config\ConfigWrapper('system.site');
  // To get a property from the wrapper.
  $site_name = $wrapper->get('name');
  // This will return a Drupal\Core\TypedData\Type\String Object
  print $site_name->getValue();
  // Would print 'Drupal'

 // Properties can be changed and added back into configuration like this.
 $site_name->setValue('New Drupal');
 $wrapper->set('name', $site_name);
 // And we can retrieved an updated array of config data ready to be saved !!!
 $config_data = $wrapper->getConfigData();
Gábor Hojtsy’s picture

Priority: Major » Critical

Highten priority level as Drupal 8 is not shippable without a solution for this.

Jose Reyero’s picture

One more version:
- Moved everything to its own namespace: Drupal/Core/Config/Metadata/*
- Cleaner implementation, renamed some objects, cleaned up metadata files (they have similar nesting to config data now).
- Wildcard marker in file names is now '%', as suggested by Gabor in #72
- Smaller an cleaner up to the point classes: ConfigStructure, NestedStructure, ConfigWrapper.
- The base class ConfigStructure fully implements TypedData/StructureInterface (And ListInterface too)
- Added API funcion to get a wrapped configuration object: config_wrapper($name);
- The wrapper is now an actual wrapper around a Config object: $wrapper = new ConfigWrapper($config);
- Renamed Variant data type to Undefined.
- Drop previous changes in Config and NestedArray, not needed anymore.

I'd say this is the 'good one'. Now waiting to see what happens with pre-requisite TypedData patch.

spearhead93’s picture

Status: Needs review » Needs work
FileSize
45.14 KB

This patch will not pass tests but it has been created after TypedData has been committed.

From http://drupal.org/node/1777268#comment-6532684, the definition of core/lib/Drupal/Core/TypedData/StructureInterface.php has been dropped from TypedData and the current patch relies on it.

Reinstating StructureInterface.php didn't fully help as a few functions from the initial patch on http://drupal.org/node/1777268 have been dropped and are used in the patch too (http://drupal.org/node/1777268 (drupal_get_data_type_info for example).

Jose Reyero’s picture

Status: Needs work » Needs review
FileSize
47.65 KB

@spearhead,
Thanks for the update but I was at the same time working on a different one.

This is a full update, now we have TypedData in core, that also properly implements (or tries to) ComplexDataInterface and ListInterface. It also has some better paremeter definition by using a ConfigDataInterface.

Jose Reyero’s picture

Status: Needs review » Needs work

The test produces a weird error which I cannot reproduce on my box so it may be related to the php version in the testbot (I am using PHP 5.3.10):

Fatal error: Can't inherit abstract function Drupal\Core\TypedData\ListInterface::isEmpty() (previously declared abstract in Drupal\Core\TypedData\ComplexDataInterface) in /var/lib/drupaltestbot/sites/default/files/checkout/core/lib/Drupal/Core/Config/Metadata/ConfigDataInterface.php on line 18
FATAL Drupal\config\Tests\ConfigMetadataTest: test runner returned a non-zero error code (255).

Help wanted!.

chx’s picture

Jose Reyero’s picture

@chx,
Thanks.

So yes, we could fix that two interfaces but we also can live without the ListInterface anyway so...

dagmar’s picture

Status: Needs work » Needs review
Jose Reyero’s picture

Status: Needs review » Needs work

Yes, the test finally passes. However I am improving the code building proper implementations of ComplexDataInterface and ListInterface, I will post the new patch today.

Jose Reyero’s picture

Status: Needs work » Needs review
FileSize
48.9 KB

And here it is the patch. Main changes:
- Moved around interface implementations and class names.
- All data objects are built now using the TypedData factory, added definitions in system_data_type_info().
- The parent/child relationships for nested elements are done now using the ContextAwareInterface

This is the class hierarchy now:

ConfigData (ConfigDataInterface)
-- ConfigDataElement (TypedDataInterface, ContextAwareInterface)
-- ConfigDataList (ListInterface)
-- ConfigDataNested
-- ConfigWrapper (ComplexDataInterface)

Lars Toomre’s picture

Is this ready for a review from a coding //documentation standards perspective? Or would you prefer me to wait @Jose Reyero?

One thing I noted from opening the patch file is that we need type hinting on all @param and @return directives.

Gábor Hojtsy’s picture

Reviewed the patch. Apart of the minor code style issues (some lines don't wrap to 80 chars, some files lack newlines at the end), the main question/concern I have is that we are still shooting at the big fancy goal of providing extensive metadata about configuration including reproducing some of the config form in a metadata .yml file for our configuration (in the form of type and label). We are not actually using this to build the form obviously, so developers just need to do it twice (or we are getting to a point where they'd use a code generator :/). However, I don't see where this patch solves marking strings as translatable, which was the trigger for this issue at the first place. Where/how are we going to solve that?!

Jose Reyero’s picture

@Lars Toomre,
Better wait, I'll be fixing most of it myself on next passes, I've learnt a lot from your latest reviews :-)

@Gábor,
Right, we were losing focus with this patch. Yes, we want to get translatable configuration in the first place.

This new version (refocusing, WIP):
- Implements Translatable interface, making it possible to get the configuration translated for any language.
- Extensive class renaming and some polishing of functions and interfaces.

WrapperInterface (TranslatableInterface) ****** which means all the classes in the hierarchy can be translated

WrapperBase (WrapperInterface)
    ElementBase (TypedDataInterface, ContextAwareInterface)
        ListElement (ListInterface)
              NestedElement
    ConfigWrapper (ComplexDataInterface)

- Translatable strings are marked as 'translatable:1' in the configuration metadata. Thus we can use a simple logic: If translatable and type string, we can translate default values with locale.
(The other option was creating a new data type for human readable strings, like 'uistrings', this question of how to mark these strings still open for discussion...)
- You can actually get translated configuration values. This is how:

// This should get a configuration data array only with the translated strings 
// (not "translatable" but actually translated).

$wrapper = config_wrapper('system.maintenance');

$strings = $wrapper->getTranslation($langcode)->toValue();

// Array ready to save into one of the locale config override files.
// Obviously we just want to do it only after first import of the configuration for now.

Major issue (marked with @todo in the patch):
----------------
The locale API needs some small improvements to be able to translate only default strings and also to be able to tell when a translation is actually a translation (and not the default string that happens to be the same). This is being worked out in #1777070: Refactor and clean up source string location handling

To do:
--------
- Once we get some agreement, mark the rest of the translatable strings at least for the examples.
- Add tests for this new translation feature.
- Cleaning up code and comments.

gdd’s picture

I had just assumed that anything 'string' would be translatable since everything else seemed to be getting a custom type (like 'path' for instance) but adding a translatable property does clarify things.

I probably need to re-review this patch because it seems a lot has changed while I was off dealing with life stuff. Last time I looked, I was starting to write tests, and to work on that I was trying to assemble some test metadata files for testing. This was really difficult to get right, I never did quite succeed. I'm a little concerned about that. On the one hand, most module authors will have very simple needs and in those cases it will be very simple. But for the more complex use cases it seems that writing the nested metadata files is going to be pretty complicated. I am going to ask the Views team to dig into this some since they will probably have the most intricate use case.

Gábor Hojtsy’s picture

@reyero: Thanks for covering the specific translation use case now! Looks much better! Now some review comments:

+++ b/core/lib/Drupal/Core/Config/Metadata/ListElement.phpundefined
@@ -0,0 +1,64 @@
+ * Defines a nested configuration metadata object that contains multiple
+ * elements that are themselves ElementBase objects or TypedData properties.

Shorten to 1 line :)

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataLookup.phpundefined
@@ -0,0 +1,102 @@
+ * The name of the configuration object may contain the placeholder '__' as the
+ * last component, which denotes generic metadata for all objects of this
+ * type. For the 'large' image style, for example, both "image.style.__" and
+ * "image.style.large" would be valid metadata files. In case both of these
+ * files exist, we load the generic metadata (in this example from
+ * "image.style.__") first and add the specific metadata (in this example from
+ * "image.style.large") on top of it.
+ */
...
+   * Filesystem marker for base metadata name.
+   */
+  const BASE_MARK = '%';
...
+    // If this is not the generic metadata (where the last component of the
+    // name is a '+'), try to load that first.
+    if ($last != self::BASE_MARK) {
+      return implode('.', array_merge($parts, array(self::BASE_MARK)));
+    }

+++ b/core/lib/Drupal/Core/Config/Metadata/NestedElement.phpundefined
@@ -0,0 +1,117 @@
+ * effects:
+ *   label: 'Style effects'
+ *   type: 'config_nested'
+ *   config_base: 'image.style.effects.*'
+ *   config_key: 'name'

Some of these docs/code are not in sync or are you in fact using 4 different types of special markers (%, +, __ and *)?

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataStorage.phpundefined
@@ -0,0 +1,50 @@
+    }
+  }
+}
\ No newline at end of file

Missing newline.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperInterface.phpundefined
@@ -0,0 +1,180 @@
+ * These may be nested structures mapping nested arrays of configuration data

Dot missing.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperInterface.phpundefined
@@ -0,0 +1,180 @@
+   * An array of metadata contains the 'label' and 'type' keys as explained above.
+   * If $key is omitted, the full array is returned, where the array keys are the

Slightly above 80 chars?

+++ b/core/lib/Drupal/Core/TypedData/Type/Undefined.phpundefined
@@ -0,0 +1,40 @@
+  /**
+   * Implements TypedDataInterface::validate().
+   */
+  public function validate() {
+    // TODO: Implement validate() method.
+  }

Missing.

+++ b/core/modules/image/meta/image.style.%.ymlundefined
@@ -0,0 +1,11 @@
+  config_key: 'name'
+  ¶
\ No newline at end of file

Missing newline, last line has 2 spaces.

+++ b/core/modules/system/meta/system.site.ymlundefined
@@ -0,0 +1,17 @@
+    label: 'Default 404 (not found) page'
+  front:
+    label: 'Default front page'
\ No newline at end of file

Missing newline.

@heyrocker: what kind of test coverage do you miss from the patch? As for strings as translatables, there are many things that are strings per say but not translatable. Eg. a node type (machine) name is a string, but not translatable; a path (eg. 404 page path) is a technically string too but not translatable, etc. Short of introducing special types for everything that is a string but not translatable, I think our better option is to just mark things as translatable specifically, which would avoid the magic behaviour and make the feature more explicit as well.

Jose Reyero’s picture

Well, my main concerns atm to be addressed first are these two:

1. The file format
I think it's just getting easier with each version of the patch. Atm it's a nested structure that maps exactly to the configuration but for pluggable objects.

@heyrocker,
Yes, I would like to see how a views looks like exported to configuration and also I think that would be the definitive proof for the configuration system (whether it fits our needs). So whatever issue is that they are working on (is there one?) it should be raised to highest priority.

2. How to mark translatable strings
While this 'translatable' flag kind of works, it may not be the better idea.
- Using generic 'string' is just not possible as it happens to be the data type of everything that is a PHP string and has not a more suitable type.
- We could use the same generic type for all of them like 'uistring' or 'readablestring' but this has the problem of not being extendable, I mean creating subtypes that need a tighter validation like 'label' (that should be a short plain text string)
- We can use more fine grained data types like 'label', 'text', 'mail_body', etc that are marked as 'translatable' in hook_data_type_info(). And these will be our translatable types from which locale will just handle the ones that are 'translatable' and of primitive type 'STRING'.

I think I'll go for this last one in next version of the patch, besides fixing the issues found by @Gábor in #93 and dropping most of the long explanations in the comments, which don't seem to be in any class atm. I'll be adding metadata for user emails too, to see how it looks like.

Gábor Hojtsy’s picture

Yes, I agree that decoupling the type from the translatability marker is a good step as done in the last patch. :)

dawehner’s picture

How to mark translatable strings
If i understand you correctly you want to get rid of "translatable" out of the metadata schema,
which would at least the DX experience seems to be worse. You would have to know a bunch of different data types
like label/text ... but on the other hand having proper validation on that level would for example is a huge win.
I guess label would be always translatable, so i'm wondering what should happen for administrative-only labels,
will they be translatable?

Yes, I would like to see how a views looks like exported to configuration and also I think that would be the definitive proof for the configuration system (whether it fits our needs). So whatever issue is that they are working on (is there one?) it should be raised to highest priority.

We are using configEntities since over a month and are running fine with that.
http://drupalcode.org/project/views.git/blob/refs/heads/8.x-3.x:/config/... is an example of that in real life.

I'm working on a longer comment/review, but just wanted to answer you the question.

gdd’s picture

I do not want to get rid of translatable out of the metadata. I was just thinking aloud mostly, and pointing out that things like 'path' and 'machine name' already had special types defined in the previous versions of this patch, so the only things left defined as 'string' were things that were translatable. However, I agree that having an explicit translatable property is better.

dawehner’s picture

Here is a try to map the current patch to the usecase of views without doing an actual review of the patch.
To sum it up in general, most of the stuff will work, but there are some details which might be problematic.

TL;DR: The patch seems to miss the feature to extend existing files, neither from another yml file or module.

  • One big file to store all it's configuration. We talked a bit about maybe split them up
    into different substructures, but the current approach is easy and works fine so far.
  • In this big file we have basic properties like $view->human_name, $view->name which is covered by this patch.
  • After that we have a nested substructure ("display") with predefined set of keys. You can have
    multiple displays per view, so you end up with $view->display[$id]... This is already covered by this patch,

    $display['display_name']
    $display['display_plugin']
    $display['display_options']

  • The display_plugin itself defines the different subkeys available in display_options. The base class
    itself defines A LOT of the, so from the DX experience it would be helpful to be able to extend/inherit the possible options instead of redefine these options for all of the plugins (this gets worse in other cases). It doesn't
    seem to be covered yet by this patch, but if views defines its own data types it might be possible to merge in other values?
  • Additional the display_options could be extended by other modules (independent from the display_plugin).
    This sounds similar to the node-types/comment-settings problematic, so i guess there will be a solution for that.
  • Under display_options you can have elements which again are a list. Let's take the fields example.
    A field plugin itself defines itself the config it has, comparable to the annotations for blocks (if you have seen that).
    So $display[%]['display_options']['fields'] contains a list of fields, which all can have different configuration.
    I guess you could solve this using views.view.%.display_options.fields.$certain_plugin.yml.
    Most of the time you have just a single addition to the baseClass of fields, so again there is the problem of extendabiltiy.

To sum up, we currently lack a way to extend metadata files.

Jose Reyero’s picture

@dawehner,
Thanks for the explanation, it clarifies a few things, though by this time I can tell you the views schema you have provided fits into the metadata schema without major issues. However I'm working on a few other improvements to the patch, so I'll post it together with a working metadata+views+translatable example.

About the issues you mention with extensibility and pluggable modules / objects, you are right about it cannot be really extensible but that's not an issue with the metadata, IMO it is an issue with Views itself or with the configuration system as a whole (if you take into account that node-types/comments etc...

The problem: I have no doubt that if you've got the right modules enabled they will handle all that views data by themselves. But there's another side of the workflow: Given a views configuration (that can be exported / imported), Can you figure out all that module dependencies? So: Views can be exported but not imported automatically, right? (yes, I've seen some module names in the config itself but besides incomplete that is really a poor solution IMO)

So, in short: The extendability issue is built in into the CMI/Views, etc... It's not metadata specific.
(Or: once you tell me how CMI/Views figures it out, we'll find a way for metadata to do the same)

My suggested solution: Everything that is 'pluggable' should by properly namespaced.
Example. Intead of 'my_nice_views_plugin', the name in the configuration itself should be 'mymodule.views.display.nice'

Please see how it is fixed for image style effects, it is included in the patch (though I was about to drop that since I've found a better solution for metadata that doesn't need it).

dawehner’s picture

Sure i read through the patch

About the issues you mention with extensibility and pluggable modules / objects, you are right about it cannot be really extensible but that's not an issue with the metadata, IMO it is an issue with Views itself or with the configuration system as a whole (if you take into account that node-types/comments etc...

Well the config system itself can handle the use-case without problems. If views could somehow inject another config wrapper class
it could handle the building of the different metadata for itself? Just thrown out that idea.

The problem: I have no doubt that if you've got the right modules enabled they will handle all that views data by themselves. But there's another side of the workflow: Given a views configuration (that can be exported / imported), Can you figure out all that module dependencies? So: Views can be exported but not imported automatically, right? (yes, I've seen some module names in the config itself but besides incomplete that is really a poor solution IMO)

If all modules are enabled you can figure this out, even there is no helper method for this yet. Maybe we could store
the needed modules in the config, but i agree if you have specified the metadata your patch will totally work! I have no doubt on that.

My suggested solution: Everything that is 'pluggable' should by properly namespaced.
Example. Intead of 'my_nice_views_plugin', the name in the configuration itself should be 'mymodule.views.display.nice'
Please see how it is fixed for image style effects, it is included in the patch (though I was about to drop that since I've found a better solution for metadata that doesn't need it).

You are right this at least would get it working and views use-case seems to be an edge-case.
Just to underline the inheritance problem: There are currently 54 field plugins provided by views, which all at least have this old metadata (+ some additional per plugin): http://drupalcode.org/project/views.git/blob/refs/heads/8.x-3.x:/lib/Dru...
A custom plugin currently just overrides defineOptions() and calls parent::defineOptions(). If you would support inheritance
you would not have to change all of them, once a single thing changes on the parent, if not you will end up with some maintenance problem,
especially for other contrib code.

So, in short: The extendability issue is built in into the CMI/Views, etc... It's not metadata specific.
(Or: once you tell me how CMI/Views figures it out, we'll find a way for metadata to do the same)

I agree, the extendability issue is totally seperated (even it works fine at the moment with CMI, because views just sets the config array as needed).

Jose Reyero’s picture

This new patch adds Views example config data and user mails into the equation (Since there's no views module yet, an example module with views data and metadata is in the config_demo module, http://drupal.org/sandbox/reyero/1635230 )

Changes:
- All metadata names are not pre-loaded so we don't rely anymore on config name to find the metadata. This allows us to simplify a few things: Drop the part of the patch for the image style names, not needed anymore.
- Metadata properties are now prefixed by a dot (.type, .label) so they don't clash with object properties of the same name (Views had a lot of 'type' and 'label' properties)
- Added nested element's support in the base class (for cases like user.mail.yml). Also we don't need that many 'nested' classes anymore, dropped some and renamed ElementBase to NestedElement (not abstract anymore).
- We don't need 'undefined' data type anymore since all the configuration properties are stored as 'string', so that will be our fallback data type.
- Added two new data types in system_data_type_info() for human readable strings: 'label', 'text'. These will be our translatable string types now.
- Added some minimal API to extract element's label from the metadata.

So smaller patch (from 57 kb down to 49 kb), more powerful, supports views. Attached screenshot from the example views configuration in #96 parsed with our metadata (produced by the config_demo module)

Jose Reyero’s picture

YesCT’s picture

Status: Needs review » Needs work

Disclaimer, I might not be understanding the standards, and/or it might be better to do a clean up later that is just standards stuff.
If this type of review is worth it now though, please point out where I've done misinterpretations, and I can continue to read through the patch later.

Oh, and, I can do many of these fixes and reroll for them.

+++ b/core/includes/config.incundefined
@@ -104,6 +106,36 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+ * @param $name

type recommended, like:
@param string $name

+++ b/core/includes/config.incundefined
@@ -104,6 +106,36 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+ *   The name of the configuration object for which we want to build a metadata object.

longer than 80 chars, wrap. http://drupal.org/node/1354#functions
use the pattern "which takes a", like:
The name of the configuration object for which we want to build a meta data
object, which takes a string.

+++ b/core/includes/config.incundefined
@@ -104,6 +106,36 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+ * @return Drupal\Core\Config\ConfigMetadata
+ *   An metadata object that contains the full metadata for the reguested name.

"A" metadata instead of An.
requested instead of reguested.
Use the ",which is an object..." pattern.

A metadata object that contains the full metadata for the requested name, which is an object of a class that implements the ConfigMetadata interface.

+++ b/core/includes/config.incundefined
@@ -104,6 +106,36 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+ * @param $name
+ *   The name of the configuration object to retrieve.

@param string $name
The name of the configuration object to retrieve, which takes a string.

(I'm trying to figure out how to word it like the standards doc says with the ", which takes a" pattern.)

+++ b/core/includes/config.incundefined
@@ -104,6 +106,36 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+ *   A configuration wrapper object.

The configuration wrapper object for the named object, which is a Wrapper.

(I'm trying to figure out how to specify the type of the object. Is it just Wrapper or is it ", which is a Drupal\Core\Config\Metadata\Wrapper.")

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigWrapper.phpundefined
@@ -0,0 +1,166 @@
+ * Defines the configuration wrapper object.

Is it usually defines "a" instead of defines "the"?

http://drupal.org/node/1353118

* Defines a ConfigWrapper

Should it specify what it defines using the same class name, or describe what a ConfigWrapper is?

http://drupal.org/node/1354#classes
Documenting classes and interfaces gives another pattern like:
* Defines a configuration wrapper object.
*
* ConfigWrapper implementation of ComplexDataInterface.

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigWrapper.phpundefined
@@ -0,0 +1,166 @@
+   * The configuration object itself.
+   *
+   * @var Drupal\Core\Config\Config

* The configuration object for this configuration wrapper's Config.
*
* @var Drupal\Core\Config\Config

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigWrapper.phpundefined
@@ -0,0 +1,166 @@
+   * @param Drupal\Core\Config $config

I think this should be the same type as the @var.

* @param Drupal\Core\Config\Config $config

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigWrapper.phpundefined
@@ -0,0 +1,166 @@
+   * Implements Drupal\Core\Config\Metadata\WrapperInterface::getKey().
+   */
+  public function getKey($key = '') {

Since this is a get, should this be:
* Returns no key...
hmm. Maybe that is why it's using the Implements pattern, because it doesn't return a key.

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigWrapper.phpundefined
@@ -0,0 +1,166 @@
+    // As this is the parent structure, there is no base key.
...
+    // As this is the parent structure, we get it right from the config object.
...
+    // This is the parent WrapperInterface so the key is empty.

Could change to be consistant:

// As this is the parent structure, the key is empty.
(or change the first couple to use // This is the parent WrapperInterface...)

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+      // If it is a NestedElement, it won't be set, but retrieved from

can wrap these lines more.

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigWrapper.phpundefined
@@ -0,0 +1,166 @@
+   * Implements Drupal\Core\Config\Metadata\WrapperInterface::setConfigData()
...
+   * Implements Drupal\Core\Config\Metadata\WrapperInterface::clearConfigData()

add period at the end of the sentence.

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigWrapper.phpundefined
@@ -0,0 +1,166 @@
+   * Implements Drupal\Core\TypedData\ComplexDataInterface::get().

How does it know it's the get for ComplexDataInterface and not for example WrapperInterface?

Would this be better written to specify the types like:
/**
* Returns the element.
*
* @param string $property_name
* Name of property, which takes a string.
*
* @return ComplexDataInterface object
* The element as an object of type ComplexDataInterface.
*/

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+      // If it is a NestedElement, it won't be set, but retrieved from
+      // the parent element. See NestedElement::setValue()

can wrap words more.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+   *   Translated string, that will be an empty string if it doesn't fit our requirements

wrap return comment

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+        // Metadata properties will be prefixed by a dot and will be scalar values.

more than 80 chars.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperInterface.phpundefined
@@ -0,0 +1,230 @@
+   * - 'elements.name', Configuration name for each of the nested elements,
+   *   may contain the placeholder '%' to replace by the element's key.
+   * - 'elements.key', Name of the configuration data property to be used as
+   *   the key for name replacement. It may contain the placeholder '%'
+   *   to be replaced by the element's key.

can be wrapped more.

+++ b/core/modules/system/meta/system.site.ymlundefined
@@ -0,0 +1,18 @@
+ ¶
\ No newline at end of file

newline at end of file

+++ b/core/modules/user/meta/user.mail.%.ymlundefined
@@ -0,0 +1,8 @@
+  ¶
\ No newline at end of file

new line again. is it needed?

YesCT’s picture

Status: Needs work » Needs review

didn't mean to change status yet. [edit: I got interrupted for a few hours, so just posted what I had noted so far. Did not get through the whole patch.]

Lars Toomre’s picture

I am glad @YesCT to see you do a detailed review of this patch. I had asked if it was appropriate yet in #89. @Jose Reyero said not yet.

I am not sure if this is the appropriate point yet, but perhaps you could include your comments in a re-roll of the patch along with a interdiff?

YesCT’s picture

Status: Needs review » Needs work

Continuing the review. Only got about half way though.

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigWrapper.phpundefined
@@ -0,0 +1,166 @@
+  /**
+   * Implements Drupal\Core\Config\Metadata\WrapperInterface::getKey().
+   */
+  public function getKey($key = '') {

chx helped me sort out that when extending an interface, the short form is ok without the @param and @return since php enforces the same types.

http://drupal.org/node/1354#classes
"Short documentation forms should be used for the overridden/extended methods (don't repeat what is documented on the base method, just link to it by saying "Overrides BaseClassName::method()." or "Implements BaseInterfaceName::method().", and then document any specifics to this implementation, leaving out param/return docs)."

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigWrapper.phpundefined
@@ -0,0 +1,166 @@
+    // Set the data into the configuration object but behave according
+    // to the interface specification when we've got a null value.

can wrap more.

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigWrapper.phpundefined
@@ -0,0 +1,166 @@
+  public function isEmpty() {
+    return !(bool)$this->toValue();

if it can't convert to a value, then it's true that it's empty? maybe add a comment unless this is a common pattern in core.

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigWrapper.phpundefined
@@ -0,0 +1,166 @@
+  /**
+   * Implements Drupal\Core\TypedData\TranslatableInterface::language()
+   */
+  public function language() {
+    $meta = $this->getElementDefinition();
+    // The default language will be English as this is hardcoded information.
+    $langcode = isset($meta['language']) ? $meta['language'] : 'en';
+    return new Language(array('langcode' => $langcode));
+  }

English, really? even if the site is installed in another language?

+++ b/core/lib/Drupal/Core/Config/Metadata/ListElement.phpundefined
@@ -0,0 +1,63 @@
+/**
+ * Defines configuration element wrapper that is a list of properties.
+ */
+class ListElement extends NestedElement implements ListInterface {
+  /**
+   * Implements ArrayAccess::offsetExists().
+   */

http://drupal.org/node/1354#classes says to "Leave a blank line between class declaration and first internal docblock, ..."

+++ b/core/lib/Drupal/Core/Config/Metadata/ListElement.phpundefined
@@ -0,0 +1,63 @@
+  public function isEmpty() {
+    return !(bool)$this->toValue();

here is another isEmpty as !(bool)...toValue();

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataLookup.phpundefined
@@ -0,0 +1,102 @@
+  /**
+   * Filesystem marker for base metadata name.
+   */
+  const BASE_MARK = '%';

I'm not sure if this agrees with http://drupal.org/node/1354#constants

I did find some similar ways of defining constant with "const ", for example in core/modules/system/system.module.

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataLookup.phpundefined
@@ -0,0 +1,102 @@
+  /**
+   * Constructs a ConfigMetadata cache object.
+   */
+  public function __construct() {
+    parent::__construct('config:metadata', 'cache');
+  }

Does it really construct a ConfigMetadata object, or is it a MetadataLookup object?

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataLookup.phpundefined
@@ -0,0 +1,102 @@
+  protected function resolveCacheMiss($offset) {
...
+  /**
+   * Get parent metadata name.
+   */
+  protected static function getBaseName($name) {
...
+      return implode('.', array_merge($parts, array(self::BASE_MARK)));

dont use short form since it's not an interface implementation. use the @param and @return to specify types.

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataLookup.phpundefined
@@ -0,0 +1,102 @@
+    // Get base metadata if available.
+    if ($basename = $this->getBaseName($offset)) {
+      $metadata = $this->offsetGet($basename);
+    }

is it a usual pattern to use a = assignment like this in an if?

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataLookup.phpundefined
@@ -0,0 +1,102 @@
+  /**
+   * Read metadata from file system.
+   *
+   * @return array
+   *   Metadata array if found or empty array if not.
+   */
+  protected function readMetadata($name) {

include @param for $name

also. says it returns a metadata array if found. but this just returns an empty array always.

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataLookup.phpundefined
@@ -0,0 +1,102 @@
+      // @todo Just log an error, but return empty array.

check for any (other) remaining @todo's

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataLookup.phpundefined
@@ -0,0 +1,102 @@
+}

http://drupal.org/node/1354#classes says to have "Leave a blank line ..., as well as before the closing brace of the class.

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataStorage.phpundefined
@@ -0,0 +1,89 @@
+class MetadataStorage extends FileStorage {
+  /**
+   * Metadata map indexing configuration names and modules.

missing blank line ("Leave a blank line between class declaration and first internal docblock, as well as before the closing brace of the class."

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataStorage.phpundefined
@@ -0,0 +1,89 @@
+  /**
+   * Overrides Drupal\Core\Config\FileStorage::__construct().
+   */
+  public function __construct() {
+    // Do nothing.
+  }

why override it if going to do nothing anyway?

similar is done in core/lib/Drupal/Core/Config/InstallStorage.php

class InstallStorage extends FileStorage {

/**
* Overrides Drupal\Core\Config\FileStorage::__construct().
*/
public function __construct() {
}

Hmm. in lib/Drupal/Core/Config/FileStorage.php it defaults to making a directory. ok.

is it worth suggesting: Change comment to Override default of making a directory and do nothing.

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataStorage.phpundefined
@@ -0,0 +1,89 @@
+  public function getFilePath($name) {
+    $metamap = $this->getMetadataMap();
+    $module = $metamap[$name];
+    // We are looking for a file in $module/meta/$name.yml
+    $directory = drupal_get_path('module', $module) . '/meta';
+    return $directory  . '/' . $name . '.yml';
+  }

Change comment to:
// We are looking for the file $module/meta/$name.yml.

(add a period, and you are not looking for a file "in" there, you are looking for that file, right?)

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataStorage.phpundefined
@@ -0,0 +1,89 @@
+  public function listAll($prefix = '') {
+    $names = array_keys($this->getMetadataMap());
+    if (!$prefix) {

should this if(!$prefix) really be if(!isset($prefix)) or maybe !exists or !empty(... ?

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataStorage.phpundefined
@@ -0,0 +1,89 @@
+      foreach ($names as $index => $name) {
+        if (strpos($name, $prefix) !== 0 ) {
+          unset($names[$index]);

add a comment to clarify this block.

if prefix is not at the beginning of the name, dont include it?

I looked up the StorageInterface listAll, and it lists all configuration object names that start with a given prefix. I was wondering if this listed all keys, or listed all datamaps... might not be the problem of this patch though.

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataStorage.phpundefined
@@ -0,0 +1,89 @@
+   * Returns a map of all metadata names and the module they belong to.
+   *
+   * @return array
+   *   Array mapping metadata names with module names,
+   */

comment describing return array is unfinished.

http://drupal.org/node/1354#functions

gives example of " * @return string
* Description of the return value, which is a string."

how to use that here?
* @return array
* Array mapping metadata names with module names, which is an array.

that seems awkward though.

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataStorage.phpundefined
@@ -0,0 +1,89 @@
+  protected function getMetadataMap() {

I could mostly follow the building of this mapping, but I think adding an example would help.

+++ b/core/lib/Drupal/Core/Config/Metadata/NestedElement.phpundefined
@@ -0,0 +1,200 @@
+class NestedElement extends WrapperBase implements TypedDataInterface, ContextAwareInterface {
+  /**

missing blank line.

+++ b/core/lib/Drupal/Core/Config/Metadata/NestedElement.phpundefined
@@ -0,0 +1,200 @@
+  public function getKey($key = '') {
+    return $this->key . ($key ? '.' . $key : '');
+  }

huh? if no arg, then $key is '', then it tests if key is set, and if it's not, it makes it ''?

add a comment and/or break this into more readable steps.

+++ b/core/lib/Drupal/Core/Config/Metadata/NestedElement.phpundefined
@@ -0,0 +1,200 @@
+  public function getConfigData($key = '') {
+    return $this->getParent()->getConfigData($this->getKey($key));
+  }

this's config data is the same as the parent's?

+++ b/core/lib/Drupal/Core/Config/Metadata/NestedElement.phpundefined
@@ -0,0 +1,200 @@
+  public function setConfigData($key, $value) {
+    $this->getParent()->setConfigData($this->getKey($key), $value);
+  }

or does it mean that the parent stores the config data? going back to look at the whole class... it's about the parent. Is parent another word for configuration wrapper?

+++ b/core/lib/Drupal/Core/Config/Metadata/NestedElement.phpundefined
@@ -0,0 +1,200 @@
+  public function getConfigKey() {
+    $parent = $this->getParent()->getConfigKey();
+    return ($parent ? $parent . '.' : '') . $this->getKey();
+  }

add a comment. I think this does: return $parent.key if there is a parent otherwise just return the key.

+++ b/core/lib/Drupal/Core/Config/Metadata/NestedElement.phpundefined
@@ -0,0 +1,200 @@
+  public function set($property_name, $value) {
+    // Set the value in the parent object that holds the Config.
+    $this->getParent()->set($this->getKey($property_name), $value);

"the Config." sounds like an abbreviation, is it? I did a grep -R "the Config" * and didn't find much. Maybe the comment should be:
// Set the value in the parent object that holds the configuration element.

+++ b/core/lib/Drupal/Core/Config/Metadata/NestedElement.phpundefined
@@ -0,0 +1,200 @@
+  /**
+   * Implements Drupal\Core\TypedData\ContextAwareInterface::getName().
+   */
+  public function setName($name) {

in the comment, it should be setName()

+++ b/core/lib/Drupal/Core/Config/Metadata/NestedElement.phpundefined
@@ -0,0 +1,200 @@
+      $this->setTranslation($langcode, (bool)$parent->isTranslation(TRUE));

isTranslation has to be typecast?

+++ b/core/lib/Drupal/Core/Config/Metadata/NestedElement.phpundefined
@@ -0,0 +1,200 @@
+   * Implements Drupal\Core\TypedData\TranslatableInterface::language().
+   */
+  public function language() {
+    return $this->getParent()->language();

I wonder why that isn't getLanguage() in core/lib/Drupal/Core/TypedData/TranslatableInterface.php

+++ b/core/lib/Drupal/Core/Config/Metadata/NestedElement.phpundefined
@@ -0,0 +1,200 @@
+  }
+}
diff --git a/core/lib/Drupal/Core/Config/Metadata/WrapperBase.php b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.php

missing blank line at the end of the class

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+abstract class WrapperBase implements IteratorAggregate, WrapperInterface {
+  /**

missing blank line before first docblock.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+   * The language code for which this is a translation.
+   *
+   * @var string
+   */
+  protected $translation;

this is the translation, or the langcode for the translation? maybe var name should be $translation_langcode

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+   * Builds properties using configuration data and metadata.
...
+  protected function buildElement($key) {

comment says it builds properties, function is called buildElement. Is an element a property? Is a property and element? Maybe improve the comment and/or the function name. Is it just building the property/element, or is it adding translation information to an already existing element definition?

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+   * @param string $key
+   *   Configuration key.

add: , which takes a string.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+   * @result Drupal\Core\TypedData\TypedDataInterface|null
+   *   A fully built TypedDataInterface object.

based on http://drupal.org/node/1354#functions
example:
" * @param string|null $string_or_null
* (optional) Description of this optional parameter, which takes a string or
* can be NULL.
*
* @return string
* Description of the return value, which is a string."

and the note "If a function may return NULL in some but not all cases, then document this possibility. For instance, if the return value may either be a string value or NULL, then use @return string|null followed by a description of the circumstances in which each may be returned."

change comment to
A fully built object which is a TypedDataInterface or NULL if property element failed the translation stage.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+   * @param mixed $value
+   *   Element value.
+   * @param array $metadata;
+   *   Element's metadata.
+   *
+   * @return bool
+   *   Whether the element fits the translation criteria.

add the ", which takes a" or ", which is a" patterns to clarify the types.

This function translates the element? Looks like it translates the element (sets the value to be the translation) if its data is a translatable type... but also does checking to see if it's translatable, and really returns wether it was translated or not. Is that the same as returning wether the element fits the translation criteria?

Actually this whole function is confusing. Add more comments.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+    // The element doesn't have a translation, if strict mode we drop it.
+    return $this->isTranslation(FALSE);

For example, here, where does strict come in? maybe it should be return $this->isTranslation($strict=FALSE); hmm. What does drop it mean? Does it mean we did not translate it, so return FALSE?

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+    return is_string($value) && in_array($type, array('label', 'text'));

Add a comment that says only strings that are a label or text are translatable. And maybe point to the issue where it's discussed that all config is of type "string".

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+   * Gets translation for string using localization system.
...
+  protected function translateString($source, $metadata) {

change comment to Translates string using...

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+   * @todo Extend the locale API to make the difference between default
+   * and changed strings.
...
+   * So far we only know how to translate strings from English so only
+   * English strings should be marked as translatable for now.

These can wrap more.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+   * @param string $source
+   *   The source string
+   * @param array() $metadata
+   *   The element's metadata that may contain a 'string_context'

array() should just be array. And add ", which takes a " pattern to specify the type.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+   *   Translated string, that will be an empty string if it doesn't fit our requirements

comment longer than 80 chars.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+    if ($source && $this->language()->langcode == 'en') {

if ($source) huh? $source is a string, not a bool. Should this be: if (!empty($string).. ?

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+      // We are only interested in actual translations.
+      if ($translation != $source) {

this makes it sound like sometimes a translation will contain something like happy translates to happy, where the translated string might be the same as the source string. In that case, this string would show as "not translated"? would that accidentally imply that it needs to be translated? when really it's the same in both languages?

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+    if (isset($data) && $element = $this->buildElement($first)) {

add comment
// Only build the element if it has config data.

add comment
// If the element could be built, check for the key.
// If it's not a subkey return the element

but why only return it if it's not a key?

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+      if (!$subkey) {
+        return $element;
+      }
+      elseif ($element instanceof WrapperInterface) {
+        return $element->getElement($subkey);
+      }

Is this checking to see if it's the element or a wrapper, and if it's a wrapper, get the actual element to return?

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.phpundefined
@@ -0,0 +1,395 @@
+      if (strpos($name, '.') === 0) {
+        // Metadata properties will be prefixed by a dot and will be scalar values.

comment longer than 80 chars.

also, maybe say here what scalar is: containing an integer, float, string or boolean

also, why the ===? === tests to see if it's equal and the same type. So is this checking if it's an integer? Is 0 an integer? I'll think on that some more, it's checking if strpos is the same type as 0, not if the property is.

Jose Reyero’s picture

@YesCT,
Thanks for the review, I'll add most of your suggestions in the next re-roll. About the questions, some will be answered in the code comments.

I'm working on the locale side atm, so hopefully next version of this will have the locale API side sorted out (there are some @todo's in the code atm), #1777070: Refactor and clean up source string location handling
Also working on adding in some more feedback from the Views team.

YesCT’s picture

Jose, Do you have something new to post, or should I finish the review of the code in #101?

Jose Reyero’s picture

Title: Add metadata for configuration in YML files » Add metadata for configuration in YML files and use it for translating configuration.
Status: Needs work » Needs review
FileSize
60.59 KB

Finally, translating configuration data!!

This patch is a development of the previous one and it actually translates the configuration strings:
- When module is enabled.
- When new translations are imported.
- When translations are updated (easier to test).
Due to performance issues (we need the patch below to fix that) when translations are refreshed, only a few objects are updated on each following page request for that language.

Also done another important cleanup, fixed most of the issues pointed out by @YesCT above, which includes renaming many functions for a clearer purpose. Though I guess I may have introduced new ones, sorry...

Some of the texts you can see usually translated include user emails (after you load a translation in your language). Or you can add/edit translations using the locale translation UI, which will be updated in the configuration (slowly, a few on every request).

Pending:
- Add tests for translation part (you can try it manually for now).
- There are some ugly parts in the new code yet so it may need some more polishing
- For proper performance, it would need #1777070: Refactor and clean up source string location handling
- Cleanup of locale.config.inc once we can use that patch (there are some unused functions there atm).

xjm’s picture

Thanks @YesCT for the reviews!

Couple suggestions for subsequent reviews: in #1809354: [Policy, no patch] Clarify point 2 of the documentation gate, we hashed out which points are (and are not) part of the documentation gate (i.e., reasons to mark an issue NW based on its documentation). Edit: See http://drupal.org/core-gates#documentation-block-requirements. In #1791872-58: [Policy, no patch] Add special tag to identify issues ready for a high level only review I proposed some guidelines for reviewing code style. Going forward I think providing a quick reroll with just the code style cleanups, an interdiff, and a brief summary, and then doing a full review on the rerolled patch, might help with the spinach problem and make it easier for @Jose Reyero and other reviewers to see the important points to address.

YesCT’s picture

Status: Needs review » Needs work

@xjm: Thanks. That really makes sense.

YesCT’s picture

FileSize
61.21 KB

In case it's helpful, an interdiff. Although, it's as big as the patch, which is probably why @Jose Reyero didn't bother. :)

YesCT’s picture

Status: Needs work » Needs review
FileSize
6.83 KB
60.61 KB

In preparation to do a more functional review, some clean up, mostly grammar or style to #109. I was careful to only change lines this patch had already changed/added, so didn't fix style of lines unrelated to this issue.

Gábor Hojtsy’s picture

Status: Needs review » Needs work

#1777070: Refactor and clean up source string location handling got committed, so should return here to reroll?

fago’s picture

I'm wondering how this relates to config entities and potentially using their translation API? IS there a reason we need this besides configuration entities?

gdd’s picture

We will definitely need this for all configuration, regardless of whether it is an implementation of ConfigEntity or not. For instance, things like the site slogan are basic static settings, but still need to be marked as translatable. I had already thinking that some helpers and sugar could be added to ConfigEntity to help manage the metadata, but it still needs to be managed outside of them too.

Jose Reyero’s picture

Reworked patch:
- Implements proper use of string locations, as the feature has been committed.
- Reworked ConfigWrapper (once again). Translation features are now on LocaleConfigWrapper, which allos simplifying the metadata classes a bit. Also having translation in locale cleans up some dependencies.
- Reworked translator that should cause some performance boost as configuration strings are now preloaded (using new locations, yup)
- Implements full workflow (configuration is translated and updated) for: module install / uninstall, language creation, string import, string UI translation...
- Some more metadata added: views. It works for views without any issue. See full views metadata using (updated), http://drupal.org/sandbox/reyero/1635230

To do:
-------
- Code cleanup. Merge cleanup from #114 (sorry, missed that for this patch)
- Clean up batch processes, not totally happy with them atm.
- Clean up workflow on some edge cases (upgrade, updates).
- Update issue description which is now mostly obsoleted.

Jose Reyero’s picture

Status: Needs work » Needs review
dawehner’s picture

Status: Needs review » Needs work

As already told heyrocker on IRC, i think it would make sense to not include the views part in the initial patch.
Reasons for that are that views is not ready to 100% support it, #1810480: Provide the plugin_id to support views metadata integration and doing all the meta yml files would increase the initial patch quite a bit.

Gábor Hojtsy’s picture

Status: Needs work » Needs review

@dawhener: I think that is true for other things as well, this patch only includes some sampling of metadata, and the rest will need adaptation patches, but those are not new features :) The important part is that this should be a useful system for the complexity of Views config even.

Jose Reyero’s picture

Re rolled the patch:
- Dropping views metadata as suggested by @dawehner and @Gábor Hojtsy in previous comments.
- Merged in code review by @YesCT, #114

gdd’s picture

I have been looking through the code some today, and I'm going to continue to over the weekend. One thing I know would help me would be if the issue summary can get updated and some sample code added. Things have really changed a lot since last I looked. It would probably help getting some other reviews as well. Thanks for keeping on this though, its well appreciated and I'm really glad someone has taken the lead on these issues.

Jose Reyero’s picture

One more reroll doing the code clean up mentioned in #119. Batch processes should be way cleaner now:
- Split batches into two different ones: one for importing translations, one for updating configuration.
- Added one more install step for updating configuration translations (only when selecting another language).
- Reusing Config/InstallStorage instead of duplicating all the stuff, this saves quite a lot of code.
(Had to change InstallStorage a little bit, as it was making the assumption of the first part of the config name being the module name, which I think is wrong so this patch should be fixing that issue too.)

So I think the code is ready now, shorter and cleaner, and yes, I'll be updating the description over the weekend.

Jose Reyero’s picture

Issue summary: View changes

Added the Views story from #30

Jose Reyero’s picture

Issue summary: View changes

Updating and explaining metadata format

gdd’s picture

Issue tags: +feature freeze

Tagging for feature freeze

fago’s picture

We will definitely need this for all configuration, regardless of whether it is an implementation of ConfigEntity or not. For instance, things like the site slogan are basic static settings, but still need to be marked as translatable. I had already thinking that some helpers and sugar could be added to ConfigEntity to help manage the metadata, but it still needs to be managed outside of them too.

I could see the simple settings being each own config entity as well. Anyway I think it's perfectly fine to have that separately - that plays well with the idea of the typed data api. I was just wondering whether it's really something required.

So here a review mostly from the typed data API perspective.

+++ b/core/modules/system/system.module
@@ -2090,6 +2090,31 @@ function system_data_type_info() {
+      'primitive type' => Primitive::STRING,
+      'translatable' => TRUE,

The translatable key isn't supported here - it's currently only supported for entity field definitions - see EntityStorageControllerInterface::getFieldDefinitions On that level it's just the interface of the class. It's used by the entity API though to define which fields are translatable.

+++ b/core/modules/system/system.module
@@ -2090,6 +2090,31 @@ function system_data_type_info() {
+    'text' => array(
+       'label' => t('Text'),
+       'description' => t('Textual content.'),
+       'class' => '\Drupal\Core\TypedData\Type\String',
+       'primitive type' => Primitive::STRING,
+       'translatable' => TRUE,
+    ),

This seems to do duplicate string (in particular as translatable is not supported here). Where is the difference to 'string'?

Why not just use
.translatable => TRUE
in the metadata with type string? That would go inline how the entity system makes use of that and make sure all the metadata is written to the metadata files.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.php
@@ -0,0 +1,293 @@
+ * Base class for all configuration data wrappers.
+ */
+abstract class WrapperBase implements IteratorAggregate, WrapperInterface {

This looks very similar to the TypedDataInterface and it's TypedData(base) class. I'm wondering whether there is actually a difference or whether there would be the possibility to merge those?

As far as I can see the $metadata plays exactly the role of the definition, so why not make $metadata and typed data definition one and the same? Typed data definitions are in fact metadata. setData() seems to map to setValue().

I realize that you may want to add your one stuff to the metadata on top of what the typed data API defines. That's perfectly possible and done by the entity system as well (see translatable), i.e. you may define any new key for typed data definitions which then may be used in all of your typed data definitions. That way, you can make sure your metadata is as expressive as you need it for config use cases, while anything working on top of the typed data API only can rely on typed data API keys only.

So, e.g. you could add a 'translatable' key as done for field definitions, plus an 'elements' or 'children' key that contains the metadata definitions for all children elements and so makes metadata == typed data definitions?

Of course, you could still add your own interface that adds configuration wrapper related utility functions, as the ones taking $key as argument. But it should be limited to functionality that's not covered by the TypedDataInterface yet.

+++ b/core/lib/Drupal/Core/Config/Metadata/WrapperBase.php
@@ -0,0 +1,293 @@
+  /**
+   * Implements Drupal\Core\Config\Metadata\WrapperInterface::toArray().
+   */
+  public function toArray() {

This is getPropertyValues() for complex data.

+++ b/core/modules/image/meta/image.style.%.yml
@@ -0,0 +1,11 @@
+  .type: config_list
+  .elements.name: 'image.style.effects.%'
+  .elements.key: '%.name'

So if I got this right this refers to another config-metadata. Having a special 'config_*' data type that is able to resolve metadata defintions of any kind is perfectly fine for the typed data API as well. The typed data way of doing so would be to add a key below the 'constraints' key to the definition (as elements.name) and to return the right definitions in getPropertyDefinitions(). For example, exactly that is done by the EntityReferenceItem class (although the run-time defintion customiazions are limited to add in the referenced $entity_type only. But you can add in whatever you want there.)

So you could support doing

$typed_config = typed_data()->create(array(
  'type' => 'config_element',
  'constraints' = array(
     'elements_name' => 'image.style.effects',
  ), $value);

The same notation could live in the metadata files:

name:
  label: 'Machine name'
  type: string
  translatable: true
label:
  label: 'Label'
  type: string
  translatable: true
effects:
  label: 'Style effects'
  type: config_element
  list: true
  constraints :
    elements_name: 'image.style.effects'

I don't know all the details and so whether this is really feasible - I just want to throw the idea here. Hopefully it's useful for you.

+++ b/core/modules/system/system.module
@@ -2090,6 +2090,31 @@ function system_data_type_info() {
+    'config_list' => array(
+      'label' => t('Configuration list'),
+      'description' => t('List of configuration data'),
+      'class' => 'Drupal\Core\Config\Metadata\ListElement',
+    ),
+    'config_element' => array(
+      'label' => t('Nested configuration'),
+      'description' => t('Nested configuration data'),
+      'class' => 'Drupal\Core\Config\Metadata\NestedElement',
+    ),

Having config_element sounds fine, I'm not sure about the list type though. Usually, a list of items is handled by having the 'list' key set to TRUE. Then you'll get an instance of the specified 'list class', whereas the individual items are of the specified 'type'. As ListElement implements the typed data ListInterface I think it should follow that and specify the ListElement as 'list class' for the config_element.

effulgentsia’s picture

I could see the simple settings being each own config entity as well.

That's a whole can of worms, but if it's worth discussing, let's do so in #1828354: Should singular config objects be instances of a class that extends Entity?.

Gábor Hojtsy’s picture

Sit down with @fago and @heyrocker reviewing the class structure. Documented here https://docs.google.com/drawings/d/162NU_5QQt6hKzdiqmfwGGHv6d-5c8RBC6JOx..., screenshot:

CMI Metadata architecture.png

Gábor Hojtsy’s picture

Have also been trying to look at the metadata (nesting) structure, especially the more complex example of views (not in the current patch but agreed it is useful to inspect for understanding how this works). A detailed overview of how the nesting happens there and what the .elements.id, .elements.key, .elements.label mean and when should they be used together and separately would be great. There is even an elements.name (without a leading dot!) in the diff...

http://drupalcode.org/sandbox/reyero/1635230.git/commitdiff/97f12c8

So a detailed explained example (think documentation :) on how this would work would be important for people to understand how this works...

Jose Reyero’s picture

Updated patch addressing all issues raised by @fago in #129. Main changes:
- Cleaner and more consistent implementation of TypedData API
- The configuration data is now contained in two TypedDataInterface classes: ConfigElement for arbitrary structures, which implementes ComplexDataInterface, and ConfigList for lists (ListInterface). Both of them correspond to the 'config_element' type.
- Moved definition properties, 'elements.key', 'elements.name', etc.. into 'list settings' (added 'label' key into 'settings'). All of them properly documented in the TypedData classes (ElementBase, ElementList). Simplified elements_key, elements_label, which now don't need any key substitution (from '%.name' to 'name').

The class structure is now like this (I think much cleaner):


ElementInterface extends TypedDataInterface, ContextAwareInterface...

ElementBase extends TypedData implements ElementInterface (TypedDataInterface, ContextAwareInterface)   
    ListElement implements ListInterface
    ConfigElement implements ComplexDataInterface

ConfigWrapper extends Config (so we can reuse stuff from this one).
    LocaleConfigWrapper implements TranslatableInterface  

Metadata (contains only some utility static functions)

@fago,
About 'translatable' property, as it is discussed above (long discussion) and I think we talked in Munich D8MI sprint, we don't want the 'translatable' property to be 'hardcoded' in each module. Instead we need some recognizable data types that locale (or other contrib modules) may want to translate.
Or in other words, it's not up to the module developer which properties are translatable, but for the site builder to decide.
As 'string' is way too generic ends up being used for anything that has no a more specific type (uuid, machine_name, language code, etc, etc, etc...) we do need some specific data type that we can match to 'human readable / end user strings' and these are the ones locale module cares about. These are atm 'label', 'text', we'd need at least one, though ideally we'd have as many as text translation formats we need to handle. I.e. we may allow 'html' for 'text' translations but not for 'label' translations, etc...

So: Could we consider adding 'translatable' to the TypedData element definitions? Because having the types hardcoded in locale, which is the other option I can think of, provides very little extendability.

@Gábor,
Very nice picture, sorry I've just changed most of it (though the new one should be easier :-) )

I'll be online next days on the IRC.

chx’s picture

Status: Needs review » Needs work

To sum up: I read the patch. I did read it. I have no idea what did I read. I am very confused, mostly. So, mostly nitpicking.

+++ b/core/includes/config.incundefined
@@ -104,6 +106,36 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+  $metadata = &drupal_static(__FUNCTION__);
+  if (!$metadata) {
+    $metadata = new MetadataLookup();

This pattern cries for a registration of the class in the container.

+++ b/core/includes/config.incundefined
@@ -104,6 +106,36 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+  return new ConfigWrapper($name, config($name)->get());

config() is a wrapper around a service from the dependency injection container. Is this something which should be in the container as well? Perhaps with a factory?

+++ b/core/includes/install.core.incundefined
@@ -1670,6 +1676,20 @@ function install_import_translations_remaining(&$install_state) {
+  include_once drupal_get_path('module', 'locale') . '/locale.bulk.inc';

Hrm, isn't this a module_load_include?

+++ b/core/lib/Drupal/Core/Config/InstallStorage.phpundefined
@@ -90,6 +89,74 @@ public function rename($name, $new_name) {
+      foreach ($names as $index => $name) {
+        if (strpos($name, $prefix) !== 0 ) {
+          unset($names[$index]);

I would much, much rather write something like

$return = array(); 
foreach ($names as $index => $name) {
  if (strpos($name, $prefix) === 0 ) {
    $return[$index] = $names[$index];
  }
}
return $return;

this inversion of logic makes it really hard to understand what happens here.

+++ b/core/lib/Drupal/Core/Config/InstallStorage.phpundefined
@@ -90,6 +89,74 @@ public function rename($name, $new_name) {
+  protected function getComponentFolder($type, $name) {

getComponentConfigFolder, perhaps?

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigWrapper.phpundefined
@@ -0,0 +1,141 @@
+   *   A string that maps to a key within the configuration data.

Maps to? Like how? What's the mapping?

+++ b/core/lib/Drupal/Core/Config/Metadata/ElementBase.phpundefined
@@ -0,0 +1,259 @@
+    return is_array($this->value) ? array_keys($this->value) : array();

What else can $this->value be if not an array? Perhaps change to protected $value = array() and simplify that way?

+++ b/core/lib/Drupal/Core/Config/Metadata/ElementBase.phpundefined
@@ -0,0 +1,259 @@
+    return $parent || is_numeric($parent) ? $parent . '.' . $this->key : $this->key;

OK, that's one tricky way to express $parent || $parent === '0' || $parent === 0 || $parent === 0.0 but are we *sure* this is what we wanted to say here?

+++ b/core/lib/Drupal/Core/Config/Metadata/ElementInterface.phpundefined
@@ -0,0 +1,94 @@
+   * - Any of the keys used for typed data definition prefixed by a dot.
+   * - Any nested array that will keyed by configuration key and will be mapped

This needs more and detailed examples. Almost every word in these sentences need an example. I can't make any sense out of them.

+++ b/core/lib/Drupal/Core/Config/Metadata/ElementInterface.phpundefined
@@ -0,0 +1,94 @@
+   *     .list: '1'

What is this? Is .list something special? Are you declaring something a list? But what?

+++ b/core/lib/Drupal/Core/Config/Metadata/ListElement.phpundefined
@@ -0,0 +1,71 @@
+      if (isset($settings['elements_label']) && !isset($meta['.label'])) {

So .label is special. Where is this documented? What other special strings exist?

+++ b/core/lib/Drupal/Core/Config/Metadata/Metadata.phpundefined
@@ -0,0 +1,85 @@
\ No newline at end of file

Hrm :)

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataLookup.phpundefined
@@ -0,0 +1,112 @@
+    catch (Symfony\Component\Yaml\Exception\ParseException $e) {

This needs a use statement.

+++ b/core/lib/Drupal/Core/Config/Metadata/MetadataStorage.phpundefined
@@ -0,0 +1,31 @@
+  protected function getComponentFolder($type, $name) {

Ah then my remark about was incorrect, this wants to be getComponentMetaFolder.

+++ b/core/modules/locale/lib/Drupal/locale/LocaleConfigWrapper.phpundefined
@@ -0,0 +1,255 @@
+    return empty($options['strict']);

This is the first time I see a strict mode. Where is this coming from?

+++ b/core/modules/locale/lib/Drupal/locale/StringBase.phpundefined
@@ -189,7 +189,7 @@ public function save() {
-        '@string' => $string->getString()
+        '@string' => $this->getString()

Is this a separate bugfix?

+++ b/core/modules/locale/locale.bulk.incundefined
@@ -400,6 +411,7 @@ function locale_translate_batch_build($files, $options) {
@@ -408,12 +420,14 @@ function locale_translate_batch_build($files, $options) {

@@ -408,12 +420,14 @@ function locale_translate_batch_build($files, $options) {
+        'operations'    => $operations,

Indentation is wrong -- it seems it was correct before?

+++ b/core/modules/locale/locale.moduleundefined
@@ -37,27 +37,27 @@
+    \{              # match object literal start

Again a wrong indent..?

+++ b/core/modules/locale/locale.moduleundefined
@@ -146,49 +146,49 @@ function locale_help($path, $arg) {
+      'title' => 'User interface translation',

And again..?

+++ b/core/modules/locale/locale.moduleundefined
@@ -211,10 +211,10 @@ function locale_permission() {
+      'locale_translate_edit_form_strings' => array(

Same.

Gábor Hojtsy’s picture

I'll try to update the figure tomorrow as soon as possible.

Jose Reyero’s picture

Status: Needs work » Needs review
FileSize
83.28 KB
77 KB

Fixed most issues found by @chx #134 (thanks for the review) and done some additional cleanup including documentation:
- Moved full documentation for metadata to config_metadata(). Posted below.
- Removed Metadata helper class, merged the static functions into ElementBase.
- All that locale module's indentation issues were by mistake, removing them makes the patch considerably shorter (and that's why the interdiff refers to #119 instead of the previous version).

About:

+++ b/core/modules/locale/lib/Drupal/locale/StringBase.phpundefined
@@ -189,7 +189,7 @@ public function save() {
-        '@string' => $string->getString()
+        '@string' => $this->getString()

Yes, it is a separate bugfix, a pretty straight forward one though.

About config_metadata() function not using DIC, since it is used for the installer too, that would require adding it in at least three different places, I think we can postpone this one for future cleanups.

This is the merged and expanded documentation in the config_metadata() function.

/**
 * Retrieves metadata for a configuration object or key.
 *
 * The metadata contains the typed data definition for entries in the
 * configuration data and follows the same nested array structure as the
 * configuration data it describes.
 *
 * For each element in the array, the element's metadata is an array that may
 * contain the following elements:
 * - Any of the keys used for typed data definition prefixed by a dot.
 * - Any nested array that will keyed by configuration key and will be mapped
 *  to a nested ElementInterface or TypedDataInterface element.
 *
 * The dot prefix allows mixing in the same array the element's definition
 * and the element's children that may contain keys with the same names.
 *
 * Example metadata from module/system/meta/system.site.yml
 * This is the metadata definition for module/system/config/system.site.yml
 *
 *   name:
 *     .label: 'Site name'
 *     .type: 'label'
 *   mail:
 *     .label: 'Site mail'
 *   slogan:
 *     .label: 'Site slogan'
 *     .type: 'text'
 *   page:
 *     .list: '1'
 *     403:
 *       .label: 'Default 403 (access denied) page'
 *       .type: 'string'
 *     404:
 *       .label: 'Default 404 (not found) page'
 *       .type: 'string'
 *     front:
 *       .label: 'Default front page'
 *       .type: 'string'
 *
 * In this example, for a given entry in the configuration data we can have a
 * corresponding one in the metadata. For site name (configuration key 'site'),
 * this is the metadata:
 *
 *   .label: 'Site name'
 *   .type: 'label'
 *
 * That would be translated in the following element definition:
 *    array(
 *      'label' => 'Site name',
 *      'type' => 'label'
 *    );
 *
 * Elements that don't have an explicit type will default to the type 'string'
 * unless they have children elements (like 'page' in the example) or the
 * 'list' property set to true, in which case they will default to the type
 * 'config_element'.
 *
 * Nested lists of elements, that contain additional metadata for each of the
 * elements in a different file are also supported.
 *
 * Example metadata from module/image/meta/image.style.%.yml that contains the
 * metadata for all the default image styles in module/image/config.
 *
 *   name:
 *     .label: 'Machine name'
 *     .type: string
 *   label:
 *     .label: 'Label'
 *     .type: label
 *   effects:
 *     .label: 'Style effects'
 *     .list: '1'
 *     .list settings:
 *        elements_name: 'image.style.effects.%'
 *        elements_key: 'name'
 *
 * In this example, for each of the image style effects we have additional
 * 'list settings' meaning for each of the elements we'll use the 'name'
 * property (Example 'image_scale') to find additional metadata with the
 * name 'image.style.effects.%' (Example image.style.effects.image_scale.syml)
 *
 * @see Drupal\Core\TypedData\TypedDataManager::create()
 * @see Drupal\Core\Config\Metadata\ElementBase
 * @see Drupal\Core\Config\Metadata\ListElement
 *
 * @param string $name
 *   The name or key of the configuration object.
 *
 * @return Drupal\Core\Config\ConfigMetadata
 *   A metadata array.
 */
function config_metadata($name) {
..
}

Jose Reyero’s picture

Issue summary: View changes

Added implementation detail and locale workflow

Lars Toomre’s picture

@Jose Reyero: If and when this next gets re-rolled, all @see directives need to follow the @return directive with a blank line in between.

Gábor Hojtsy’s picture

Updated the diagram for the patch in #133. I hope it did not change architecture dramatically again (yet) so my work can be useful. There are lots of new things and wholly new structures here... Compare it with the diagram above... Link to this one: https://docs.google.com/drawings/d/10GfN2Fpxq4W340BZg5U9GSREWj6U6FadBE_e...

New CMI Metadata architecture.png

Gábor Hojtsy’s picture

Rearranged a bit and updated for the change in #136 (Metadata removed). Should be fully up to date as of #136. Google URL still the same as above.

New CMI Metadata architecture (1).png

effulgentsia’s picture

I read the issue summary and patch, but not the comments, so apologies if anything here is redundant.

+++ b/core/modules/system/meta/system.site.yml

Can this be moved into the config directory (e.g., core/modules/system/config/meta/system.site.yml)? Also, when in my IDE I grep for files named "system.site.yml", both the config file and the meta file match, and I then need to mentally juggle which is which. Would it make sense to name the meta ones differently, like system.site.meta.yml?

+  .label: 'Site name'

I can't tell by reading this patch where the 'label' metadata is used or how it's supposed to be translated. If its only purpose is to support a config translation ui contrib module, can we remove it from the scope of this issue and have a follow up for it? It would make this patch easier to review.

+++ b/core/modules/system/system.module
@@ -2112,6 +2112,27 @@ function system_data_type_info() {
+    'label' => array(
+      'label' => t('Label'),
+      'description' => t('Short object name.'),
+      'class' => '\Drupal\Core\TypedData\Type\String',
+      'primitive type' => Primitive::STRING,
+      'translatable' => TRUE,
+    ),
+    'text' => array(
+       'label' => t('Text'),
+       'description' => t('Textual content.'),
+       'class' => '\Drupal\Core\TypedData\Type\String',
+       'primitive type' => Primitive::STRING,
+       'translatable' => TRUE,
+    ),

Defining that 'label' and 'text' types are translatable makes sense. But what if I want to make some specific config element translatable without changing its type? For example, would it make sense to allow page.front to be translatable by allowing a .translatable: '1' in the meta file? Or is that a bad idea or better punted to a follow up?

I'll try to post a more thorough review soon, but wanted to ask these questions in the meantime.

Jose Reyero’s picture

@effulgentsia,

Can this be moved into the config directory (e.g., core/modules/system/config/meta/system.site.yml)?

I guess yes, metadata may be easier to find when grepping manually so adding to the list for next reroll.

About changing the file names, I find it cleaner if meta name = config name, and anyway it doesn't look to me like a good reason to add a any bit of complexity... Also file names get more complex when we've got multiple nesting levels of objects like for views, see the example in the config_demo module.

I can't tell by reading this patch where the 'label' metadata is used or how it's supposed to be translated. If its only purpose is to support a config translation ui contrib module, can we remove it from the scope of this issue and have a follow up for it? It would make this patch easier to review.

Well, it is used more for demostrating 'metadata capabilities', you can see it in the demo module again. But if this is going to help reviewing the patch I guess yes, they can be removed, and also some metadata files that don't have any translatable properties.

Defining that 'label' and 'text' types are translatable makes sense. But what if I want to make some specific config element translatable without changing its type? For example, would it make sense to allow page.front to be translatable by allowing a .translatable: '1' in the meta file? Or is that a bad idea or better punted to a follow up?

We could find reasons for and against, but these are the main ones:
- Adding the property in the metadata, it is the module author who decides what is translatable and what is not. And I think it should be the site builder who enables whatever other modules to make that stuff translatable or not. Since the metadata itself is suppossed to be hardcoded, the 'translatability' property should be somewhere else so atm it is up to the 'translator module' (Locale) to decide which are translatable properties.
- For the scope of this patch we are only translating strings with Locale module. Other contrib module may decide 'paths' or 'emails' are translatable too, well, that's ok and possible but it needs to use a different system (not locale module that is only for readable strings). Then it is up to the guys translating whether they want to actually translate a specific bit of data or not, provided a module supports translating that data type.

However, if you see @fago's comments in #129 about 'translatable' property, we may need to drop that anyway (unless we can add it to the typed data api) and we'll need to run with 'hardcoded' translatable data types.

Jose Reyero’s picture

Issue summary: View changes

Updated metadata format.

Gábor Hojtsy’s picture

Issue summary: View changes

Largely updated summary

Gábor Hojtsy’s picture

Issue summary: View changes

Fix headers, image

Gábor Hojtsy’s picture

Dramatically updated the issue summary:

- cleaned up text for language, made problem/motivation hopefully much clearer; choice quote: "We cannot say Drupal is a multilingual CMS in 2013 onwards when it cannot even translate the site name you enter!"
- added a lot more basic examples gradually becoming more complex on metadata file format and placement, so it is much easier to understand
- also added actual .yml config files to pair up the metadata with, which made it way easier to grasp for *me*, especially when it comes to nested data structures
- added full class structure image
- added UI steps to verify that the patch works as we went through with @effulgentsia last week

Any more excuses to review the patch and provide feedback?!

Gábor Hojtsy’s picture

Title: Add metadata for configuration in YML files and use it for translating configuration. » Introduce configuration metadata and use for translation

Shorten title for simplicity.

Gábor Hojtsy’s picture

Two questions/problems with the metafile format came up while writing the updated issue summary:

+++ b/core/includes/config.incundefined
@@ -104,6 +106,120 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+ *     .list: '1'
+ *     .list settings:
+ *        elements_name: 'image.style.effects.%'
+ *        elements_key: 'name'

Can we somehow unify the naming style used? "list settings" with space but "elements_name" with underscore?

+++ b/core/modules/config/lib/Drupal/config/Tests/ConfigMetadataTest.phpundefined
diff --git a/core/modules/contact/meta/contact.category.%.yml b/core/modules/contact/meta/contact.category.%.yml
index 0000000..a0095d0

index 0000000..a0095d0
--- /dev/null

--- /dev/null
+++ b/core/modules/contact/meta/contact.category.%.ymlundefined

+++ b/core/modules/contact/meta/contact.category.%.ymlundefined
+++ b/core/modules/contact/meta/contact.category.%.ymlundefined
@@ -0,0 +1,17 @@

@@ -0,0 +1,17 @@
+.settings:
+  label: label

+++ b/core/modules/image/meta/image.style.effects.%.ymlundefined
diff --git a/core/modules/image/meta/image.style.effects.image_scale.yml b/core/modules/image/meta/image.style.effects.image_scale.yml
index 0000000..30b017e

index 0000000..30b017e
--- /dev/null

--- /dev/null
+++ b/core/modules/image/meta/image.style.effects.image_scale.ymlundefined

+++ b/core/modules/image/meta/image.style.effects.image_scale.ymlundefined
+++ b/core/modules/image/meta/image.style.effects.image_scale.ymlundefined
@@ -0,0 +1,11 @@

@@ -0,0 +1,11 @@
+.label: 'Image scale'

What do top level meta markers do? Are these used in nested files and apply to the upper level where they are nested? What does a .settings meta do? Can we get a full list of metakeys?

YesCT’s picture

@Gabor that helps me get re-oriented to this issue. I was feeling overwhelmed when I was thinking about jumping back into this.

Looks like effulgentsia might be part way through the rest of a review. @effulgentsia is that guess accurate? Maybe you could post an update.

The history of this issue has had the patch be refactored a couple times. IMO, moving forward on this and helping people work on this will be easier if at least for a while (few days) there is no refactoring.

I talked with @chx briefly in irc and he might have some time to look at this during the feature freeze time crunch. And @heyrocker might be able to get @catch or @Dries to pop in and reassure us that this is (... or is not) bound by feature freeze. My inclination is that @Gabor thinks it is and @heyrocker added the tag in #128 Re-reading through this issue, it might also be that some of the far reaching stuff might be able to be separated out and follow up (post feature freeze) issues could be made, so that we can focus here on just what needs to be done.

On the other hand, for new enthusiastic folks jumping into this, don't be afraid! I'm sure there are tasks we can come up with (see below for an example). And, if we are getting some feedback, interdiffs on changes that the feedback inspires would be really helpful.

A next simple step (for me maybe, or anyone else that wants to jump in) would be to use the newly updated issue summary section "Test drive the patch!" and verify those steps are accurate.

Gábor Hojtsy’s picture

After a long time today with this patch, I think the most contested complicated part being the nesting pattern/file structuring is hard to avoid so long as the metadata is separate from the data itself, because we need to describe arbitrarily nested data structures with plugins providing the sub-structures. Eg. when describing a views .yml file, you know some values at the top level, then there are display plugins and display plugins have some common fileds, but then they are comprised of more plugins and so on and on. @heyrocker alluded to it before that we might be better off producing one big metadata file for one .yml file, but we need to piece that together according to rules and nesting patterns used from smaller metadata structures, so we need to define the smaller (plugin specific) metadata structures and their nesting system *somehow*. It does not seem like we can forego that. There were no other proposals for this that was demonstrated to solve this problem.

Then the metadata files themselves are structured with dot elements and nested items where dot elements are akin to FAPI # items, so there is a differentiation between nested elements and metadata about elements, it is basically the same as in FAPI, this just uses a dot to signify API specific values vs. data nesting instead of a #.

Gábor Hojtsy’s picture

BTW had a discussion earlier today with @alexpott about this:

[1:29pm] GaborHojtsy: alexpott: hi!
[1:29pm] GaborHojtsy: alexpott: I made http://drupal.org/node/1648930 as well documented as it can get for now  any feedback welcome!
[1:29pm] Druplicon: http://drupal.org/node/1648930 => Add metadata for configuration in YML files and use it for translating configuration. => Drupal core, configuration system, critical, needs review, 141 comments, 56 IRC mentions
[1:30pm] alexpott: GaborHojtsy: I've started reviewing that issue so many times… I'm really sorry that I haven't posted a review yet.
[1:30pm] GaborHojtsy: alexpott: you are not alone 
[1:31pm] GaborHojtsy: alexpott: however, it will not get better without quality feedback (if there are any problems with it that is)
[1:31pm] alexpott: Heyrocker and I discussed how complex it was…
[1:32pm] alexpott: GaborHojtsy: you are right…
[1:33pm] GaborHojtsy: alexpott: I think the key realisation is if you try to construct a parallel proposal that supports image styles and views, then you figure out how all kinds of tricks are needed
[1:33pm] GaborHojtsy: alexpott: essentially you need parts of metadata for the parts of data provided by plugins and then you need to assemble it
[1:34pm] alexpott: GaborHojtsy: yep I've been wondering about that too.
[1:34pm] GaborHojtsy: alexpott: heyrocker said he would rather see one big file, but still then we need the individual descriptions and the merging rules somehow defined to even build such a compound file
[1:35pm] GaborHojtsy: alexpott: so CMI is an arbitrary nested system of key-values with possibly repeated patterns appearing or wholly different patterns showing up and a metadata system needs to account for this
[1:37pm] alexpott: GaborHojtsy: yep somehow we need metadata to be able to be inherited … so we have a general view metadata file and a file per plugin or something like that.
[1:37pm] GaborHojtsy: alexpott: yup, which is what Jose proposes, so you can say if there is a nested level there and provide a key by which to look up the nesting
[1:38pm] GaborHojtsy: alexpott: so eg. views has plugins that have 'type: xyz', so you can say look up the right nested data structure for views + plugin + type xyz for that part of the data
[1:38pm] GaborHojtsy: alexpott: and then that part of data can also have other plugins (eg. a display plugin that has row plugins, etc.)
[1:39pm] GaborHojtsy: alexpott: it is possible to predict based on how a module uses the key-value tree as to where nesting of other defined structures occur
[1:40pm] GaborHojtsy: alexpott: but those can be of varying types, so there needs to be a "condition system", which is what Jose built in the form of the list elements_name and elements_key
[1:41pm] alexpott: GaborHojtsy: I guess we need to add views metadata to the patch to see how this pans out.
[1:42pm] alexpott: GaborHojtsy: has anyone tried to do this?
[1:45pm] GaborHojtsy: alexpott: it is posted in one of the patches above
[1:46pm] GaborHojtsy: alexpott: it is also included with this demo module that Jose was building along: http://drupal.org/sandbox/reyero/1635230
[1:47pm] GaborHojtsy: alexpott: http://drupal.org/node/1648930#comment-6649534 patch hash views
[1:47pm] Druplicon: http://drupal.org/node/1648930 => Introduce configuration metadata and use for translation => Drupal core, configuration system, critical, needs review, 143 comments, 57 IRC mentions
[1:47pm] alexpott: GaborHojtsy: thanks… you deserve a big...
[1:47pm] alexpott: GaborHojtsy++
[1:48pm] alexpott: for all your work on this...
[1:50pm] GaborHojtsy: alexpott:  
[1:51pm] GaborHojtsy: alexpott: well, this one is Jose's battle, I'm just trying to support and find reviewers 
[1:51pm] GaborHojtsy: so
[1:51pm] GaborHojtsy: reyero++
[1:52pm] alexpott: for sure...
[1:52pm] GaborHojtsy: alexpott: anyway, in case you have time, I detailed out the file naming and nesting process in the issue summary, so it should hopefully be as clear as possible  ie. starting out from a simple 1-1 CMI-meta file then a pattern in the filename then a nesting
[1:52pm] alexpott: reyero++
[1:52pm] GaborHojtsy: alexpott: so hopefully you could explore this in mode detail and provide any feedback and/or imagine a parallel system that could somehow be simpler 
[1:53pm] alexpott: GaborHojtsy: I don't think this is a simple problem 
[1:57pm] pcambra is now known as pcambra_afk.
[2:01pm] GaborHojtsy: alexpott: that sounds like an understatement 
alexpott’s picture

Status: Needs review » Needs work

I followed that instructions in "Test drive the patch!" section of the summary and can confirm that it works if you install the standard profile.

I initially used the minimal profile and enabled the image module after I added the language which meant that in order to view the strings in admin/config/regional/translate/translate?langcode=af I had to delete the language and add it again as I could not find an obvious way of rerunning the config import transalation batch. Being monolingual I'm probably missing something obvious...

Some small nits from gazing at the code...

+++ b/core/includes/config.incundefined
+ * @return Drupal\Core\Config\ConfigMetadata

This should be Drupal\Core\Config\Metadata\MetadataLookup

+++ b/core/includes/config.incundefined
+ * @return Drupal\Core\Config\Metadata\Wrapper

This should be Drupal\Core\Config\Metadata\ConfigWrapper

+++ b/core/modules/system/system.moduleundefined
@@ -2112,6 +2112,27 @@ function system_data_type_info() {
+    'config_element' => array(
+      'label' => t('Configuration element'),
+      'description' => t('Configuration data'),
+      'class' => 'Drupal\Core\Config\Metadata\ConfigElement',
+      'list class' => 'Drupal\Core\Config\Metadata\ListElement',
+    ),
+    // These are UI string types.
+    'label' => array(
+      'label' => t('Label'),
+      'description' => t('Short object name.'),
+      'class' => '\Drupal\Core\TypedData\Type\String',
+      'primitive type' => Primitive::STRING,
+      'translatable' => TRUE,
+    ),
+    'text' => array(
+       'label' => t('Text'),
+       'description' => t('Textual content.'),
+       'class' => '\Drupal\Core\TypedData\Type\String',
+       'primitive type' => Primitive::STRING,
+       'translatable' => TRUE,
+    ),

I'm confused about the description fields for "text" and "label" - for example - in system.site.yml the name is a label and slogan is text - aren't both textual content?

Also shouldn't the classed defined in config_element's class and list class be prefixed with a \ like all other entries here... or perhaps should none of them have a leading \?

Furthermore: I tried to translate the site name and slogan but I could not do this. Looking at the code it seems to be because I've changed them from the defaults! But I might well be going mad by now. However, I confirmed this behaviour by changing the defaults in system.site.yml and re-installing the site, enabling locale and there they were in the translate interface screen - I am missing something here?

However once I do this I can confirm that my site name is translated when I visit the site in different languages! :)

effulgentsia’s picture

Status: Needs work » Needs review

There's still a lot in this patch that I don't fully grok, partially because I don't have that much experience with all of Drupal's i18n subtleties. But at a high level:

  1. I like the general approach of the ElementBase, ConfigElement, and ListElement classes. It makes a lot of sense to me to use TypedData API in this way, and allow for traversing a Config tree via these typed data objects. An alternative that I think might have been discussed is implementing config elements as fields (in the Entity API sense), but I don't think that's viable, because Entity API fields are by design more constrained in their structure (fields contain field items contain simple(ish) elements) and translation only happens at the field level, whereas config elements can be nested to arbitrary depth and translated at any depth, so unless someone comes up with a very clever way to unify these different needs, I like the separation of Drupal\Core\Entity\Field\* classes from this patch's Drupal\Core\Config\* classes, but with both extending Drupal\Core\TypedData\* for what they share in common.
  2. There's a couple things I find strange about the ConfigWrapper class. First of all, the name: I'm generally suspicious of any class with "wrapper" in its name, as it does not clarify the nature or reason for the wrapping. As far as I can tell, it's a class that's similar to Config except that get() returns TypedData objects instead of bare string values. So would TypedConfig be an ok name? Secondly, it extends Config, but not fully: it neither calls the parent class's constructor nor overrides save(), so I'm guessing that calling $config_wrapper->save() would make something blow up. I don't know whether it would be more logical to not extend from Config at all, or to override save() and other problematic methods, or to integrate better with ConfigFactory and chain the constructor.
  3. I'm concerned about the need for LocaleConfigWrapper to be a separate class from ConfigWrapper. It's different than what we have for entities, where the base Entity class includes awareness of translations (i.e., it has a getTranslation() method). There's probably some good reasons for this difference, but it jumped out at me as a possibly unnecessary inconsistency.
  4. It seems like heyrocker's main concern with this patch is the proliferation of meta files, one per plugin, and the corresponding impact on DX. I don't think it makes sense to not split it out by plugin, though, because each plugin can define completely different config elements that it needs. But in #59, yched references #1764380: Merge PluginSettingsInterface into ConfigurableInterface, and maybe as part of that issue, we can provide a way for plugins to expose their config metadata that doesn't rely on meta files? Then, we would only need meta files for the config objects themselves (i.e., at the image style level, not the image effect level). They key benefit I would see to that is that it would keep plugins as single things (i.e., a class, not a class plus a meta file), which was a key design goal for the plugin system.

I don't consider any of the above as commit blockers though. Even if this is something that core committers approve of as a post-feature-freeze thing, I would personally prefer to see it go in before feature freeze, and be improved upon after.

effulgentsia’s picture

Status: Needs review » Needs work

Unintentional status change. Back to #148's status.

effulgentsia’s picture

Furthermore: I tried to translate the site name and slogan but I could not do this. Looking at the code it seems to be because I've changed them from the defaults!

That's by design. Default config values shipped with modules is considered "part of the software" and should therefore be translated with t(). Custom config values are not "part of the software": to translate them, you need to use a Config Translation UI, which will be a contrib module, a work in progress demo of which is in Jose's sandbox.

effulgentsia’s picture

I can't tell by reading this patch where the 'label' metadata is used or how it's supposed to be translated. If its only purpose is to support a config translation ui contrib module, can we remove it from the scope of this issue?

After discussing this with Gabor, I take this back. It makes sense that we need labels for a config translation ui. And since the values in meta files are not themselves user-configurable, they will get translated with t(). And Gabor explained that it's ok to add specific rules to the potx module (like translate .label values, but not .type values) and keep that up to date with popular contrib modules that add more meta keys (like .description). It would be silly to need metadata for our metadata, so reducing the problem to just needing to keep the potx module up to date seems like a reasonable approach. That addresses all of my concerns about the .label meta key.

yched’s picture

Regarding plugins :
A given plugin only knows about itself - i.e its own "configuration entries" - which is what I'd like to settle on the term "settings" with #1764380: Merge PluginSettingsInterface into ConfigurableInterface.
But it has no knowledge on how an enclosing system will use it, and which shape this system's CMI files are going to be structured - or, for that matter, it has no knowledge that it can end up in a CMI file to begin with.

For instance, a field Formatter :
The 'text_summary_or_trimmed' formatter knows it has one 'trim_length' setting, and can easily provide metadata about it (it's an int) - and #1764380: Merge PluginSettingsInterface into ConfigurableInterface would most probably help making that easier.
But the configuration for a Formatter can be enclosed in both :
- the CMI file for a field instance ("article nodes in teaser mode show this field with this formatter and these settings")
- the CMI file for a view ("this display shows this field with this formatter and these settings")
- the CMI file for some unknown contrib ConfigEntity

So, only the enclosing system knows its CMI files happen to embed config entries for a given type of plugin, and only this enclosing system can provide metadata about the structure of its CMI files, including pieces of metadata collected from the various plugins themselves.

So I'd agree with @effulgentsia that plugins should not have to provide files of metadata snippets. They provide metadata about their settings, preferably at the same place where they already expose info about their settings (names, default values) for the systems that want to instantiate them - with the current patch in #1764380: Merge PluginSettingsInterface into ConfigurableInterface, that's in their "plugin definition" (annotations or info hook), but depending on discussion over there, that could be split to a dedicated static method.

Then, only ConfigEntities have to provide yml snippets of metadata for their own CMI files, and that seems consistent.

alexpott’s picture

Status: Needs work » Needs review
FileSize
79.53 KB
12.74 KB
5.16 KB

Fixed the module enable / disable issue. Basically once a language was enabled config default strings for modules enabled after that would never make it into the translating interface due to an improperly keyed array produced by _locale_config_component_names(). I've added to the test so that this situation is covered. I've left a @todo in the test as testing a translation after module install seemed difficult. Also we need to test config entity translation.

Patch also fixed the incorrect classes from #148

Why two interdiffs?
Interdiff utility was behaving weird - that's the "evasive-interdiff.txt"
"interdiff-meta-1.txt" is an interdiff created from my git history and it's easier to review.

effulgentsia’s picture

Status: Needs review » Needs work

Then, only ConfigEntities have to provide yml snippets of metadata for their own CMI files, and that seems consistent.

Actually, ConfigEntities are themselves plugins, so can provide their config metadata via their class (annotations or static method), same as for any other plugin. So the only meta files that would be needed are for singular (non-entity) config objects, and those do not need a wildcard.

effulgentsia’s picture

What do you all think about this as a way to proceed? Remove all '%' handling from this patch, so that this issue only needs to concern itself with singular config objects, like system.maintenance.yml and system.site.yml. Then we can add plugin (including ConfigEntity) support in a follow up, either after resolving #1764380: Merge PluginSettingsInterface into ConfigurableInterface or in parallel with it.

yched’s picture

Actually, ConfigEntities are themselves plugins, so can provide their config metadata via their class (annotations or static method), same as for any other plugin

True. Then again, we're trying to limit the amount of data contained in entity_info (ideally, strictly what's needed to list types and instanciate objects). Also, any potentially dynamic (= often read from config) part in entity_info is problematic, because of recursion (see #1822458-6: Move dynamic parts (view modes, bundles) out of entity_info()).
So the metadata would more probably be in a method rather than in the entity_info entry (same argument probably applies to plugin setttings...)

Other than that, #156 sounds fine, but I'm not the one that's supposed to agree :-).

fago’s picture

+++ b/core/lib/Drupal/Core/Config/Metadata/ElementBase.php
@@ -0,0 +1,328 @@
+ * Supported settings (below the definition's 'settings' key) are:
+ * - 'label', Name of the configuration data property to be used as
+ *   the label for the element definition.

Looks like that's not what getLabel() does.

+++ b/core/lib/Drupal/Core/Config/Metadata/ElementBase.php
@@ -0,0 +1,328 @@
+      // The element label may be contained in the data.
+      if (isset($element['settings']['label']) && isset($data[$element['settings']['label']])) {
+        $element['label'] = $data[$element['settings']['label']];
+      }

Oh, here it is. But then, it's not a setting that applies to the typed-data class. It's actually pre-processed.

However, it's a bit strange having the label being read from the data. The label should describe the property in general, e.g. "An image effect.". It's not supposed to be the data's label, it generally describes the data in a human readable way.

+++ b/core/lib/Drupal/Core/Config/Metadata/ElementInterface.php
@@ -0,0 +1,66 @@
+   * Gets raw metadata array for this element.
+   *
+   * @return array
+   *   The full metadata array for this element.
+   *
+   * @see config_metadata()
+   */
+  public function getMetadata();

I think the naming could be improved here. What's the difference between metadata and definition? Both are metadata.

So maybe, this could be could ConfigurationMetadata or so.

+++ b/core/lib/Drupal/Core/Config/Metadata/ElementInterface.php
@@ -0,0 +1,66 @@
+  /**
+   * Gets all contained typed data properties as plain array.
+   *
+   * @return array
+   *   List of TypedDataInterface objects indexed by full name (keys with dot
+   *   notation).
+   */
+  public function toList();

The plain array/value is what should get by getValue(). So imo that could be removed.

However, this points me to an issue of the typed data API. Lists miss a way to get/set by an array of values vs. the internal representation. Right now it assumes that the internal representation of lists is an array, but it shouldn't. It should be able to use it with any kind of data structure while implementing the necessary interface.

That said, we probably should introduce something like getItemValues(), setItemValues, analog to getPropertyValues and setPropertyValues for complex data structures.

However, this should not hold up this patch - we can take care of that in a follow-up.

+++ b/core/modules/image/meta/image.style.effects.image_scale.yml
@@ -0,0 +1,11 @@
+.label: 'Image scale'
+data:
+  width:
+    .label: 'Width'
+    .type: 'integer'
+  height:
+    .label: 'Height'
+    .type: 'integer'

That maps well to typed data definitions now what makes it much more consistent. Great!

+++ b/core/modules/system/system.module
@@ -2113,6 +2113,27 @@ function system_data_type_info() {
+    'label' => array(
+      'label' => t('Label'),
+      'description' => t('Short object name.'),

Not sure why this needs a separate type, but it's fine to introduce new ones. Maybe it should be config_label or config_element_label? (I don't know how it's used).

We generally we'll needd a way to get a label of complex data. So I coud see us adding an interface having the label() method, so optionally you can support having a human readable label. But then, for implementing this on the entity-level a type => 'label' won't fly, so I don't think that's an approach we want to take in general. But it could be the approach used by the config system. This means, it should have a config_ prefix then.

+++ b/core/modules/system/system.module
@@ -2113,6 +2113,27 @@ function system_data_type_info() {
+    'text' => array(
+       'label' => t('Text'),
+       'description' => t('Textual content.'),
+       'class' => '\Drupal\Core\TypedData\Type\String',
+       'primitive type' => Primitive::STRING,
+       'translatable' => TRUE,

I'm still concerned by this data type. In an overally system perspective they don't make sense. E.g. we have 'string', 'text', 'text_field' now. What the heck should I use?

@translatable types:

However, if you see @fago's comments in #129 about 'translatable' property, we may need to drop that anyway (unless we can add it to the typed data api) and we'll need to run with 'hardcoded' translatable data types.

Well, if you introduce your own type you can just decide to treat everything of that type as translatable. I guess it would make sense to reflect this in the name somehow then, or just have 'config_text' and make it translatable by default, i.e. as long as there is no .translatable => false in the metadata?

As said, it's fine for the config system to introduce new typed data definition keys on top of the existing ones - as long as you use it to implement the TranslatableInterface the rest of the system can know (even without supporting config-definition keys).
Note, that I'm talking of the actual data definitions here, you cannot invent new keys for data type definitions as there is a central and so shared place for those (e.g. an enttiy api and config system 'translatable' key would clash).

Jose Reyero’s picture

Wow, looks like we finally got some good reviews :-) These are replies and thoughts about open questions here (I'm afraid they will take me more than one post):

@Gábor Hojtsy #144

What do top level meta markers do? Are these used in nested files and apply to the upper level where they are nested?

Top level ones are just some help to be able to create a data browser and have elements properly labelled. These may be good help for translators in some future UI.
This is not strictly required it just helps some more consistent handling of '.label' for all elements. Note any piece of the metadata can be a nested element itself. See ConfigElement::getLabel() method.

What does a .settings meta do? Can we get a full list of metakeys?

This one, like 'list settings' and all the metadata keys are typed data properties as defined by the typed data api so I don't think they need further definition, though maybe we should better explain metadata properties are typed data properties.
Both 'settings' and 'list settings' are type specific arbitrary properties supported by the typed data api.
(The exception is the 'translatable' property for elements that may be considered as an extension to the typed data api).

About the nested structure for the metadata:
- I think the need for it is clear since we need to support arbitrary nested config data for plugins and a config object may have a list of different pluggable elements (I.e. image styles may have different effects provided by different plug-ins each of them needing different data)
- Something not mentioned here yet is the possibility for modules to define their extensions to ConfigElement if needed to handle some specially difficult nested data structure. However all the configuration examples found in core so far fit in the current elements.
Even for views, they are cleanly supported, the metadata for them was included in some version of the patch then dropped following suggestions for making the patch smaller (moved to the config demo module). Maybe we should add them back to the patch (?)

fago’s picture

- I think the need for it is clear since we need to support arbitrary nested config data for plugins and a config object may have a list of different pluggable elements (I.e. image styles may have different effects provided by different plug-ins each of them needing different data)

I agree. With configs having a different translation model than the entity api, there is now way to avoid a direct implementation of the typed data API (instead of possibly relying on config entities only).

- Something not mentioned here yet is the possibility for modules to define their extensions to ConfigElement if needed to handle some specially difficult nested data structure. However all the configuration examples found in core so far fit in the current elements.

Yeah, so I could just define '.class' in the metadata and have my own, as usual in the typed data api -right?

Both 'settings' and 'list settings' are type specific arbitrary properties supported by the typed data api.
(The exception is the 'translatable' property for elements that may be considered as an extension to the typed data api).

Yep. But as said, extension on type definition level are problematic. So either, we've to introduce it as general typed-data key or it we keep it specific to config and call it 'config_translatable' or so. Or we make it implicit as suggested in #158.

I'm not sure about making it a general key as it duplicates the information we have by the class implementing the translatableinterface - what might depend on the actual implementation used. Also, the interface is not implemented by the config string class so it doesn't really map well. (although it might be a good idea to implement it?)

Gábor Hojtsy’s picture

@effulgentsia: as for why ConfigWrapper and LocaleConfigWrapper are separate, the design of the entity API is that language is always present, all the field data is keyed by langcode, regardless of whether the site uses language/locale module, so accessors and APIs naturally need to support language right away. For configuration, the design goal was to keep config simple, so it does not in itself have language support. A generic overrides and event listener system was added, so that locale can register its event listener that provides overrides. The same separation is implemented with the wrappers here, so that locale defines the language based wrapper. If locale is not enabled, there is no language based access, so this is very different from entities. Although this patch uses this wrapper directly, the config factory patch at #1763640: Introduce config context to make original config and different overrides accessible introduces a layer that abstracts the use of the wrapper, so people will not use it directly.

Jose Reyero’s picture

@effulgentsia #149

1. ... An alternative that I think might have been discussed is implementing config elements as fields (in the Entity API sense), but I don't think that's viable, because Entity API fields are by design more constrained in their structure
...

I don't think there's too many of fields that can be reused here, to I'll be looking into it.

2. There's a couple things I find strange about the ConfigWrapper class. First of all, the name: I'm generally suspicious of any class with "wrapper" in its name, as it does not clarify the nature or reason for the wrapping. As far as I can tell, it's a class that's similar to Config except that get() returns TypedData objects instead of bare string values. So would TypedConfig be an ok name? Secondly, it extends Config, but not fully: it neither calls the parent class's constructor nor overrides save(), so I'm guessing that calling $config_wrapper->save() would make something blow up. I don't know whether it would be more logical to not extend from Config at all, or to override save() and other problematic methods, or to integrate better with ConfigFactory and chain the constructor.

The 'TypedConfig' object sounds like a good idea to me.
About extending Config, right, thay may not be the cleaner way, though both objects have a lot in common. I guess a better option would be to separate the data structure in config (ConfigData? with getters/setters supporting dot notation) from the data handling/storage/etc (Config extends ConfigData), and then having TypedConfig extending ConfigData too.
However since this patch is already big enough, too big I'd say, maybe we could postpone that refactoring for a follow up. I like also the fact that this patch on its current form, is mostly an add-on to the current config system (Self contained in the Config/Metadata folder) which should be way easier to review without concerns of it directly impacting the config system itself.

3. I'm concerned about the need for LocaleConfigWrapper to be a separate class from ConfigWrapper. It's different than what we have for entities, where the base Entity class includes awareness of translations (i.e., it has a getTranslation() method). There's probably some good reasons for this difference, but it jumped out at me as a possibly unnecessary inconsistency.

Atm the 'translation' functionality is a feature provided by Locale module so it makes sense to me having it contained there for now. Otherwise we may end up with weird dependencies like the config metadata system requiring locale and not being suitable for any other use. In other words, metadata is a feature of the config system, config translation is a feature of locale module.
However, there's a patch ready to fix this and add a 'Translation' interface in the base system, that then can be extended by locale (to provide locale way of translation). It is here and if we get that in too we can think of moving back the 'config translation' feature into the base system, see #1813762: Introduce unified interfaces, use dependency injection for interface translation

4. It seems like heyrocker's main concern with this patch is the proliferation of meta files, one per plugin, and the corresponding impact on DX. I don't think it makes sense to not split it out by plugin, though, because each plugin can define completely different config elements that it needs. But in #59, yched references #1764380: Base class for management of Plugin 'settings', and maybe as part of that issue, we can provide a way for plugins to expose their config metadata that doesn't rely on meta files? Then, we would only need meta files for the config objects themselves (i.e., at the image style level, not the image effect level). They key benefit I would see to that is that it would keep plugins as single things (i.e., a class, not a class plus a meta file), which was a key design goal for the plugin system.

As I see it plug-ins add their own data to the big configuration object and for the system to be consistent they do need metadata files. But this would be only in the case they actually need to add new data. An example for the image case:
- Any plugin not needing additional data may just use image/meta/image.style.effects.%.yml
- Only plugins really providing extensions to the config data, like 'image_scale' would need a specific metadata.
About providing any other mechanism for plugins to add their metadata, I think providing a metadata file is just the simplest way so I don't really understand this concern. Though we can (and maybe we need to) add some hook_metadata_alter(), really requiring plugins to implement it will make it all more complex that just dropping a metadata yml file in the folder.
As a reminder, metadata is not a requirement, since we provide a fallback to typed data 'string' in every case. Atm only if a module/plugin needs to expose the data to the translation system it will need metadata.

Jose Reyero’s picture

@effulgentsia #156

What do you all think about this as a way to proceed? Remove all '%' handling from this patch...

I don't think this is a good idea since there are multiple config objects that share the same structure like different image styles so either we provide a different field for each image.style.* or we use current image.style.% that fits them all

@fago #160

Yeah, so I could just define '.class' in the metadata and have my own, as usual in the typed data api -right?

Well, the idea is you use 'type', and have your typed defined in 'hook_data_type_info()' (class ?)

Then, multiple comments about 'label', 'text' and 'translatable' property:

Some different classes of translatable text like these 2 are handy for knowing which kind of string we are dealing with for:
- Providing a widget (it may be textfield for label and 'textarea' for text)
- Doing proper validation (I.e. maybe we don't need to allow html for 'label' translations but we do for 'text' translations).
* Current 'locale_string_is_safe' is way too generic for validating translation and does only worry about 'safety' but not about proper formatting or validation. It is way too easy atm to break the whole site's html structure by injecting and oddly formatted translation.
However these two need better definition like:
- 'label'.. Single line (=textfield), plaintext (validation)
- 'text'... Multiline (=textarea, line delimiter=\n), html (validation)....

Extensions to these classes would be handy too like 'mail_body' which would allow different validation for translations used for mails (depending on mail system, may not allow html)

* The big issue we have atm is original config (settings page) goes through proper validation but translations never do.

Then, provided we follow this approach, we need a way for modules to add their own 'translatable text types' and this is suppossed to be the 'translatable' property because just hardcoding 'label' and 'text' into locale won't work for this. However, the idea of this flag is "this is a human readable text" (== it needs translation by locale system) so the name could be any other, maybe not "translatable"...

(Still scanning the thread for more unanswered questions...)

chx’s picture

> I don't think this is a good idea since there are multiple config objects that share the same structure like different image styles so either we provide a different field for each image.style.* or we use current image.style.% that fits them all

But image styles are ConfigEntity and that's exactly what effulgentsia says: ConfigEntities should provider their own metadata via annotations and simple config objects need this patch. Sounds the first sane idea I have seen in this issue for some time now, to be honest. The whole patch is frightening, simplifying it is a very, very good idea.

Jose Reyero’s picture

@effulgentsia, @chx,

> I don't think this is a good idea since there are multiple config objects that share the same structure like different image styles so either we provide a different field for each image.style.* or we use current image.style.% that fits them all

But image styles are ConfigEntity and that's exactly what effulgentsia says: ConfigEntities should provider their own metadata via annotations and simple config objects need this patch.

Yup, I had missed that point, sorry.

But I still fail to see how sticking metadata into annotations will make it more readable than providing a metadata.yml file.
Also an external parser (po extractor) to get strings out of configuration will need to access that metadata. Will it need all the modules enabled and running for that?.

Really, yes, we can replace all metadata.yml files by some hook_metadata info (or some of these trendy 'anotations' that are needed just because we don't have a proper metadata system in config that can tells us which ConfigEntity maps to which config name.).

However, the whole purpose of having metadata in yml files as oppossed to some hook_info() was being able to parse config data and metadata without needing the full Drupal and all the modules providing the config running.

fago’s picture

Without knowing the CMI details, I think that "image.style.*" shuold be handled like an entity reference, i.e. have separate metadata for all the effects and refer to them via reference. Something like that can be done with the typed data for non-entities as well (by following what is done for entities).

>Well, the idea is you use 'type', and have your typed defined in 'hook_data_type_info()' (class ?)
Yes, but you can specify 'class' in the data defintiion if you want to override it.

>* The big issue we have atm is original config (settings page) goes through proper validation but translations never do.
Work on validation is done over at #1696648: [META] Untie content entity validation from form validation it will be re-usable from the typed data API layer. Thus, validation constraints being part of the definitions will be applied.

So, I could see the case for text having different validation than 'string', but html or not plays more into display. Widget is its own story, I'd prefer to keep out for now and solve it in another issue. We cannot solve everythin at once.
Anyway, I fear string vs text has potential for confusing. We need to watch out for doing good names here, so I'm hesitant to introducing some-quick addons to fulfill a certain use-case. Instead, we should do so carefully, list all needed variations and come-up with a list of names that play well together. 'string' imo should stay as the basic type without any particular validation or something built-in.

In short, to not hold up this patch I'd suggest to go with
.type => string and
.type => config_label
and default to 'translatable' => TRUE when processing config metadata.

Jose Reyero’s picture

Yes, but you can specify 'class' in the data defintiion if you want to override it.

Oh, right, I was missing that.

The point is by using the typed data api, and since the config (dot)properties map right into it, this is definitely extendable in many ways.

About ConfigEntities, really (and sadly) they are useless for our purposes unless we could map every config object into an entity. So by using entity mechanisms we will be duplicating stuff as we still need a non-entity alternative.

Also I think using 'string' for translatable stuff is not an option, since every property in config defaults to 'string' type.
Really, as explained in #141, adding a per item 'translatable' property is a bad choice too.

Trying to reword the problem: Unlike for translatable entities for which you want to know exactly which properties are to be translatable (so translatable is a per-property flag) to provide the right content editing UI, from the 'localization' perspective what you want to know is 'which data types' the system has the ability to translate, and then the user (site builder) can translate them or not. It's quite a different approach IMHO what we need for localization.

So could we settle on having a single data type, that is a basic string data type and contains 'human readable' (thus translatable by locale) text? Say 'text', 'human readable string' or whatever?

As I see it the problem (aka why there's so strong opposition) to creating generic data types is needing to create on top of it a new data type definition for the field system, am I right?
(Which I think is some serious flaw in the typed data API, since we seem to be condemned to duplicate the definition of any new data type, one for generic typed data, one for fields)

effulgentsia’s picture

But I still fail to see how sticking metadata into annotations will make it more readable than providing a metadata.yml file.

Per #153, the issue is that plugins can be used in multiple contexts, so requiring a meta file for each parent configuration it appears in creates problems. Also, a design goal of the plugin system has been to allow a plugin to be a single thing. So rather than an info hook plus a bunch of disparate callback functions, we have a class with annotations. That way, the behavior and metadata of a plugin are contained within a single PHP file. We have not yet converted image effects and many other plugin-like things to actual plugins yet, but we have issues for that. So, expressing the configuration metadata within the plugin class (whether as annotations or as a static method) rather than in a .%.yml file allows:
- Plugin configuration to be embedded in multiple different kinds of larger configuration objects without the plugin author needing to know what all of them are.
- Plugins to remain a single class, rather than a class (in a deep PSR-0 directory) plus a meta file (in some totally different directory).

For ConfigEntities, the first point doesn't apply, since by definition, they define their own config file rather than being embedded in a larger one, but the second point still does.

Also an external parser (po extractor) to get strings out of configuration will need to access that metadata. Will it need all the modules enabled and running for that?

Hm. Good question. If it knows which plugin class is responsible for the config file or section of config file, then it can get the metadata it needs from that class without the module being enabled or Drupal even running. I don't think that condition is currently true, but is possibly something we might want to make true in a separate issue.

effulgentsia’s picture

So by using entity mechanisms we will be duplicating stuff as we still need a non-entity alternative.

I think much of what's in this patch will stay. The only thing we'll be duplicating is where we get the metadata from: a plugin class annotation/method when we have a plugin/ConfigEntity, and a non-wildcarded meta file for non-entity config objects.

Gábor Hojtsy’s picture

So is it safe to assume that for *any* configuration in Drupal 8 that has repetition in sub-structures and/or variable sub-structures, all of those sub-structures would come from plugins, period?! For example, the current patch provides metadata for the user.mail.yml file that has a list of subject/body pairs as a pattern with that sub-structure and a master meta file that says the file contains a list of sub-structures with that pattern. Would these be plugins or would the metafile need to repeat the metadata for every possible key in the item list? What about any module that has structured data in lists in config files? The list items would *always* *no exceptions* need to be plugin instances or the list would need to be predefined in terms of items, order of items and name of items? I think its significant to realize that this proposal assumes that anything with a substructure is a plugin and nothing else would be possible to be able to provide metadata.

effulgentsia’s picture

If the substructure of each list item can vary (e.g., image effect X requires a different set of configuration than image effect Y), then yes, this needs to be a plugin. The risk, I suppose, is what if we don't convert all such things to the plugin system in time? Frankly though, just as we're committed to converting all configuration to CMI, I think we equally need to be committed to converting all plugin-like things that require configuration to actual plugins.

If the substructure of each list item does not vary (e.g., for each top-level item in user.mail.yml, we have 'body' and 'subject'), then those do not need to be plugins, and we need the meta/user.mail.yml to be able to express that. It should be possible to express that within meta/user.mail.yml itself though, without branching out into separate files.

effulgentsia’s picture

@Jose: I also want to say that despite my long silence followed by opinionated feedback offered late, I actually really like what you've done here, so thank you! One way or another, we need to get this in, and I think (hope) we're close to that.

sun’s picture

I am extremely sorry for not having had the time to review this and chiming in earlier. I still wasn't able to spare time for that. However, I tried hard to track and follow the discussion silently via mail.

For now, I only can and want to note that major parts of the topic and discussion rightfully focused on the TypedData aspect of this implementation, which adds a layer of defining dynamic/partial schemata to the configuration system, and, inherently adds substantial processing overhead. One of the primary reasons for staying away from Symfony's Config component was, in fact, to avoid that "seemingly needless" overhead of explaining configuration data as typed and schema-alike "trees" of objects.

Fact is, this entire approach and discussion seems to duplicate the (obscurely hidden) TreeBuilder functionality in Symfony's Config component, which essentially performs the exact same task: Distilling and translating raw [configuration] [array] data into precisely defined data objects all the way down to individual values. It's beyond my sense, understanding, and horizon why Symfony decided to hide that kind of functionality in a deeply obscured subdirectory, given that they have a similarly generic Validator component already [which must duplicate some good chunks of the TreeBuilder on its own already]. The fact that we invented our own TypedData component clearly indicates that there's a common generic need for these things.

In short: Given the direction this issue/patch attempts to move Drupal's configuration system, I wonder whether it still makes sense to stay away from Symfony's Config component, and whether we aren't actually blatantly re-inventing the wheel here, and lastly, whether we shouldn't rather investigate to revamp and turn our Config into a hybrid implementation between theirs and ours. — To clarify this crystal clearly, I am not saying that Sf's Config component magically solves all problems we're facing (because, by design, Symfony does not face a range of our problems), but in hindsight, all of that utterly ugly amount of abstraction that's visible in there slowly but certainly starts to make sense in light of this change proposal here.

fago’s picture

Trying to reword the problem: Unlike for translatable entities for which you want to know exactly which properties are to be translatable (so translatable is a per-property flag) to provide the right content editing UI, from the 'localization' perspective what you want to know is 'which data types' the system has the ability to translate, and then the user (site builder) can translate them or not. It's quite a different approach IMHO what we need for localization.

Sry, but I don't see the difference. To me, translatability is an inherent property of the data property, not of its type. Either it's a human readable string it makes sense to translate, or not. (Leaving the ML context override use-case beside). If we've multiple types for strings in future, it doesn't make much sense to re-define each with a translatable option. Then, I don't see the problem with defining translatability in the definitions. By specifying the "translatable" data type you'd do it already anyway?

As I see it, this API would enable translating for data items, but this does not mean you have to. If you are not using a localization module you won't. If you don't translate some strings, you won't. What problem am I missing?

Also I think using 'string' for translatable stuff is not an option, since every property in config defaults to 'string' type.
Really, as explained in #141, adding a per item 'translatable' property is a bad choice too.

I'd suggest to opt-out per string. Regarding the default I'd prefer making that explicit anyway, I've gone the by-default 'text' route with the entity property api in d7 and imo it just lead to unnecessary confusion. Still, *if* you define your metadata wrong and something is translatable what shouldn't, nothing breaks and it's easy to fix.

Jose Reyero’s picture

Mixed comments, clarifications, thoughts. My main concern at this point though is that we seem to be losing focus on what we've got and back into the "we should use..." path.

About plugins configuration:
- We don't need to duplicate plugin.yml files for using a plug-in in multiple places, a single metadata file is reusable for all the configuration objects.
- Thinking about moving some metadata into that annotations, I'd like to have an idea first of )how would it look like b) how are we doing the name resolution (jumping from the config data to the right class)

So I'm looking atm into entity/classes/annotations/name-class-resolution. However to be honest I think annotations are an awful idea (that we may need to use if everyone else thinks otherwise though) and there are some very good questions by @Gábor in #170

@effulgentsia,
I like oppinionated comments as long as they come, like yours, with constructive ideas :-)

@sun,
Though you certainly have a point about what config/typed data should be I think we are really late for that and we should try to get the best of what we've got atm.... given we are too close to feature freeze.

So this is my todo list atm, trying to get this back on track for feature freeze.
- Some minor clean up and fixing small issues mentioned above.
- Some more simplification (remove labels? / drop 'label' type and mark all as 'text' / drop metadata for config that has no translation, that can be added later / )
- Exploring the metadata in annotations for config entities.

And also I would ask everybody to consider whether it could be better to just try to get the feature in and then work on improvements (like plugins/annotations/etc)

Jose Reyero’s picture

Sry, but I don't see the difference. To me, translatability is an inherent property of the data property, not of its type. Either it's a human readable string it makes sense to translate, or not.

.

Ok but what we are trying to do is to extend data types so they provide more clues on whether they are translatable or not (and we don't need a metadata file filled with 'translatable' => true/false properties).

So, what's wrong with making 'human readable string' a data type? Which is actuall a different data type to, say, a machine name that is restricted to a subset of utf8-chars. Or if you like it beter, 'utf8string' can be a data type too.

I thought the whole idea of getting a typed data api was then being able to easily extend it and we could extend it also the other way, creating specific types for machine_name, path, url, etc... Then we'd be left with a few 'strings' and then ok, we can add a few 'translatable' => FALSE.

For fields, translatability has nothing to do with its data type, as you can make any kind of field translatable or not, right?
For a localization system whe need to know which kind of things the system (aka locale module) is able to translate.

Gábor Hojtsy’s picture

@Jose Reyero: I don't agree with removing the item labels from the metadata, because that and the type are required for us to build a minimal translation system in contrib, and if core does not provide these, then contrib will not do that either, and it would all be down to i18n module again to do for *every single module* from the outside. The prospect of that sounds much worse than people needing to add labels into metadata files for their modules. @effulgentsia revoked his position on removing that earlier too.

@effulgentsia: on implementing substructure metadata in the same metadata file, I guess you should propose a reference/naming structure for that. I've actively used XML Schemas a few years back for example, which define these as new "types" which you can reference from upper/lower structures. You need syntax for defining new types, naming them and referring to them from places. Jose's syntax uses separate files and file name patterns. This can technically be represented in the same file, in which case other config using the same structure will just need to repeat the same metadata. If we don't have this capability, the metadata system cannot describe the user emails config or the recently committed display's configs (which have a 2 level nested array block information array with them):

       array(
         'block1-configkey' => array(
           'region' => 'content',
           // store the region type name here so that we can do type conversion w/out
           // needing to have access to the original layout plugin
           'region-type' => 'content',
           // increment by 100 so there is ALWAYS plenty of space for manual insertion
           'weight' => -100,
         ),
         'block2-configkey' => array(
           'region' => 'sidebar_first',
           'region-type' => 'aside',
           'weight' => -100,
         ),
         'block2-configkey' => array(
           'region' => 'sidebar_first',
           'region-type' => 'aside',
           'weight' => 0,
         ),
         'maincontent' => array(
           'region' => 'content',
           'region-type' => 'content',
           'weight' => -200,
         ),
       );

The metadata type system needs to accomodate for these structures in your system if the list of items and the individual items are not plugins themselves. Issue at #1817044: Implement Display, a type of config for use by layouts, et. all.

@sun: thanks for coming with some great feedback. We had an anxious discussion about this metadata system back at Drupal Developer Days Barcelona (that was 5 months ago) and we thought we were already late with that. Now here we are 15 days before feature freeze discussing this critically missing feature and finally got some renewed attention on this issue. Great stuff!

So the question is who and how would rearchitect the metadata system and especially the whole config system overall in 15 days to accommodate for the missing feature of metadata and translatability of config pieces. Is that a reasonable ask at this point? Given that refactoring and reworking things is entirely possible after December 1st (for 5 months), do we want to put features on hold (and most probably not to be done) and go do some refactoring fun instead? Looks like the fear is that those who propose the refactoring do not necessarily want to be involved in doing so and afraid it would not happen otherwise later, so instead would want to block this getting in without the refactoring. On that, given that we don't even have any kind of agreement, it seems hard to promise anything, honestly.

I'm sad that despite this being among the top 3 priority issues of CMI for half a year and us continually pinging people about it for months, none of this earth shattering feedback came on time to be implemented in a reasonable fashion.

Gábor Hojtsy’s picture

Issue summary: View changes

Fixed 2 code examples

chx’s picture

FileSize
81.78 KB

OK so finally I get it. And, I have written a monster of a comment into config_metadata to make sure others get it too. I also don't think the ConfigEntity as described above my effulgentsia and me is a good idea. However, the merging of specific and generic files must go, I discussed this on IRC with Jose. It's bad from both angles: when editing a YML file, you have no idea whether the specific key you are editing will end up in the system or not. And dumping in PHP, you have no idea where stuff is coming from. Also, you need to come up with recursive merging rules. Those are never pretty. By removing the merging, we clarify this. And, it's the same templates work: you copy node.tpl.php to node--article.tpl.php and customize.

Jose Reyero’s picture

FileSize
7.06 KB

Attaching interdiff for latest @chx's patch, which only adds comments, and looks good to me.

So after talking about it, removing the 'merging' for making this a bit simpler, at the expense of duplicating a few of the metadata doesn't look like a bad idea to me, planning to include it in next re-roll.

chx’s picture

To clarify: the list merges seem to be necessary. I am not sure I like them but they are.

On the other hand, merging image.style.effects.%.yml with image.style.effects.image_scale.yml must go, plain and simple. As I said: apply the "copy node.tpl.php to node--article.tpl.php and customize" pattern. Do not merge at this level.

YesCT’s picture

in #175 @Jose Reyero

So this is my todo list atm, trying to get this back on track for feature freeze.
- Some minor clean up and fixing small issues mentioned above.
- Some more simplification (remove labels? / drop 'label' type and mark all as 'text' / drop metadata for config that has no translation, that can be added later / )
- Exploring the metadata in annotations for config entities.

Jose, can you give us an update on if that is still your plan, or an updated 3 or 4 bullet point on how the plan is now after the newest conversations you have had?

I suggest when you come back with your latest changes, to present them in a couple of patches.
One with the minor clean up and in interdiff for that.
I think the simplification you were first talking about are not in the plan now? But if they are, a separate patch for that if possible with it's interdiff.
And then a patch and interdiff for the big architectural changes you and @chx have been discussing.
And I'll also throw out the idea that any major major refactoring be held off at this point. Because if the changes that come back in the next patch are too wide, I think we'll frustrate the people who have been looking at this and they might give up. So we want to find a way to present the changes so that people can remain active reviewers (I'm thinking of @effulgentsia @fago @yched @alexpott .. and me) and not get so overwhelmed.

Other recent reviewers have just awesome skills and overall knowledge of D8 in general they can probably handle a massive patch. :)

Between Jose's todo list and now, there were some new points brought up though that might change that todo list.

There is some clarification on what cannot go and what needs to stay by @Gábor Hojtsy in #177. (Gabor, I followed most of that, but could you summarize again?)

@chx 's feedback on the merges

And I think @heyrocker looked at this recently and might have a couple of comments.

In general, it looks like a lot of people are coming together and being productive with their feedback and there is some hope for this yet. :)

Gábor Hojtsy’s picture

@YesCT: I don't think my comments in #117 are relevant if we keep something close to the current system, because they were relevant for things that an imaginary plugin / one .yml metafile system would need to fulfil and that does not seem to be the current direction based on chx's review. Also I think/hope the meta-merge-related changes that chx proposed will not be that complicated, some metafiles will be somewhat bigger and the nesting scheme will be easier to understand, which is the developer facing change.

chx’s picture

My changes are not big. Change MetadataLookup::resolveCacheMiss to

$metadata = $this->readMetadata($offset);
$this->storage[$offset] = $metadata;
return $metadata;

remove getBaseName() and that's it. Of course, you need to change a few yml files too.

I was thinking on different approaches (like using data.width as a key and $definition as a value) but that sinks because it can't handle the top level .label and .type and stuff. So reluctantly I think ... we might actually want to go ahead with this :/

chx’s picture

That's oversimplified, it's

$metadata = $this->readMetadata($offset);
if ($metadata === FALSE && $basename = $this->getBaseName($offset)) {
  $metadata = $this->offsetGet($basename);
}
$this->storage[$offset] = $metadata;
return $metadata;

also, here are some other concerns, none of them are big (and I already said this above): config_metadata() should read return drupal_container()->get('config.metadata');. I think we need to rewrite getBaseName() it's very baroque for what could be done with a simple regexp. Also we need a followup for un-hardwiring new MetaDataStorage() as well. (There needs to be a setMetaDataStorage on MetadataLookup and a config metadata factory class which passes in a MetadataStorage object to it. Alternatively wait for the 5.3.10 bump and add another argument to the constructor.)

YesCT’s picture

Thanks for the clarifications. This is sounding good. I think Jose is working hard on getting those changes in and once we have the latest from Jose, we should look at getting it ready for serious consideration to be committed. That will probably involve the usual few rounds of reviews and also creating some followups that document concerns and possible approaches to deal with concerns.

gdd’s picture

After reading the comments above and going through another review I, like others, am becoming convinced that a simpler file architecture is probably not on the table (certainly not for this patch anyways.) The ideas around plugins and annotations are intriguing, but if we want to look into that stuff it will have to happen after feature freeze. I agree with Gabor that it is much more important to get this and #1763640: Introduce config context to make original config and different overrides accessible committed and move forward from there.

I was thinking about actually taking a lot of Gabor's comments in the issue summary and using them as the basis for a big docblock explaining the system as a whole. Most of this seems to be above config_metadata() now, I wish there was a better place for it though. For instance, the large scale overview of the database subsystem is at the beginning of database.inc. Maybe at some point it will make sense as part of a similar block in config.inc. Mostly I'd like to make sure it is somewhere really obvious, because my biggest concern the whole time has been the DX for module developers wanting to provide metadata for their modules. I want to make sure the docs are super-clear and accessible.

Thanks for all the renewed activity on this issue. It has all been really useful and productive.

Jose Reyero’s picture

Status: Needs work » Needs review
FileSize
47.19 KB
81.21 KB

Updated the patch addressing most of the concerns above:
- Removed toList() method, better comments on the difference between configuration metadata and element definition, @fago
- Removed the 'feature' of element's label being taken from the data itself, @fago
- Using 'text' instead of 'label' as data type for all human readable strings (simplifying our aditions to typed data)
- Unify the name style using spaces instead of underscores for properties ('elements key' instead of 'elements_key') as suggested by @Gábor #144
- Do not merge metadata for a name with the base name, but use it as fallback only, suggested by @chx #178
- Removed 'translatable' typed data property, added 'locale context' under text element's definition's 'constraints', which I think should be more consistent with typed data api, @fago?

All of them are small changes, though they add up to some considerable diff so not done yet moving the metadata files to a config/meta subfolder which would have ended up in a huge diff, @effulgentsia #140

About #184, MetadataLookup::resolveCacheMiss(), I had it already done a bit differently but I think the result should be the same. Also renamed getBaseName() to getFallbackName() which is what we are doing now.

Follow up patches should include the full views metadata (which is already done in the config_demo module, just pending these updates) and then metadata for all the other core modules.

Now if there are no other major issues (that cannot be postponed till after feature freeze), it would be nice if we could move this down the rtbc path...

chx’s picture

Assigned: Jose Reyero » chx

I am going to clean up whatever I find and also -- after talking to effulgentsia -- I would like to majorly clean up the meta format by changing to

name:
  .meta:
    label:
    type:
page:
  .meta:
    label
    type
  403:
    .meta
      label
      type

Yay no mixing of what's what! That mixup of config names and $definition elements was very very confusing.

The code change should be small. I do not want to burden Jose with this so I will do it right now.

chx’s picture

FileSize
29.55 KB
86.9 KB

So I changed the format, look at how simple the explanation became: "The metadata has the same structure as the configuration object itself but the configuration values are replaced by an array containing exactly one element. The key is .meta and the value is a $definition array to be passed to typed_data()->create()". No mixups. If you have ever seen a config object and used typed_data->create then you can use this. There is no mix of dot and non-dot. Let me tell you that when I first I reviewed the patch my eyes scanning those dotted keys wanted to put them together (much like config paths are put together) and that so doesnt work and it's so confusing. Gone. Just gone.

I nuked the hardwired classnames from config.inc. Boo.

The rest is just doxygen cleanup, mostly applying the \Drupal rule and in LocaleTypedConfig the source was doc'd as TypedConfig but it's LocaleTypedConfig so I fixed that while at it.

Jose Reyero’s picture

I'm not sure this is really more readable, but if you guys think it is I have nothing against it, maybe using 'definition' instead of 'meta'.

Umm... thinking, the cool thing is this can make metadata extendable for the future, allowing us to add more stuff than the typed data definition in there(under a different (dot)key) :-)

Gábor Hojtsy’s picture

@chx: thanks! Earlier our explanation was "dot keys are just like # keys in formapi", they signify property data instead of nesting. Now, you still use .meta as a dot-key, so the explanation is the same except there is a new sub-level and those sub-keys are typed data info. So when looking at the file in an IDE instead of looking at the key (whether it has a dot or not), you now need to look at the parent. If the parent has a dot, then you are looking at a meta-key, otherwise you are looking at a data structure key. Eagerly waiting for whether or not people agree this is easier.

@heyrocker: are you still thinking of writing more docs? What do you think of chx's addition in #178 (interdiff in #179)?

chx’s picture

FileSize
87.26 KB

Sure .definition is even better.

chx’s picture

FileSize
665 bytes
87.24 KB

See, my friends, this is why you typehint. I fixed a hardwired LocalTypedConfig while in there.

chx’s picture

Re #191, form API is not a good example, I wish we had the time to throw that one out too but that should wait for Drupal 9. I have considered form API as the counterexample of what's good and moved to this. As for looking for the parent, metadata is small so the parernt is never far but also the keys are very telling because now the meta keys are together in one bunch unlike before when you can had a meta key and a nonmeta key in total chaos.

YesCT’s picture

Status: Needs review » Needs work

I tried out the patch from #194. I'm writing up my thoughts and also the steps I used to test drive it and will post soon.

YesCT’s picture

steps to test drive, with specific detail

  1. Apply the patch, install a fresh Drupal 8
    git checkout 8.x
    sudo rm -r sites
    git checkout sites
    git pull --rebase
    git reset --hard
    drush am 1648930
    git add core*
    git commit -m "194"
    drush -y si --account-pass=admin --db-url="mysql://root:root@localhost/d8-patch" --site-name=config_metadata-1648930-194
    
  2. Under Extend, Enable the Locale module (Language module is a dependency so it gets enabled too.)
  3. Under Configuration » Regional and languages » Languages add a custom language, for example: a Language code: yy and Language name: Language YY
  4. Go to the string translation page at Configuration » Regional and languages » User interface translation
  5. Search for strings like "Medium" (should find Medium (220x220)) or "Account cancellation request" (should find the email subject). If it does not find that souce string, might have to visit the page at Configuration » Media » Image styles and click on Medium (220x220).
  6. (optional) If desired to observe the effect later, make a new branch and git add sites/default/files* and commit them. For example:
    git checkout -b metadata-1648930-#194-databefore
    git add sites/default/files*
    git commit -m "before"
    
  7. Translate either the image style name or the email subject. For example translate Medium (220x220) to Middle (220x220)
  8. (optional) For ease, add language switcher block.
  9. Switch to Language YY and go to Configuration >> Media >> Image styles (yy#overlay=yy/admin/config/media/image-styles)
  10. ... see it says Middle..
  11. Observe change
    1. part 1 .yml data files

      Observe change applied by .yml files in active storage appearing with the translation overrides.

      Can observe, if previously commited the config files using git, by doing something like:

      git checkout -b metadata-1648930-#194-dataafter
      git add sites/default/files*
      git diff metadata-1648930-#194-databefore
      

      Which for me resulted in:

      diff --git a/sites/default/files/config_X3nyhr0UiW2Sjh9EyvTFP93nHcf7SRzPGAD1LADO6Oc/active/locale.config.yy.image.style.medi
      new file mode 100644
      index 0000000..6aec9d6
      --- /dev/null
      +++ b/sites/default/files/config_X3nyhr0UiW2Sjh9EyvTFP93nHcf7SRzPGAD1LADO6Oc/active/locale.config.yy.image.style.medium.yml
      @@ -0,0 +1 @@
      +label: 'Middle (220x220)'
      
    2. part 2 in database

      Observe similar data structures appearing in the database as config overrides.

      I did this by searching in phpmyadmin for Middle and selected all the tables to search in.
      in cache_config (row for: cid locale.config.yy.image.style.medium and data blob)
      http://drupal.org/files/configmeta-s01-dbsearch_a-2012-11-16_0753.png
      in locales_target (row for: lid 26 and translation blob)
      http://drupal.org/files/configmeta-s02-dbsearch_b-2012-11-16_0756.png

I'll update the issue summary with a more generic version of these test drive steps.

Note the issue summary is not updated with all the changes from #194. We might want to wait a day or so before totally updating the issue summary to make it exactly match the file examples.

background: these are based off the steps to test drive that were in the issue summary that @Gábor Hojtsy and @effulgentsia used at BADcamp. When I tried the instructions, I had trouble at first when I just added Spanish, getting the strings to show up on the language page Configuration » Regional and language » Languages (like where it says 0/22 (0%) in the Interface Translation column. So I tried visiting pages in the ui to get more strings there, running cron and clearning the cache. Gabor pointed out that we could look in the tests to see exactly what the tests were doing, and they are adding a custom language. So I did that and it worked for me. It may not be necessary, as it should work for other languages, not just Custom languages.

YesCT’s picture

Issue summary: View changes

Updated issue summary.

Gábor Hojtsy’s picture

@YesCT: the feature should not depend on the code of the language being added, so it should either work with non-custom or custom languages. There was probably some other condition that failed for you at first. There is no distinction of custom or non-custom languages once added to the system.

YesCT’s picture

FileSize
11.36 KB

notes and some confusion as I went through this. Note, I haven't looked at the patch, I was just using the patch. Attached is a bit of an irc log as I was going through it, that can most likely be safely ignored.

I dont think any of this is blocking and could be discusses and acted on if need be in follow-ups.
But if it gets re-rolled anyway for something else, then maybe consider some of these things for inclusion.

I linked to some pastes for a somewhat easy way of seeing exactly what I was seeing.

config/image.style.medium.yml http://paste2.org/p/2485789
core/modules/image/meta/image.style.%.yml http://paste2.org/p/2485697

1. Just getting oriented to how the config is stored and what things mean

name is the: Machine name: medium of type string
label is the: Label: Medium (220x220) of type text
effects is the: Style effects: .... listed of type list
it's a list of things that have a name
but part of the definition of each effect has a name... so reusing the word name confused me, but no way around it. it's reasonable. No todo.

2. list: '1'

type: list
would be better
what does the '1' mean? I dont think it means a list of one thing... Can we just leave that out? http://paste2.org/p/2485697#line-12

3. order

I'm distracted… by the ordering
config/image.style.medium.yml http://paste2.org/p/2485789
has order name, data, weight, ied
meta/image.style.effects.%.yml http://paste2.org/p/2485656
has order name, data, weight, ied
ok….
meta/image.style.effects.image_scale.yml http://paste2.org/p/2485684
has order name, weight, ied, data
Since some of these terms like name mean one thing in one place and another in another place, if it doesn't break things, having the order consistant might help a human. The computer does not care since they are at the same level and are associative keys.

4. type on ieid

should there be a type on ieid?
More generally, should there be a type *eventually* for things that dont have a type earlier, for example at level:
meta/image.style.effects.%.yml http://paste2.org/p/2485656
data does not have type there, but does
by the time I get down to the specifics: meta/image.style.effects.image_scale.yml http://paste2.org/p/2485684
everything has a type, except ieid

5. types sometimes are surrounded by single quotes

also in meta/image.style.effects.image_scale.yml http://paste2.org/p/2485684
it's a bit weird the types for the data have single quotes on them.

YesCT’s picture

Based on conversation with Gabor... maybe a day for people to look at this, and then make sure it passes tests and the docs gates (and other gates), get the issue summary updated, and we'll see if we can get it rtbc after a day or so to free up people who have been looking at this to move on to other areas. so there will definitely be follow-ups.

YesCT’s picture

I'm about to start a core gates doc review, do anything I can to address the test fail, and I got one code fix from chx via irc I'll address. I'll be in irc for a few hours until I post back.

YesCT’s picture

+++ b/core/includes/config.incundefined
@@ -169,6 +171,198 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+ * For example, for the config file core/modules/system/config/system.site.yml:
+ * @code
+ *   name: Site-Install
+ *   mail: admin@example.com
+ *   slogan: 'This is an example site'
+ *   page:
+ *     403: 'pages/403'
+ *     404: 'pages/404'
+ * @endcode

My file looks like:

name: Drupal
mail: ''
slogan: ''
page:
  403: ''
  404: ''
  front: user
+++ b/core/includes/config.incundefined
@@ -169,6 +171,198 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+ * The metadata in core/modules/system/meta/system.site.yml:

my file matches this but also has:

  front:
    .definition:
      label: 'Default front page'

Is the intention in the comment to point out the relative parts of the file, and not the entire file?

+++ b/core/includes/config.incundefined
@@ -169,6 +171,198 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+ * The metadata is in core/modules/user/meta/user.mail.yml:
...
+ * For every element, meta/user.mail.%.yml is read as specified by the
+ * 'elements name' list settings:
...
+ * The % in meta/user.mail.%.yml is a placeholder and the array key can be

in most places the whole /core/modules/user/meta/.. is used, do we want to give the full path here too? The context of the comment block is probably sufficient.

I'm still going on the docs stuff. But wanted to ask this in the meantime.

Jose Reyero’s picture

Looking at #194, the reason why the tests are breaking is because, as it's using DIC now instead of static variables, after a new module is installed, the list of modules/config names in InstallStorage doesn't get updated.

Instead of regenerating the whole DIC, I think this is a good chance for a minor improvement to the InstallStorage, so it can read config/meta for installed and uninstalled modules so this very small (2 lines) patch will use drupal_system_listing() instead of module_list(), which makes sense since this is an storage used for installing modules and themes, thus it should allow fetching the module config before the module is actually installed.

YesCT’s picture

Non critical, but I'm curious:

+++ b/core/modules/locale/lib/Drupal/locale/LocaleTypedConfig.phpundefined
@@ -0,0 +1,257 @@
+  /**
+   * Get translated configuration data.
+   *
+   * @param \Traversable $elements
+   *   Configuration elements.
+   * @param array $options
+   *   Array with options that will depend on the translator used.
+   *
+   * @return array
+   *   Configuration data translated to the requested language.
+   */
+  protected function getTranslatedData($elements, $options) {

That \ seems unusual. I wonder if it should be

* @param Traversable $elements
and
protected function getTranslatedData(Traversable $elements, $options) {

Other examples of Traversable that I could find were in twig, and those params had other types like:

* @param array|Traversable $brokers

So their parameters in the fuction were not typed.

YesCT’s picture

This patch includes the test fix from the interdiff from @Jose Reyero in #204

and it includes the note I got from @chx in #202 (I attached a interdiff just for that.) [edit: for grammar]
chx doubted that $textarea = current($this->xpath('//textarea')); was E_STRICT compatible.

in general

I only changed things that were already changed by the patch from #194. Several of the files that this patch touches in general could use some docs cleanup and general code standards stuff. I saved some notes.

docs changes

  • C1: I only re-wordwrapped one or two comments that were over 80 chars.
  • C2: There were some more namespace used in @ directives where I added \ There could be a follow-up to handle the files in general. But that's also a core wide thing too, so we dont have to worry about it. I just wanted to clarify that I added them only in the code that this patch added or touched.
  • C3: there were a few functions that were missing @params, types in the @params, or @returns
  • C4: I made the comments that have examples of the core yml files match exactly the files, after confirming with Jose in irc about my question in #203

not changed

  • NC1: (not a change, follow-up) There are some comment blocks that the lines were under 80 and could have more wrapping, but since that is not a core gate. I/we can open a follow-up for that. http://drupal.org/core-gates#documentation-block-requirements
  • NC2: (not a change, no change needed) at the end of #203 those partial file names are fine since it's clearer with the short names and they are referring to files in the same folder.

docs questions

  • Q1: core/modules/locale/lib/Drupal/locale/LocaleTypedConfig.php
    had some strange language, using the word translator. it's used a few other places and is not referring to the people. :)
    * TRUE if this translator supports translations for these languages.
  • Q2: in core/includes/config.inc
    some of the @see's are not at the end of the doc comment block. is that for clarity? or should we move them to the end all in one group?
    + *         label: 'Width'
    + *         type: 'integer'
    + * @endcode
    + *
    + * @see \Drupal\Core\TypedData\TypedDataManager::create()
    + * @see \Drupal\Core\Config\Metadata\ElementBase
    + * @see \Drupal\Core\Config\Metadata\ListElement
    + *
    + * @param string $name
    + *   The name or key of the configuration object, the same as passed to
    + *   config().
    + *
    + * @return \Drupal\Core\Config\Metadata\MetadataLookup
    + *   A metadata array.
    + *
    + * @see \Drupal\Core\Config\Metadata\ElementInterface::getMetadata()
    + */
    +function config_metadata($name) {
    
  • Q3: One change of mine I reverted and left out was:
    Putting types in the function param list... but in core/modules/locale/lib/Drupal/locale/LocaleTypedConfig.php
    -  public function translateElement($element, $options) {
    +  public function translateElement(TypedDataInterface $element, array $options) 
    

    caused an error

    An AJAX HTTP error occurred. HTTP Result Code: 200 Debugging information follows. Path: /d8-patch/batch?id=2&op=do_nojs&op=do StatusText: OK ResponseText: Recoverable fatal error: Argument 1 passed to Drupal\locale\LocaleTypedConfig::translateElement() must be an instance of Drupal\locale\TypedDataInterface, instance of Drupal\Core\TypedData\Type\String given, called in /Users/me/foo/d8-patch/core/modules/locale/lib/Drupal/locale/LocaleTypedConfig.php on line 165 and defined in Drupal\locale\LocaleTypedConfig->translateElement() (line 193 of /Users/me/foo/d8-patch/core/modules/locale/lib/Drupal/locale/LocaleTypedConfig.php).

Next steps

I tested it locally just using the test drive steps and it worked.

If it comes back from the test bot as passing, then I think we update the issue summary... and get someone to give this a final look to mark it RTBC.

YesCT’s picture

Status: Needs work » Needs review
Jose Reyero’s picture

Great review!

Moving to 'needs review' to trigger testbot.

About the questions:

Q1: As this object 'translates' I refer to it as a translator but yes, we could reword it as 'this object' or whatever makes sense.

Q2: I think comments are more 'human readable' if we have the '@see' on the related content, but since it seems to be some coding standard, we can move it to the end.

Q3: It is a namespace issue, that would be fixed with 'use Drupal\Core\TypedData\TypedDataInterface' (if not it uses the current class namespace, which produces that 'Drupal\locale\TypedDataInterface' which doesn't exist)

However since we only iterate that argument, I think Traversable is ok for that one.

chx’s picture

Q3 it is NOT a namespace issue, why do you think it is? Read the error message.

Edit: nevermind, I see now that it is: class String extends TypedData implements TypedDataInterface, above it: use Drupal\Core\TypedData\TypedDataInterface. This should not stop commit of course.

fago’s picture

Good work - improvements look good to me. :)

I found nothing commit-blocking, but here some remarks:

- Removed 'translatable' typed data property, added 'locale context' under text element's definition's 'constraints', which I think should be more consistent with typed data api, @fago?

I don't think 'locale context' fits under 'constraints' - everything below 'constraints' will probably used by the validation api - see #1696648: [META] Untie content entity validation from form validation. It cannot be below 'settings' as this is the playground of implementation classes, so I think it should be just at the main $definition level. It's an extension to typed data definitions that is specific to config data definition, so I think it should be documented at config_metadata(). Then what about the ability to opt-out from translation, e.g. by specifying false as context? Or 'translatable' => FALSE ?

+++ b/core/includes/config.inc
@@ -169,6 +171,202 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+ *  status_blocked:
+ *    body: "[user:name],\n\nYour account on [site:name] has been blocked.\n\n--  [site:name] team"
+ *    subject: 'Account details for [user:name] at [site:name] (blocked)'
+ *   status_canceled:
+ *     body: "[user:name],\n\nYour account on [site:name] has been canceled.\n\n--  [site:name] team"
+ *     subject: 'Account details for [user:name] at [site:name] (canceled)'

Inconsistent indentation.

+++ b/core/lib/Drupal/Core/Config/Metadata/ElementBase.php
@@ -0,0 +1,293 @@
+  protected function buildElementMetadata($key, $data) {
+    return isset($this->metadata[$key]) ? $this->metadata[$key] : array();
+  }

We should use if/else here to avoid unnecessary array copies. See #1838368: Do not blindly use the ternary operator; it always returns copies of non-object values

yched’s picture

Sorry, spent a couple days on the road, and my phone's browser doesn't let me post on issues once the page gets too large :-/.

I very much regret the obligation for plugin implementations to provide a metadata yml snippet file in addition to the implementation class, but I am fine with waiting after Dec 1st to try and revive an approach based on #1764380: Merge PluginSettingsInterface into ConfigurableInterface - esp. given that we currently have very little systems that are currently both 1) implemented as plugins and 2) used in systems converted to CMI. Field formatters used in views is the only example that comes to mind.

However, and regardless of possible followups, the nesting model in this patch still feels off :
Each image effect provide image.style.effects.[effect_name].yml matadata files, whose name and content make assumptions on the name and content of the containing image.style.[style_name].yml config files. In fact, each of them actually *duplicate* the 'weight', 'name', 'IEID' entries found in the container.
That really doesn't look right, means all kinds of unsync could happen - for instance, this prevents the supersystem (image style) to change its internal format in point releases without breaking all of contrib effects.

In other words, the 'looping' and 'include variable part' mountpoints, which are currently merged, should rather be distinct- something like the following structure (names of the 'loop' and 'include' keys voluntarily made up) :

- image.style.%.yml (provided by image.module, which defines ImageStyle config entity)
name:
  .definition:
    label: 'Machine name'
    type: string
label:
  .definition:
    label: 'Label'
    type: text
effects:
  .definition:
    label: 'Style effects'
    loop_on: 'image.style.effects.%'

- image.style.effects.%.yml (ditto) :
name:
  .definition:
    label: 'Machine name'
    type: string
ieid:
  .definition:
    label: 'IEID'
weight:
  .definition:
    label: 'Weight'
    type: integer
data:
  .definition:
    label: 'Data'
    include_from: 'image.effect.%[name]'

- image.effect.%image_scale.yml (provided the module that provides the image_scale effect)
.definition:
  label: 'Image scale'
  width:
    .definition:
      label: 'Width'
      type: 'integer'
  height:
    .definition:
      label: 'Height'
      type: 'integer'
Gábor Hojtsy’s picture

@yched: that is funny, because the metadata structure was similar (although the image.style.effects.%.yml and name specific metadata merges were implicit, not provided in the file). @chx asked for that to go away ASAP (see #180), so those two files were merged. In the patches before 180, the image styles were defined in image.style.%.yml which referred to image.style.effects.%.yml which was the general file that got merged to effect specific files as needed, such as image.style.effects.image_scale.yml, replacing the effect name like you suggested. So the general style specific metadata was in image.style.%.yml while the general effect metadata was in image.style.effects.%.yml and the specific metadata for this effect was in image.style.effects.image_scale.yml. That is essentially what you suggest with a little bit more explicit referencing. What happened after #180 is that the general metadata file for the effect is not merged anymore, so the specific metadata file should specifically include all properties for the effect, even general ones. This is not to provide higher level metadata in a lower level, like you imply, it is just to provide the metadata for the general base implementation in the specific implementation as well (instead of any merge happening). So although it is called image.style.effects.image_scale.yml, it DOES NOT contain property information for the styles, it merely contains property information for the effect. It just contains all general properties as well, which @chx said in #180 that it should absolutely must happen. What do you think?

chx’s picture

Before vilifying me, #180 still was .label and stuff which made an absolute mess, now we are at .definition which is a lot clearer. Ie. you had a structure which had randomly metadata and non-metadata thrown in, merging that was an absolute, horrifying mess. Also there were no documented merge rules. So we can go back to merging if that's what people want but the merge rules must be explicit and documented.

chx’s picture

Assigned: chx » Unassigned
FileSize
5.49 KB
89.08 KB

OK, I did what yched wanted, hoping that we can now RTBC this. I unassign, it was a mistake from me to get involved with CMI again, I wish I didn't.

YesCT’s picture

Assigned: Unassigned » YesCT

I re-read the recent comments. Most can be addressed in follow-ups and are non blocking.

The first part of @fago in #210 regarding locale and contraints can be a follow-up. I think some discussion will be needed to orient some of us to "contraints".

The second minor part @fago mentioned regarding indenting has been fixed in the patch from #214

Fixing the ternary operator is not done yet and can be addressed in a follow-up (or it is straight forward enough it can be in another re-roll here, if one happens, we might not need one here, and we don't want to just keep re-rolling).

We might want to double check with @yched that #214 unblocks.

My plan for today is to do what I can so that others can focus on other things. There is so much going on these days!

My plan:
post some notes that resulted from an irc discussion about the unsyncing concern that was mentioned in #211
manually test
verify actual files match the examples used in the docs in the code
update the issue summary to match the current state

YesCT’s picture

Earlier I was thinking about #211
@yched

all kinds of unsync could happen - for instance, this prevents the supersystem (image style) to change its internal format in point releases without breaking all of contrib effects.

Here is what I understand based on an irc conversation with @Jose Reyero and @Gábor Hojtsy. I'm not trying to put words in their mouths, and this is not a direct transcript, but it is a digestion of the information, and in places I attribute their helpful clarifications to them. :)

This are not as bad as it might seem. We are going around discussing how to represent it with files and how to reference the files together. So the rest of this is in good shape, and nailing down that representation can totally fit under http://buytaert.net/updated-drupal-8-release-schedule post feature freeze: "..Refactoring of existing subsystems, smaller API changes and API additions are allowed if they help improve consistency and coherence of the existing functionality or are necessary in order to resolve specific critical tasks and bugs."

At first I was thinking yched is saying that some information is repeated in the nested files (file namings) and that if a file changes, that it wont get propigated to the other files that have that repeat info and so stuff can get out of sync. but is that really the case? or are the files regenerated when a change is made? So I took a step back and tried to understand how do these yml files get changed? are they saved as a result of changes via the ui only (so, that duplicate info would get saved ok, all the places it's duplicated will get the new values). Could someone edit one of the yml files that has some of the data that is duplicated, and push it up to their site… and then do some kind of cmi "reload", and that is the concern? [no one needs to answer these questions. I got them answered and what follows is my new understanding.]

Jose pointed out: But these files are not that kind of cmi information. They are hardcoded with the module so it's not supposed to be changed.

I recall doing the test drive, and changes I made in the ui to translation configuration were saved (*just* the changes were saved) in sites config data, for example: sites/default/files/config_X3nyhr0UiW2Sjh9EyvTFP93nHcf7SRzPGAD1LADO6Oc/active/locale.config.yy.image.style.med and not saved here with this meta data. That realization helped clarify things for me since I was not too familiar with the CMI stuff before. Gabor pointed out to me: The files with modules should be as-is all the time, the changed files are in the active store / staging store with your sites/*/files

Also, these are effects. Gabor thought yched might be thinking of the styles container which is one level up from here.

I thought it might help to come up with steps to reproduce a condition of the files being out of sync, which lead to the following.

it's about maintainability from the perspective of core components or module mantainers. (and they can pay attention and be careful to change things in the duplicate places). Jose agreed: Since we don't have any 'shared' metadata for 'image.style.effects...', effects provided by modules need to duplicate it all and then if the main module, say, adds some property to all image style effects, we need to update all of them. [This is reference to the patch from #206, not #214]

Gabor explained the relation between the concern and the patches: well, image effects are base classes, which specific effects inherit (or if not yet, they will probably be like so). [regarding the patch in #206] if core adds something to the base class, then all contrib modules implementing effects would need to adjust their metadata. whereas in the version before #180, they would not need to.
I think with the #214 change to address yched's concern and take out some of the duplication, it might not be the case now, if I'm understanding things. With the most recent patch, modules would not need to update their metadata if something they depend on (core) adds something to the base class.

yched’s picture

re @Gabor #212 / @chx #214:
Latest patch is better in the sense that it removes duplication of properties from the image style level within effect metadata info.

However, there is still the major drawback that effect metadata are only included because the files are named after their container (image.style.effects.[effect_name].yml). So, won't work for plugins being used in different ConfigEntities - again, typical example is field formatter, used in regular entity displays and field views - and panels in D7 / possibly blocks in D8. All of those cases will have different CMI structures and properties,
- Why would views.displays.fields.%.yml fetch snippets from entity.view_modes.fields.formatters.text_summary_or_trimmed.yml ?
- There's no guarantee that entity.view_modes.fields.formatters.[formatter].yml will "merge" nicely in whatever core or contrib context that will want to use them.

So IMO it's not just about naming :
The current patch could be changed so that effects metadata files are renamed image.effect.[image_scale].yml.
But then, I guess image.style.effects.%.yml (that contains the "generic" properties inside each effect, like 'weight', 'ieid'), wouldn't be included anymore ?

I think it comes down to my point in #211 that the current patch has only one single mechanism for "loop and include" (which thus does a merge of "defaults" and "specific"), while what we in fact have is a three-level (not two-level) stack, with different 'loop' and 'include' points.
1) image.style.%.yml describes the 'name', 'label', 'effects' properties.
The latter involves a 'loop' on image.style.effects.%.yml
2) image.style.effects.%.yml describes the 'name', 'ieid', 'weight', 'data' found within 'effects'
The later has an 'include' on image.effect.[name].yml, using the value of the 'name' key
3) image.effect.*.yml files are completely agnostic about the above.

I'm not even sure that level 2) ('loop') actually requires a separate file, btw. We could probably find a syntax that would inline the information that "all entries beneath 'effects' will be structured that way" within the 'effects' entry in image.style.%.yml itself. In which case, looks like this also could be used for user.mail.yml.

Jose Reyero’s picture

re @yched #217

However, there is still the major drawback that effect metadata are only included because the files are named after their container (image.style.effects.[effect_name].yml). So, won't work for plugins being used in different ConfigEntities

Not really, we are defining 'elements name' in the metadata too so they could be under any other namespace. From image.style.%.yml:

    list settings:
      elements name: 'image.style.effects.%'
      elements key: 'name'

The only assumption we are making here is that any 'image effect' will be defined under the same namespace, which I think makes sense since any other module providing an image effect can use too image.style.effects.xxxx.yml metadata.

....All of those cases will have different CMI structures and properties,

When we are talking about plugins used in different places, there should be a generic description somewhere I guess, with all the data that plugin can handle. If some properties are not present in whatever configuration, then no problem, we use only the metadata to describe 'actual' configuration, non existent properties will be ignored.
On top of that, specific cases can use a different definition if needed, and that is why we have the config/config base overrides, that are not tied to the container name, you can use any.

The current patch could be changed so that effects metadata files are renamed image.effect.[image_scale].yml.

Right, this makes sense if 'image.effects' are not specific of 'image styles' (I guess they are not, so adding to the 'todo')

In summary, this metadata system *works for all objects in the current core configuration* (Included views which are kept atm in a sandbox module to avoid big rewrites). So I think future possibilities should be kept in mind but not be a blocker for this patch.
On top of that, we are using the typed data api to define nested elements so any future contrib module whose configuration doesn't fit this schema, can define its own 'config metadata' element classes, to handle its own metadata.

And last, if you still have major concerns about 'nesting levels', would a '.include' directive in the metadata fix that? (Not an option I dislike but so far we haven't needed it for current core configuration...)

Jose Reyero’s picture

Status: Needs review » Needs work

So, after a long talk on IRC with @YesCT and @yched, trying to convince @yched his objections didn't make sense, actually he convinced me of the opposite. I'll try to put the issue on my own words:

For nested objects, like 'image effects', there are two parts in the configuration:
- One that only depends on the 'effects' (data, name)
- Another one that depends on the 'container object' (image style): weight, ieid

If reusing that same 'image effects' inside a different object (never done in D8 yet) but reusable components is certainly a goal, we need a different piece of metadata to describe that same effects. While the common part (data, name) may be the same, there may be other properties instead of 'weight', 'ieid', etc, depending on how the container object handles its plugins.

This can be fixed either by using include directives (though they may need to take a 'key' too, like '.include image.effect.%.yml') or maybe with the (undocumented, completely missed but it is in the patch 'elements_base' property).

Short: doing a reroll, fixing this issue and @fago's latest comments (constraints) too.

YesCT’s picture

Assigned: YesCT » Unassigned

@yched++ for good conversation regarding "separation of concerns" ... which I googled: http://en.wikipedia.org/wiki/Separation_of_concerns
Unassigning from me since waiting on the new patch before updating the issue summary.

Gábor Hojtsy’s picture

@Jose Reyero/@yched: thanks, I finally understood from #219 what is the concern and agree this needs a solution :) One more level of merges seems like will complicate this system even more, but as @yched argues it is needed to describe the complexities that the data structures and their wrapping/nesting can be involved in.

Jose Reyero’s picture

Once stablished we need some kind of include directive (which will solve multiple levels of nesting), the hard thing is not how to get the feature, but to choose between the many options we have (.include, 'element include', etc..) and implement just one because we don't want to duplicate the functionality, right?
(like adding 'include' to current 'list settings' that would provide many different options to include the same things)

I've been considering many different options for a while and here's some idea. Let's keep in mind our number #1 goal should be *making metadata understandable and easier to write* for module maintainers, not saving a few lines of code in this patch.

It goes like this:
- Drop all 'list settings'.
- Add '.include' directive with some replaceable [variables], see the examples.
- Add '.element metadata' directive that will contain the metadata for nested elements

Example: user.mail.yml:
* We don't need two files anymore unless the 'mail' structure is to be reused)

.definition:
  label: 'User mails'
  list: '1'
.element metadata:
  .definition:
    label: 'Mail text'
  subject:
    .definition:
      label: 'Subject'
      type: 'text'
  body:
    .definition:
      label: 'Body'
      type: 'text'

Example: image.style.%.yml
• Again we don't need two files as the data for 'image effects' that is specific to 'image style' is contained here too
• We have an include directive but instead of a generic '%' for replacement it contains the full 'key name' to use for replacement, in this case it is '[name]' that will be replaced by the element's 'name' property (like 'image_scale')

name:
  .definition:
    label: 'Machine name'
    type: string
label:
  .definition:
    label: 'Label'
    type: text
effects:
  .definition:
    label: 'Style effects'
    list: '1'
  .element metadata:
    .include: 'image.effect.[name]'
    .definition:
      label: 'Image style effect'
    weight:
      .definition:
        label: 'Weight'
        type: integer
    ieid:
      .definition:
        label: 'IEID'

Example: image.effect.image_scale.yml
• Note this adds an explicit 'include' for the common image.effect properties (% is not replaced anymore)
• We are doing nested includes here: 'image.style.%' includes 'image.effect.[name]' which includes 'image.effect.%'

.include: image.effect.%.yml
.definition:
  label: 'Image scale'
data:
  width:
    .definition:
      label: 'Width'
      type: 'integer'
  height:
    .definition:
      label: 'Height'
      type: 'integer'
  upscale:
    .definition:
      label: 'Upscale'
      type: 'boolean'

I think this has some important advantages:
• There are less files, by using '.element metadata' we can have all the specifics for an object in the same file.
• The structure is closer to the main config data structure. Only the nested key is always '.element metadata' (instead of the 'mail name' for user mails or the 'image ieid' for image styles.
• We can have unlimited nested includes, but they are always explicit so a given metadata file can include any other, making it more flexible in case some plugin is reused differently for a given configuration object.
• The include names can use any name or more than one from the configuration data. (.include: [name].[type].[etc..]) and I'm planning on adding a special one for the key elements are indexed by, like '[%key]', which I'm afraid is going to be needed at some point.

I'm finishing the patch for this (which is not more complex nor longer than the previous one) but since there are some changes I wanted to get some feedback about them before if possible. And anyway I think we should focus more on the metadata format, which is the important thing and not that much on the implementation detail. Actually the hardest part of the patch is rewriting the comments :-) which may take me a while.

Jose Reyero’s picture

As suggested by @Gábor on the irc, the 'list:1' looks weird and we don't really need it, so considering this update:

Replacing '.element definition' by '.list' and we don't need explicit lists anymore (will be assumed by our parser when we have a '.list' directive, it will look like this:

.definition:
  label: 'User mails'
.list:
  .definition:
    label: 'Mail text'
  subject:
    .definition:
      label: 'Subject'
      type: 'text'
  body:
    .definition:
      label: 'Body'
      type: 'text'
YesCT’s picture

taking out

list: '1'
 .element metadata:

and replacing it with

.list

is nice

if I'm understanding, it's like type: list, where what follows is then the list.

the words that start with . they are part of the syntax that this patch is defining.
why did we lose the . on .label and .type?
These system/meta yml files match up with system/config
And the key words from system/config
appear in the system/meta without the dots. That makes sense to me, and makes it easy for me to see which words are part of the syntax this patch is defining, and which come from the system/config files
For example, I can tell because "effects" does not start with a dot that it's not part of the syntax of this patch, but comes from the system/config file.

I'm still not understanding why sometimes the types (integer) have single quotes and sometimes they do not.
If there is no reason, please do not use quotes around types like boolean, text, integer

it makes sense to me that the value for the labels uses single quotes.

YesCT’s picture

Oh, I see in #189 the .meta was introduced (which is nice and became the .definition). That made it so we dont need the . on label and type. OK.

Jose Reyero’s picture

Status: Needs work » Needs review
FileSize
23.49 KB
90.79 KB

New patch containing:
- Changes explained in #222 + #223 (upated metadata accordingly)
- Some minor cleanups in the metadata: removed 'string' type (default), not quoting types (suggested by @YesCT)
- Updated tests and added some tests for the config includes
- Moved 'locale context' to definition + added 'translatable' flag that can be set to false (@fago 210)

I've updated the documentation accordingly, though I'm assuming that will need some fixes yet. Since the one on top of config_metadata() function is really big, could we consider moving it to a separate (follow up) issue?

Jose Reyero’s picture

Issue summary: View changes

Updated issue summary steps to test drive.

Jose Reyero’s picture

Issue summary: View changes

Updating elements, wip

Jose Reyero’s picture

Issue summary: View changes

Updated the rest

Jose Reyero’s picture

Issue summary: View changes

Added more explanations and justification for multiple levels of nesting

Jose Reyero’s picture

Posting new version that I'll explain later (experiencing troubles with laptop atm)

Jose Reyero’s picture

About the previous patch:
After talking with @yched again on irc and checking the image code we came to the conclusion that for image style effects, all the information depends on the container (image style) and only the 'data' array is specific of each effect.
This is basically supported by the existing code with an small addition: to include the specific effect, we need to use for the include a value that is on the containing data so we need something like this:

effects:
  .list:
    name:
      ...
    data:
      .include: 'image.effect.[%parent.name]'

Nothe the name we need to use is not in the 'data' array but one level higher.

This last patch supports this and other more complex naming schemes in case we need it, like:
- Parents multiple levels higher: [%parent.%parent.%parent.keyname]
- Values a few levels below: [.key.subkey.subsubkey]
see the comments in ElementBase::replaceName() for details

Image metadata has been changed again, and metadata has been added for all existing image style effects.

Note: As I wrote above, I am experiencing some ugly troubles with my laptop atm (writing this from somewhere else) so I may not be able to work more on this for a while.

Gábor Hojtsy’s picture

Is this extra syntax worth it so that we don't have as many files to juggle but this much more complexity in the files?

yched’s picture

Yay - this works for me :-)
Thanks for bearing with me, Jose - er, and good luck with the laptop...

Nitpicks:

The metadata samples for image.* in config.inc are out of sync

+++ b/core/modules/image/meta/image.effect.image_scale.yml
@@ -0,0 +1,5 @@
+.include: 'image.effect.image_resize'
+upscale:
+  .definition:
+    label: 'Upscale'
+    type: boolean

That works, but strictly speaking, that's not really correct, the 'scale' effect doesn't extend the 'resize' effect, they just happen to have some settings that have the same names and metadata (width and height). It seems wrong to introduce coupling on their metadata where there is no coupling in the implementation.
I guess the intent is to provide an example use case for includes, but as is, I fear it encourages a use of the pattern in cases we wouldn't want to encourage it :-)

Jose Reyero’s picture

I think this is a small addition, that may be temporary, if we ever get a clean definition of how plugin configuration is included into a container configuration object. But right now we need to support these use cases in Drupal core configuration.
About the syntax, it's the simpler I can think of. The multiple levels of %parent or the nested properties [name1.name2] may not be needed eventually but right now we are dealing with the lack of consistency and definition in the configuration data across modules, and also the configuration system itself has serious issues to figure out depencencies, like the latest addition in the form of 'manifest' configuration.

yched’s picture

Status: Needs review » Needs work

if we ever get a clean definition of how plugin configuration is included into a container configuration object

I doubt we will ever get that. Some config objects happen to refer to stuff that leverage plugins during runtime execution of whatever the config object is supposed to represent, and thus they happen to include the information to instantiate those plugins. But that is a "soft" dependency, that cannot really be formalized anywhere IMO.

Also, I'm not questioning the need for includes in general - just saying that using them this way on image.effect.*.yml provided in core is not a good practice to encourage :-)

Gábor Hojtsy’s picture

So looks like it is needs work on #230 and it not passing.

Jose Reyero’s picture

The test not passing is not big deal, it should be just the metadata test that I didn't have the time to update properly (with new metadata)

the 'scale' effect doesn't extend the 'resize' effect

No, it doesn't though the implementation of scale (and many others) reuses the form definition from resize, see image.admin.inc.
Anyway, I agree it may not be a good idea, I don't mind adding a few 'width, height' here and there, I'll do on the next version for all the image.effect.*.yml
Metadata includes work nicely though :-)

Jose Reyero’s picture

Status: Needs work » Needs review
FileSize
94.64 KB
12.9 KB

New patch which fixes tests and adds a few things:
- Metadata for image effects now without includes, @yched #230
- Cleaned up includes and variable replacement and metadata preprocessing.
And:
- Added back '.label' and '.type' as a shorthand for .definition['label'] and .definition['type']. The reason for this is making metadata easier to write and read, while still supporting the full '.definition' array.

Really, as I've experienced myself when writing (big) views metadata, needing to write always the '.definition' array is a pain.
I've created some views metadata with plugins/fields/filters/etc/etc to see whether it fits nicely with the new model, includes, etc, and the answer is yes. The only issue is with plugin definitions written in hook_views_data() (see node.view.inc) but I think that needs to be upgraded anyway in views to use something else (because otherwise we cannot determine dependencies from the views data)
This is how (working) views metadata looks like with all the plugin mess: http://drupalcode.org/sandbox/reyero/1635230.git/tree/8facc18e7fd8fd09d9...
We may need some bether namespaces for plugin metadata, as mentioned before, to be able to write the metadata side by side with the plugin but anyway that should be a follow up patch.

effulgentsia’s picture

I haven't reviewed recent code, but the meta YAML files in #236 look really good to me. I still have hope for getting plugin related metadata out of meta files and into the plugin classes. Especially for Views, where there's hundreds of plugins, I think that will be important for DX. As part of that, I'd also like to re-evaluate #16
: if the dynamism of Views is really just a dynamism of plugin wiring, but if each individual plugin's configuration schema is relatively well defined, then quite possibly, Rx could express it. But, that can all be a follow up.

Returning to this issue's scope, it now seems odd to me that we have .definition inside the yaml files, but the folder name is meta. Can we rename to definition? I also find it odd for this folder to be outside of the config directory. If it's possible to move into the config directory without messing up existing CMI code, let's do that. Otherwise, how about renaming it to config_definition until such time as we're able to move it inside?

sun’s picture

Naming is important. But I guess it could be addressed and adjusted in a follow-up issue.

That said:

Neither ./meta nor ./definition directories really make sense to me. I'd actually prefer to go back to to '.meta' keys in the metadata files, just because it's shorter and equally gets the point across. And in light of that and earlier comments, a ./meta directory within an extension doesn't really make sense, because there is no relationship to config there. I'd see two possible resolutions: 1) Either change ./meta into ./meta/config on the grounds/idea that we might have further stuff that could use metadata like this, 2) Move ./meta into ./config/meta (inherently accepting the fact that this would prevent a "meta" context to exist for configuration, since all subdirectories should be considered as context override sources).

YesCT’s picture

This is scaring me a little. With 4 days left, I dont know if we should be changing something like that. Really, I think we have just enough time to write good documentation that will let someone be able to understand this issue and what it is doing.

@sun do you feel strongly? Can we get @heyrocker to check out the current status?

Gábor Hojtsy’s picture

@Jose: I don't think having two alternate syntaxes is going to make things easier. It will make things harder to understand. We need to agree on one syntax IMHO. It sounds like it would be a major pain to follow for humans if your contrib module uses an include to include some core plugin with the including file using different syntax vs. the included file. I'd like to reiterate this that the three things we are going in circles about is: (a) the placement of the metadata files (b) the syntax of metadata types (c) nesting and inclusion. Nobody is arguing about the underlying architecture of the classes or the reuse of TypedData, etc. All we seem to be circling around is how the files express the data structure and where are the files located. I think these are details we can tweak further extensively for any given length of time until API freeze (if people wish so that is), especially that they do not affect the overall architecture at all.

I personally don't mind if its ./config/meta or ./meta or ./definition or ./config/definition or ./config_meta or whatnot (or that we pull metadata from annotations in some cases, not .yml files), these are relatively easy things to adjust. The key seems to be that everybody agrees we need this feature and nobody is arguing about the architectural choices of the feature or how it maps to the config system, how it is being used by the locale system, etc. To me that says we should settle down instead of swinging around and get the feature in.

effulgentsia’s picture

I completely agree with #240. I would RTBC this myself if I understood the code details well enough, but I haven't taken the time to absorb it well enough yet. But, I have no objection to this going in as-is, and deferring the rest to follow ups, if someone feels they understand the code well enough to RTBC it.

Jose Reyero’s picture

@Gábor,

I'd like to reiterate this that the three things we are going in circles about is: (a) the placement of the metadata files (b) the syntax of metadata types (c) nesting and inclusion.

Yes, this is a good summary of the latest 100 comments here :-)
Also it seems most people here would agree on postponing most of this fo follow up issues.

a) About the placement
I don't mind that too much either but really the two places that make more sense are: as close to the configuration data it describes as possible or as close to the code using it (Plugins?, Config entities?) as possible.
@sun #238

2) Move ./meta into ./config/meta (inherently accepting the fact that this would prevent a "meta" context to exist for configuration,

Moving to '/config/meta' would be an option. We are not using any folder context at all atm, are we? Are they planned?
And anyway this sounds to me like another one to address later depending on whether we have other folders or not eventually inside the 'config' folder.
I think for now, having its own folder ('meta') will keep it clear and separated from any other 'problem', then we can make it more 'compact' later if there's place for it.

b) About the syntax

I don't think having two alternate syntaxes is going to make things easier. It will make things harder to understand. We need to agree on one syntax IMHO. It sounds like it would be a major pain to follow for humans if your contrib module uses an include to include some core plugin with the including file using different syntax vs. the included file.

Agree about the first sentence, but about includes, humans never get to see the merged data (which has been preprocessed, so let's think of it as some internal data format).
So if the problem is to make it consistent, how about using always:
- '.type' and '.label' (which is all you're going to need for 99% of the metadata)
- '.definition' array for any other TypedData API property so we can always use the full power of this API if needed and the thing is still extendable.

c) Nesting and inclussion
I think this is ok as it is atm. I mean the problem here is having it as simpler as possible, still being able to describe whatever config we have. For this, includes with variables ([]) which is what we have implemented look good to me and they work (and that's why I've been busy doing all that metadata for views, just to see whether it works)

So, in short, I would fix (b) for consistency rewriting current core config with '.label' and '.type' and this should be RTBC (?).

Then there may be some follow ups for polishing depending on how the rest of Drupal core ends up looking like, whether we eventually have context folders in config, the format we end up using for node type and views configuration, how plugin dependencies in configuration are eventually worked out (manifest?) etc...

YesCT’s picture

Status: Needs review » Needs work

Jose, I agree with the plan to fix just b, and then handle the others in follow-ups.

If possible, when you fix b, please update the issue summary. And then lets get a committer to get a look at this. :)

Jose Reyero’s picture

Status: Needs work » Needs review
FileSize
16.93 KB
92.2 KB

Updated:
- metadata in modules to use '.label', '.type'
- documentation and tests.

YesCT’s picture

Assigned: Unassigned » YesCT

I'm going to work on getting this ready.

YesCT’s picture

Issue summary: View changes

Updates about translion api

Gábor Hojtsy’s picture

Updated architecture diagram to be included in new issue summary (which YesCT is working on). The upper right section changed from before, the wrappers were renamed to TypeData, LocaleTypedData, and a factory is included for them.

YesCT’s picture

This is api docs clean-up, added types to function parameters, corrected some meta file examples in the comments, removed a bunch of blank lines from the meta files.

I only added the \ for namespaces in lines that this patch was already touching. Some files that had no other changes, had some of \namespace fixes, but they were not consistant, and made the patch longer than it needed to be.

I have some follow up questions I'll post. But I think it will be easier to use dreditor to do it on the new patch and interdiff.

YesCT’s picture

0. Follow-ups

I updated the issue summary to add a section for follow-ups.
If you have refactoring suggestions, please make a follow-up and link it in the summary.
But here, let's just work to polish this to get it ready for a committer to review.

Specifically,
Please unless it is critical to functionality, lets take our time to discuss the best file format in #1851498: Polish file format of configuration metadata (for translation).
And the best filenames and locations in #1851502: Better placement of meta files for configuration metadata (for translation).

1.

+++ b/core/includes/config.incundefined
@@ -169,6 +171,209 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+ * The metadata has the same structure as the configuration object itself but
+ * the configuration values are replaced by an array containing some special
+ * elements, whose keys are prefixed by a dot:
+ * - '.type', the 'type' property for the data type definition of the element.
+ * - '.label', the 'label' property for the data type definition of the element.
+ * - '.definition', array containing any other property for the data type
+ *   definition of the element.
+ * - '.include', name of additional metadata file to be added. The current
+ *   metadata will then be merged on top of it using
+ *   NestedArray::mergeRecursive()
+ * - '.list', contains an array of metadata to be used for each of the children
+ *   elements when there's no other metadata for them. This will also imply
+ *   that the type definition's 'list' property for the parent element will be
+ *   set to TRUE.

1.a

nit: this needs a period at the end:
NestedArray::mergeRecursive()

1.b

The best format to use is being discussed in a follow-up #1851498: Polish file format of configuration metadata (for translation). But my question is what is the format used currently. I thought .definition was taken out (for better or worse). If it was taken out, then the comments need to be updated. I had this lingering sense that .definition was recently *optional*, but that it was pointed out there should not be multiple ways. In the end it was removed.

2.

+++ b/core/includes/config.incundefined
@@ -169,6 +171,209 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
+ * For example, for the config file core/modules/system/config/system.site.yml:
+ * @code
+ *   name: Drupal
+ *   mail: ''
+ *   slogan: ''
+ *   page:
+ *     403: ''
+ *     404: ''
+ *     front: user
+ *   admin_compact_mode: '0'
+ * @endcode

admin_compact_mode: '0'

has not always been in the examples.

I wonder if it is new.
The meta files should include it, they do not.

3.

+++ b/core/lib/Drupal/Core/Config/Metadata/ElementBase.phpundefined
@@ -0,0 +1,421 @@
+  public static function buildDefinition($data, array $metadata) {
+    $definition = isset($metadata['.definition']) ? $metadata['.definition'] : array();
...
+  public static function buildMetadata($data, array $metadata) {
+    $metadata = self::processInclude($data, $metadata);
+    // Process shorthand '.type', '.label'
+    foreach (array('type', 'label') as $key) {
+      if (isset($metadata['.' . $key])) {
+        $metadata['.definition'][$key] = $metadata['.' . $key];
+        unset($metadata['.' . $key]);
+      }
+    }
+    return $metadata;
+  }

The idea of a definition is still (correctly) here in the code. But I think the litteral string '.definition' is not being used and so should not be here.

4.

+++ b/core/modules/locale/lib/Drupal/locale/LocaleTypedConfig.phpundefined
@@ -0,0 +1,264 @@
+  /**
+   * Sets translation parameters.
+   *
+   * @param string $langcode
+   *   The language code for the translation, which is a string.
+   * @param \Drupal\locale\LocaleTypedConfig $source
+   *   Configuration wrapper object used as translation source.
+   *
+   * @return \Drupal\locale\StringStorageInterface
+   *   The storage interface object.
+   */
+  public function setTranslation($langcode, LocaleTypedConfig $source) {
+    $this->translationLangcode = $langcode;
+    $this->translationSource = $source;
+    return $this;
+  }

Why does this return something? Seems odd for a set method.

My description of what it returns needs to be checked.

5.

5.a

+++ b/core/modules/locale/lib/Drupal/locale/Tests/LocaleConfigTranslationTest.phpundefined
@@ -0,0 +1,154 @@
+    $lid = (string) $textarea[0]['name'];
+    $edit = array(
+      $lid => $site_name,
...
+    $lid = (string) $textarea[0]['name'];
+    $edit = array(
+      $lid => $image_style_label,

5.b

+++ b/core/modules/locale/locale.bulk.incundefined
@@ -627,3 +689,252 @@ function locale_translate_delete_translation_files($langcode) {
+/**
+ * Gets the configuration names associated with strings.
+ *
+ * @param array $lids
+ *   An array with location string identifiers.
+ *
+ * @return array
+ *   An array of configuration object names.
+ */
+function _locale_config_string_names(array $lids) {
+  $names = array();
+  $locations = locale_storage()->getLocations(array('sid' => $lids, 'type' => 'configuration'));
+  foreach ($locations as $location) {
+    $names[$location->name] = $location->name;
+  }
+  return $names;
+}

5.c

+++ b/core/modules/locale/locale.moduleundefined
@@ -794,6 +808,28 @@ function _locale_refresh_translations($langcodes, $lids) {
+ * Refreshes configuration after string translations have been updated.
+ *
+ * The information that will be refreshed includes:
+ * - JavaScript translations.
+ * - Locale cache.
+ *
+ * @param array $langcodes
+ *   An array of language codes for updated translations.
+ * @param array $lids
+ *   List of string identifiers that have been updated or created, which is an
+ *   array.
+ */
+function _locale_refresh_configuration($langcodes, $lids) {
+  if ($lids && $langcodes) {
+    include_once drupal_get_path('module', 'locale') . '/locale.bulk.inc';
+    if ($names = _locale_config_string_names($lids)) {
+      locale_config_update_multiple($names, $langcodes);
+    }
+  }
+}

I added some detail to the comments, but the accuracy needs to be checked. I'm not sure what $lids are.

Is it the location of a string,
or is the string a locale?
Is it like a file or folder in the system that is the "location" of config, or meta, or just a string, or
is it a place in the world, like a locale?

I suggest adding examples to the comments.

YesCT’s picture

0.

I'm not going to put the whole interdiff in here. But I wanted to clarify some, and point out some things.

In general, @Jose, can you check this closely that the changes I've made to the types, and the comments are accurate?

1.

diff --git a/core/includes/config.inc b/core/includes/config.inc
index 4d04b9a..733ddfc 100644
--- a/core/includes/config.inc
+++ b/core/includes/config.incundefined
@@ -1,5 +1,10 @@
 <?php
 
+/**
+ * @file
+ * This is the API for configuration storage.
+ */
+
 use Drupal\Core\Config\Config;
 use Drupal\Core\Config\FileStorage;
 use Drupal\Core\Config\NullStorage;
@@ -8,11 +13,6 @@
 use Drupal\Core\Config\StorageInterface;
 
 /**
- * @file
- * This is the API for configuration storage.
- */
-
-/**
  * Config import lock name used to prevent concurrent synchronizations.
  */
 const CONFIG_IMPORT_LOCK = 'config_import';

This looks confusing, but I just moved the @file to right after <?php where it is supposed to be.

2.

+++ b/core/includes/config.incundefined
@@ -172,7 +172,7 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
 /**
  * Retrieves metadata for a configuration object or key.
- * 
+ *

This takes out whitespace.
There were other places with whitespace fixes.

3.

+++ b/core/includes/config.incundefined
@@ -172,7 +172,7 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
@@ -202,6 +202,7 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf

@@ -202,6 +202,7 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
  *     403: ''
  *     404: ''
  *     front: user
+ *   admin_compact_mode: '0'
  * @endcode

this is that new line mentioned in the previous comment. point 2.

4.

+++ b/core/includes/config.incundefined
@@ -225,12 +226,11 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
- * unless they have children elements (like 'page' in the example) or the
- * 'list' property set to true, in which case they will default to the type
- * 'config_element'.
+ * unless they have child elements (like 'page' in the example) or the 'list'
+ * property, in which case they will default to the type 'config_element'.

I word wrapped this. I think going forward, I resisted doing that as long as the lines were not too long. I also changed the grammar here a bit from "they have children elements" to "they have child elements". But I'm thinking that I should have left it. So I didn't change any more of that pattern. And I'll re-roll to put it back if people want me to, since that is not critical.

5.

+++ b/core/includes/config.incundefined
@@ -225,12 +226,11 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
  * Nested lists of elements, that contain additional metadata for each of the
- * elements in a different file are also supported.
+ * elements in a different file, are also supported.

this was an added comma for clarity.

6.

+++ b/core/includes/config.incundefined
@@ -253,7 +253,6 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
  *       .type: 'text'
-
  * @endcode

whitespace
There were other whitespace removals. this one was in the middle of a doc block.

7.

+++ b/core/includes/config.incundefined
@@ -253,7 +253,6 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
@@ -303,13 +302,17 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf

@@ -303,13 +302,17 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
  *   effects:
  *     .label: 'Style effects'
  *     .list:
- *       .include: 'image.effect.[name]'
- *       .label: 'Image style effect'
- *       weight:
- *         .label: 'Weight'
- *         .type: integer
- *       ieid:
- *         .label: 'IEID'
+ *       .label: 'Image effect'
+ *      weight:
+ *        .label: 'Weight'
+ *        .type: integer
+ *      ieid:
+ *        .label: 'IEID'
+ *      name:
+ *        .label: 'Machine name'
+ *      data:
+ *        .include: 'image.effect.[%parent.name]'
+ *        .label: 'Data'
  * @endcode

7.a

the comment block was using an out of date example.

7.b

The words here in this block need a very careful going over as it is kind of the main point of the patch and the issue summary. So an important @todo here to review the paragraphs!

8.

+++ b/core/includes/config.incundefined
@@ -330,22 +333,16 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
  * For example here is meta/image.effect.image_scale.yml:
  * @code
- *   .label: 'Image scale'
- *   data:
- *     width:
- *       .label: 'Width'
- *       .type: 'integer'
- *     height:
- *       .label: 'Height'
- *       .type: 'integer'
- *     upscale:
- *       .label: 'Upscale'
- *       .type: 'boolean'
- * @endcode 
...
+ *   width:
+ *    .label: 'Width'
+ *    .type: integer
+ *  height:
+ *    .label: 'Height'
+ *    .type: integer
+ *  upscale:
+ *    .label: 'Upscale'
+ *    .type: boolean
+ * @endcode

updated the comment block to match exactly the actual meta file.

9.

+++ b/core/includes/config.incundefined
@@ -330,22 +333,16 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
- * @see \Drupal\Core\TypedData\TypedDataManager::create()
- * @see \Drupal\Core\Config\Metadata\ElementBase
- * @see \Drupal\Core\Config\Metadata\ListElement
...
@@ -354,6 +351,9 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf

@@ -354,6 +351,9 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
  * @return \Drupal\Core\Config\Metadata\MetadataLookup
  *   A metadata array.
  *
+ * @see \Drupal\Core\TypedData\TypedDataManager::create()
+ * @see \Drupal\Core\Config\Metadata\ElementBase
+ * @see \Drupal\Core\Config\Metadata\ListElement
  * @see \Drupal\Core\Config\Metadata\ElementInterface::getMetadata()

moved the @see's to the end as that is the doc standard and it was mentioned before that that was ok to do. #137, #206 Q2, #208

10.

+++ b/core/includes/config.incundefined
@@ -354,6 +351,9 @@ function config_sync_get_changes(StorageInterface $source_storage, StorageInterf
diff --git a/core/lib/Drupal/Core/Config/Config.php b/core/lib/Drupal/Core/Config/Config.php

diff --git a/core/lib/Drupal/Core/Config/Config.php b/core/lib/Drupal/Core/Config/Config.php
index f07c798..6fc2a87 100644

index f07c798..6fc2a87 100644
--- a/core/lib/Drupal/Core/Config/Config.php

--- a/core/lib/Drupal/Core/Config/Config.php
+++ b/core/lib/Drupal/Core/Config/Config.phpundefined

+++ b/core/lib/Drupal/Core/Config/Config.phpundefined
+++ b/core/lib/Drupal/Core/Config/Config.phpundefined
@@ -323,7 +323,7 @@ public function clear($key) {

@@ -323,7 +323,7 @@ public function clear($key) {
   /**
    * Loads configuration data into this object.
    *
-   * @return \Drupal\Core\Config\Config
+   * @return Drupal\Core\Config\Config

This was the only touch to that file. Should be a follow-up to fix the whole file.

11.

+++ b/core/lib/Drupal/Core/Config/Config.phpundefined
@@ -323,7 +323,7 @@ public function clear($key) {
diff --git a/core/lib/Drupal/Core/Config/ConfigFactory.php b/core/lib/Drupal/Core/Config/ConfigFactory.php

diff --git a/core/lib/Drupal/Core/Config/ConfigFactory.php b/core/lib/Drupal/Core/Config/ConfigFactory.php
index c959673..ca36ce7 100644

index c959673..ca36ce7 100644
--- a/core/lib/Drupal/Core/Config/ConfigFactory.php

--- a/core/lib/Drupal/Core/Config/ConfigFactory.php
+++ b/core/lib/Drupal/Core/Config/ConfigFactory.phpundefined

+++ b/core/lib/Drupal/Core/Config/ConfigFactory.phpundefined
+++ b/core/lib/Drupal/Core/Config/ConfigFactory.phpundefined
@@ -58,7 +58,7 @@ public function __construct(StorageInterface $storage, EventDispatcher $event_di

@@ -58,7 +58,7 @@ public function __construct(StorageInterface $storage, EventDispatcher $event_di
    * @param string $name
    *   The name of the configuration object to construct.
    *
-   * @return \Drupal\Core\Config\Config
+   * @return Drupal\Core\Config\Config
    *   A configuration object with the given $name.
    */
   public function get($name) {

This was the only touch to this different file. Should be another follow-up to fix the whole file.

12.

+++ b/core/lib/Drupal/Core/Config/InstallStorage.phpundefined
@@ -147,15 +147,15 @@ public function getComponentNames($type, $list) {
diff --git a/core/lib/Drupal/Core/Config/Metadata/ConfigElement.php b/core/lib/Drupal/Core/Config/Metadata/ConfigElement.php

diff --git a/core/lib/Drupal/Core/Config/Metadata/ConfigElement.php b/core/lib/Drupal/Core/Config/Metadata/ConfigElement.php
index b3c4479..a5d756e 100644

index b3c4479..a5d756e 100644
--- a/core/lib/Drupal/Core/Config/Metadata/ConfigElement.php

--- a/core/lib/Drupal/Core/Config/Metadata/ConfigElement.php
+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigElement.phpundefined

+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigElement.phpundefined
+++ b/core/lib/Drupal/Core/Config/Metadata/ConfigElement.phpundefined
@@ -2,7 +2,7 @@

@@ -2,7 +2,7 @@
 
 /**
  * @file
- * Definition of Drupal\Core\Config\Metadata\ConfigElement.
+ * Contains \Drupal\Core\Config\Metadata\ConfigElement.
  */
 
 namespace Drupal\Core\Config\Metadata;
@@ -21,7 +21,7 @@

@@ -21,7 +21,7 @@
 class ConfigElement extends ElementBase implements ComplexDataInterface {
 
   /**
-   * Implements Drupal\Core\TypedData\ComplexDataInterface::get().
+   * Implements \Drupal\Core\TypedData\ComplexDataInterface::get().
    */

There were other significant changes to this file, so the \namespace and @file fixes are in here.

13.

+++ b/core/lib/Drupal/Core/Config/Metadata/ElementBase.phpundefined
@@ -238,16 +237,16 @@ public function getIterator() {
-  public static function buildElement($value, $metadata, $key, $parent = NULL) {
+  public static function buildElement($value, array $metadata, $key,  \Drupal\Core\Config\Metadata\ElementInterface $parent = NULL) {

There are other places where I put types in the function parameter list. If I should not have, I'll undo those.

14.

+++ b/core/modules/locale/lib/Drupal/locale/LocaleTypedConfig.phpundefined
@@ -217,13 +220,13 @@ public function translateElement($element, $options) {
diff --git a/core/modules/locale/lib/Drupal/locale/StringStorageInterface.php b/core/modules/locale/lib/Drupal/locale/StringStorageInterface.php

diff --git a/core/modules/locale/lib/Drupal/locale/StringStorageInterface.php b/core/modules/locale/lib/Drupal/locale/StringStorageInterface.php
index 0fb26ad..d02774f 100644

index 0fb26ad..d02774f 100644
--- a/core/modules/locale/lib/Drupal/locale/StringStorageInterface.php

--- a/core/modules/locale/lib/Drupal/locale/StringStorageInterface.php
+++ b/core/modules/locale/lib/Drupal/locale/StringStorageInterface.phpundefined

+++ b/core/modules/locale/lib/Drupal/locale/StringStorageInterface.phpundefined
+++ b/core/modules/locale/lib/Drupal/locale/StringStorageInterface.phpundefined
@@ -79,7 +79,7 @@ public function getLocations(array $conditions = array());

@@ -79,7 +79,7 @@ public function getLocations(array $conditions = array());
    *   (optional) Array with conditions that will be used to filter the strings
    *   returned and may include all of the conditions defined by getStrings().
    *
-   * @return \Drupal\locale\SourceString|null
+   * @return Drupal\locale\SourceString|null

The only changes to this file were \namespace. This should be a follow-up.

yched’s picture

So, to clarify my position on the patch if needed :
- I'm fine with the current state of the file expressiveness regarding includes, and how image styles are handled - thanks again Jose for bearing with me !
- I have not reviewed the code itself, and don't really have the bandwidth right now :-/
- I have no real opinion either on the question of the '.label / .type' vs '.definition' syntax.

Jose Reyero’s picture

@YesCT, thanks for the cleanup, almost everything looks good to me, just a few issues (fixed tests too):
- Fixed indentation for some metadata in comments (elements under '.list' are actually contained in it)
- For LocaleTypedConfig::setTranslation(), the 'return $this' is supossed to be the default for object methods when we don't have other logical return value, return the object itself so methods can be chained: $object->method1()->method2().
- Fixed parameters types for some functions in locale.bulk.inc() + fixed function invocation for _locale_config_data_compare(), which was getting sometimes a 'FALSE' when configuration object was empty.
- $lids are string identifiers (though some functions use the same name for 'location ids', when we have both, this may need some cleanup too but it is unrelated to this issue.

Thanks for creating the follow up issues too.

alexpott’s picture

Status: Needs review » Reviewed & tested by the community

Okay we have agreement on the approach from @yched and @efflugentsia, great diagrams and cat herding from @GaborHojtsy and some great clean up and follow-up issues from @YesCT and I've reviewed this patch several times. This one is good to fly!

Awesome work @reyero

gdd’s picture

Status: Reviewed & tested by the community » Needs review

I've been taking a look at this for the first time since the .include changes went in, and I have to say I really like the new format. I think the files are much more maintainable now, huge thanks to yched and jose for working through that, it was worth the effort. I generally agree with everyone that the remaining issues can be relegated to cleanups. The code has just gotten more understandable as we've moved through the patches. While I have not reviewed the latest patch in detail, I'm personally comfortable with the level of consensus that has been reached by the participants in this issue. If someone wants to RTBC before I get back here, I'm OK with this going in.

gdd’s picture

Status: Needs review » Reviewed & tested by the community

Hey! Look at that!

gdd’s picture

Issue summary: View changes

Updated issue summary added a follow-up section.

YesCT’s picture

all the points from #250 are addressed, or have their own follow-ups created and listed in the issue summary, or are a todo in the Remaining Tasks section in the issue summary.

YesCT’s picture

Still open questions that need answers from #248

(nit-ignore-it) 1.a
(major) 1.b about .definition and 3. about .definition [getting an answer would be highly good. dealing with change if needed could be follow-up.]
(normal-nonblocking) 2. admin_compact_mode: '0' [would be nice to get an answer, not critical]

Everything else from #248 is addressed.

chx’s picture

Please, go ahead and commit and disregard everything I said but most definitely do not credit me on this.

chx’s picture

Issue summary: View changes

Updated issue summary with followups for code standards, added remaining tasks section.

Jose Reyero’s picture

re @YesCT #248

1.a: (code style)
Yes, it needs a period
2. (admin_compact_mode)
Yes, that was added to system.site I don't know when.

1.b. & 3: The best format to use is being discussed in a follow-up #1851498: polish file format of configuration metadata (for translation). But my question is what is the format used currently. I thought .definition was taken out (for better or worse)...

No, definition is still there just in case we need to add additional properties for an element, for full support of the TypedData API.
Only we haven't needed it yet, since all of our current metadata just uses '.type' and '.label' and by using these properties we make the metadata more readable and easier to write.
But we want it also to be extendable and even if we eventually don't need it in Drupal core at all I'd still let it there for contrib modules to use it if they need it.

Gábor Hojtsy’s picture

There is clear disagreement between chx and Jose on what is easier to read/write, but *nobody else* took sides on either side so far (AFAIS). @yched abstained and it was not clear to me if @sun or @effulgentsia clearly weighted for or against. It is apparantely a strong point for both involved so far, so that is why we opened #1851498: Polish file format of configuration metadata (for translation) where this can be further discussed without the scare of 260 existing comments and huge swaths of unrelated code (unrelated to the file's format that is). That should be more contributor friendly. Also once the metadata is in the hands of the wider audience of core developers, lots more feedback will flow in. (We have seen similar happening with annotations for example).

Gábor Hojtsy’s picture

Issue summary: View changes

Updated issue summary.

Jose Reyero’s picture

Issue summary: View changes

Updated for latest patch (added back .type, .label)

YesCT’s picture

Assigned: YesCT » Unassigned

I did not get to the issue summary yet. But @Jose Reyero is doing it now. So unassigning myself.

YesCT’s picture

Issue summary: View changes

Fixing weird formatting for percentage char

YesCT’s picture

I edited the issue summary to have the exact examples from the patch from #252
There are two points in the issue summary that need clarification. But the issue summary is really very accurate at this point.

Someone could verify the steps to test drive are still accurate.

YesCT’s picture

Issue summary: View changes

Updated issue summary to match the files from the patch from 252, clarify grammar, point out 2 things that need clarifying with @jose, and add verify test drive to remaining tasks. -c

Jose Reyero’s picture

Issue summary: View changes

Updated issue summary.

YesCT’s picture

Issue summary: View changes

Updated issue summary corrected variable name in psuedo code for getValue() site name example and added how to get 'Site name', but it's not part of this patch. add See: ?

YesCT’s picture

Issue summary: View changes

Updated issue summary, small text change. changed string to it. so as not to confuse things of type string being not translateable.

YesCT’s picture

Issue summary: View changes

Updated issue summary updated steps to test drive and remaining tasks: all done!

YesCT’s picture

This is totally ready for review. Issue summary is accurate. All remaining tasks are done.
(of course this is not the end. but it is a milestone.)

webchick’s picture

Holy cow. I really want to thank you for an incredibly detailed issue summary. That is going to be a HUGE help in orientation.

Here's the scoop.

chx and I live in the same town, and we're going to get together this weekend and this is one of the patches we plan to go over. So hopefully on Saturday night I should be able to report back with a list of things to fix and/or a green patch. :)

Of course, if Dries/catch want to wade in here before me, PLEASE feel free. :)

Jose Reyero’s picture

@webchick,
Just trying to save your time for more valuable work, like actually committing stuff ;-)

About the plan, it sounds good to me, I'm sure @chx will be able to fill in the details of anything we've missed in the issue summary. My only objection is to his comment in #258, he definitely should get credited on this one if this ever gets committed.. :-)

Unfortunately I may not be around too much after Dec 1st (Yes, I know, timespan extended, etc.. I just have other important things to do) so I want to state beforehand, if it's not clear enough from the above 250+ comments, that I don't have any special preference about the metadata format itself, so I'd be ok with any change to it
(as long as it works for complex plugins like views and other people find it readable/writable for human beings, which is my main concern about it)

webchick’s picture

Yep, that's fine. Take a well-deserved break. :) Hopefully whatever changes there are will be fairly minor/easy, since this issue looks to have pretty broad buy-in.

Gábor Hojtsy’s picture

Issue summary: View changes

Updated issue summary. nit fixed a html /ul to ul

yched’s picture

Adjusted the issue summary
- Removed "* So far we don't have such examples in Drupal core though components / plugins are meant to be reusable (so metadata for them too)." - we do :-)
- Added an example about field formatters instead.

yched’s picture

Issue summary: View changes

Further explanation about includes and separation of concerns.

gdd’s picture

When/if this lands, we will need a followup issue to create the metadata files for all existing config, and we should probably add it is a requirement for all patches in the queue.

Jose Reyero’s picture

When/if this lands, we will need a followup issue to create the metadata files for all existing config,

Yes. However that is easy for most config and the hardest one which is views is almost done in the config_demo module.
We still need to figure out something better about where to place plugin metadata though.

and we should probably add it is a requirement for all patches in the queue.

I think for now maybe only the ones having translatable strings, which is what we are using it for.

Not having metadata for a while for a config file or a part of it shouldn't break anything as we've got reasonable defaults for everything (strings, arrays).

cosmicdreams’s picture

and I'm fully prepared to rework the recently landed regional CMI patch to follow the lead of this issue. Let's push this forward.

webchick’s picture

Assigned: Unassigned » Dries

Ok, as promised, chx walked me through the patch tonight (thanks, chx!). I have a review I'm working on, but it's now 1am and I'm leaving to get on a plane in 4 hours, so this will have to wait until tomorrow night, most likely.

The gist:

- I found several things that made me go "hmmmm", but I don't think any of these are worth holding up this patch. However...
- This is a big enough DX change that my "spidey sense" says that Dries is going to want to sign-off on this. Therefore, assigning to him. I believe the soonest he can look at it will be Wednesday though. :\ Since the plane ride I'm going on is to attend meetings all day Monday and Tuesday with Dries, I'll see if I can give him the scoop sooner than that.

Thanks again for your patience, folks. I know you've all worked really hard on this. But there's a lot here to take in, and we want to make sure we do this right.

pounard’s picture

I always thought that XML would have been a better choice than YAML at the very beginning[EDIT: As @chx pointed out, I meant: long before YAML was choosen] because you can write your own doctype and include all this meta information directly in a single, complete and validable file which the module would provide to declare its variables: all of typing, name and description declaration would be natural in such files. The solution proposed in this issue is hackish usage of YAML for which it has not been designed for. It forces the developers to write two files cleanly register their variables, where it could be only one; This is neither natural nor validable. It forces the core config system to parse 2 files instead of one. It forces to write more documentation for making this understandable. This issue shows the lack of anticipation which happened at the very beginning which takes form today as a very ugly workaround. The only clean solution is to change the choosen serialization format. I know this is probably too late now, but I think we are shooting ourselves in the foot with the actual proposed solution. The choice being made today is not rational.

EDIT: This post is troll-esque, I'm not blaming anyone here, I'd like to see a more consistent response to this problem. I'm not sure if YAML will ever allow us to provide one.

catch’s picture

@pounard: a big reason for splitting this up was to avoid having to load all the metadata just to get a single configuration value out of a file - i.e. it should only be necessary to translate something if it's displayed - either on a form, or a configuration value displayed to visitors like the site name or field labels.

I just caught up on the comments here but haven't reviewed the latest patch yet, so don't know if that's the case still.

Not sure if I'll get to the patch before Dries, I'll try to do a proper review if I do though.

Jose Reyero’s picture

@webchick,

I found several things that made me go "hmmmm", but I don't think any of these are worth holding up this patch.

Well, anyway knowing which are your concerns, will help in case this needs one more rerolll...

This is a big enough DX change that my "spidey sense" says that Dries is going to want to sign-off on this.

Yes, makes sense, actually I think this should be (have been) the way for any DX change, I mean more thorough review...

@pounard,
Though I agree XML would have been a better option, for which it may be too late, I don't see any other constructive suggestion in you comment about how to addres this problem. Do you mean "this is the only option we've got atm, as ugly as it can be"?
Really, what we are doing here has some drawbacks (needing to write another yaml file) but also has advantages (just needing to write another yaml file) as oppossed to define and document and write a ton of info hooks for modules.

pounard’s picture

@catch

a big reason for splitting this up was to avoid having to load all the metadata just to get a single configuration value out of a file - i.e. it should only be necessary to translate something if it's displayed - either on a form, or a configuration value displayed to visitors like the site name or field labels.

This is not the problem, module provided YAML files only ships a definition. Parsing files is done only when necessary (no cache synchronized, potential manual resync asked, or module install) while working values are stored into database/cache.

@Jose Reyero

Though I agree XML would have been a better option, for which it may be too late, I don't see any other constructive suggestion in you comment about how to addres this problem. Do you mean "this is the only option we've got atm, as ugly as it can be"?

My suggestion was at the time , before the very first line of the config API was written[EDIT: Ok, as @chx pointed out, I didn't said that out loud, I nevertheless pointed out systems such as gconf; Anyway this is still true I raised this when XML was being questionned] that we could write a simple DTD for XML, including data type, name and description attributes in the definition itself. And that we could read this definition from originating module XML files when needed (administration screens) without taxing the normal runtime (reading from cleaned up files, potentially, or from a custom key/value external source or database schema) allowing to write fully dynamic administration screens for modifying config.

--

Anyway, XML is not the question at all, JSON has been abandonned once for similar reasons (no comments possible, the need of rewriting custom formatters for having pretty result, no possibility of putting metada except by using a key name convention) and we are using YAML and re-introducing all these stuff for which we abandonned other formats.

In the end, it's sad for performances too, because today libxml2 is very very fast for both read and write operations and uses native code on all systems, while our YAML parser is written as user space PHP and may be slower than libxml2 (although I didn't benched so I cannot be sure). And we are gonna make it parse twice the file count.

chx’s picture

We are not rehashing #1470824: XML encoder can only handle a small subset of PHP arrays, so switch to YAML here, if you want to do that please open a separate issue. Posts on that topic following this comment will be deleted simply.

catch’s picture

This is not the problem, module provided YAML files only ships a definition. Parsing files is done only when necessary (no cache synchronized, potential manual resync asked, or module install) while working values are stored into database/cache.

The difference is whether you store all the metadata in that cache too, or only get it when needed. It'd be possible to store it in the same file, parse it, but then store it separately but that's not really been proposed.

Also, there's absolutely no need for the metadata to be copied across to the actual storage in terms of the YAML files - it's metadata, and it doesn't have a one-to-one relationship with each configuration file - that's true for configuration entities, but it's also true for say a multi-site where you have one module in sites/all/modules but plenty of CMI files in the file storage. If metadata is updated, you don't want to be having to update the actual configuration values either, and having everything in the same file would mean two file formats - one with metadata provided by modules, and one in the configuration storage without.

pounard’s picture

Also, there's absolutely no need for the metadata to be copied across to the actual storage in terms of the YAML files

but it's also true for say a multi-site where you have one module in sites/all/modules but plenty of CMI files in the file storage. If metadata is updated, you don't want to be having to update the actual configuration values either

Indeed, but what I said never meant this. Updating metadata is just updating the definition wherever you stored it (or not if you didn't stored it in an intermediate storage): you never need to alter the stored configuration except if it conflicts with the changed definition (type mismatch for example).

and having everything in the same file would mean two file formats - one with metadata provided by modules, and one in the configuration storage without

This isn't true, consider your metadata being optional properties in your format: you can generate raw files without for working storage and keep originating one in modules' dir for being able to read metadata, but still be manipulating the same file format in both cases in the end. You don't need two parsers nor two readers, you just need to put an option somewhere that says "with or without metadata".

--
I really think the dual file (one for raw values and the other for metadata) is a huge DX mess and hackish workaround.

Nevertheless, as chx said, this discussion has past a long time ago. So I will let you do this in peace. As I said upper, this was primarily trollesque because I already fought for that a long time ago saying you would encounter this need. I just wanted to tell you that right now you are encountering it: good for you.

penyaskito’s picture

Status: Reviewed & tested by the community » Needs work

weight_select_max was added in core, so the assertion should be for 6 elements and not 5.

penyaskito’s picture

Status: Needs work » Needs review
FileSize
89.82 KB
1.08 KB

Added patch. No credit please.

Gábor Hojtsy’s picture

Status: Needs review » Reviewed & tested by the community

Very simple update, thanks!

chx’s picture

@pounard: "I already fought for that a long time ago saying you would encounter this need" I left you a message on IRC to stop setting yourself up as the oppressed hero but you didn't stop so let's see the facts: in the beginning you preferred INI format and hesitantly pro-XML at best and haven't said a word about "this need". Also your claim "my suggestion was at the time, before the very first line of the config API was written, that we could write a simple DTD for XML" is patently false because the first line of CMI happened to be written in 2011 at a sprint in Denver I participated in so I have first hand knowledge of the events and you never mentioned DTD before this March and only after nod mentioned it.

pounard’s picture

EDIT: Google is easy for targetting exactly what you want, nevertheless it's not an history book. Oh... I just don't want to answer that. Please don't explode.

Jose Reyero’s picture

I think it may be about time of doing some recap of this issue and the 280+ follow ups, before we get over 300 :-)

1. This patch adds metadata for configuration. I think there's some wide consensus about the need for it or at least the convenience of having it, and the contents of it (types, labels). This is what the first hundred comments are about.
2. We are also addressing an actual issue that is translating the default strings in configuration, which as a result of the new configuration system have moved from t-wrapped strings in the code to plain strings in the configuration files.
(We could call it a regression if we ever had them properly translated in previous versions of Drupal, which unfortunately is not the case)
3. Then there's the metadata format for which, though there's some important consensus too, it seems not everybody is really happy about (though we don't have much better ideas for now or at least not without full reworking the config system), and also there's the concern of it being some important DX change.
4. There are these two follow up issues to improve the format so it gets easier / more readable, also counting on broader community feedback once metadata pops up in Drupal code, and find a better placement for plugin's metadata.
#1851498: Polish file format of configuration metadata (for translation)
#1851502: Better placement of meta files for configuration metadata (for translation)

About moving forward, IMO:
A. The most important thing here would be to know whether we can count on having such metadata available, so we can do fancy things with it like translating stuff, browsing / validating configuration data or autogenerating settings forms (though it looks like for these last ones it's already a bit "late" to think they may be in core for next Drupal).
B. From the localization system perspective it would be important too (if we get this) to get it as soon as possible because it will have huge implications on the whole site translation workflow (install, update, deployment, administration).
(C) (Or if we don't, to find a viable option to fix configuration translation though you can take for granted the DX for it is not going to look very nice either since we'll need every module to handle translatability for its own configuration strings.)
D. Then about the metadata format itself, I won't say I don't care how it looks finally, as it is some important DX issue, but however it looks like, it won't have a major impact on what we can use it for. (And that's why this is the last concern in the list, though I understand it may be the most important one in order to get this thing committed or not.)

Dries’s picture

I started looking into this, but need a bit more time. I'll spend more time on this this week. Please stay tuned.

Gábor Hojtsy’s picture

The fail was unrelated. Back to RTBC.

chx’s picture

> This patch adds metadata for configuration. I think there's some wide consensus about the need for it or at least the convenience of having it, and the contents of it (types, labels). This is what the first hundred comments are about.

Well, some metadata yes, but this implementation? I am not sure.

> We are also addressing an actual issue that is translating the default strings in configuration, which as a result of the new configuration system have moved from t-wrapped strings in the code to plain strings in the configuration files.
(We could call it a regression if we ever had them properly translated in previous versions of Drupal, which unfortunately is not the case)

So to put it short, it's not a regression. It's a new feature.

> Then there's the metadata format for which, though there's some important consensus too, it seems not everybody is really happy about (though we don't have much better ideas for now or at least not without full reworking the config system), and also there's the concern of it being some important DX change.

What's here is not consensus but teeth gritting admittance that we do not have a better idea and noone had the time to come up with something better. However, that was the case with the December 1 deadline. With feature squeeze in, I am less and less convinced every time I look at it.

> The most important thing here would be to know whether we can count on having such metadata available, so we can do fancy things with it like translating stuff, browsing / validating configuration data or autogenerating settings forms (though it looks like for these last ones it's already a bit "late" to think they may be in core for next Drupal).

Some metadata yes, but I am more and more worried about how we do it. Like, why is it not in code if only developers need to do this?

> From the localization system perspective it would be important too (if we get this) to get it as soon as possible because it will have huge implications on the whole site translation workflow (install, update, deployment, administration).

Yes but given everyone needs to work with this eventually, we would better have something good in place. Even with the followups the form API like interleaving of two kinds of properties in a YAML file has a certain foreboding and not a positive one.

> Then about the metadata format itself, I won't say I don't care how it looks finally, as it is some important DX issue, but however it looks like, it won't have a major impact on what we can use it for.

Well I am very, very concerned about the format because that's what every. single. Drupal. developer will need to use for 5-6 years after we release this :/

I am not setting this to CNW , but I do not know why.

Anonymous’s picture

i've discussed this patch with catch and chx over the last few days. i don't like that we're introducing metadata for all the things instead of just a narrowly focussed feature based on locale's needs.

i'd really like us to consider something that simply ships files identifying the config paths (config path == some.config.file:some.config.key) that are targets for translation, with a label, and that's it. there may be more, but that should be driven solely by the needs of the locale system, not by a metadata system for all the things that allows modules to do all the things with it.

while talking to chx about this today, he came up with some ideas about how we might implement this and created #1861640: Provide config metadata solely for translation.

i hope it's not too late for us to take one step back here and consider a simpler, more targeted implementation that meets locale's needs. i've posted something similar to #1763640-59: Introduce config context to make original config and different overrides accessible.

Gábor Hojtsy’s picture

Some metadata yes, but I am more and more worried about how we do it. Like, why is it not in code if only developers need to do this?

Once again it could be in code IF it is statically parseable without actually running the code *and* can be related to the "exported" .yml data itself, so a static code/data analyser that is used on localize.drupal.org can extract all the translatable pieces. Given these constraints, even if it is embedded in PHP code for example, it would need to follow very strict structural rules to be readily parseable (eg. strictly no call-out to API functions or plugin methods), so it would equally need to be declarative, just encoded in an executable-looking format.

Jose Reyero’s picture

Rerolled for changes into the locale bulk / update api in #1804688: Download and import interface translations
The only bits changed are locale backend api integration and 1 test update for a new property in system.site.

@chx #295,

What's here is not consensus but teeth gritting admittance that we do not have a better idea and noone had the time to come up with something better. However, that was the case with the December 1 deadline. With feature squeeze in, I am less and less convinced every time I look at it.

Well, that's what happens with deadlines. However this was not a last minute thing and I don't know whether we are producing anything much better in another 2 or 3 months.
(And certainly not me, as I had scheduled enough of my time around the previous deadline, but other people may have the time for it...)

Anyway, we have that follow up issues to make the format better, but no one is likely to invest much time on them unless we are sure we really have some metadata to improve.

@beejeebus,

... instead of just a narrowly focussed feature based on locale's needs.

Because if we do so, any contrib module trying to implement any extended translation with some decent UI would need to completely ditch this and add it's own hook / metadata / whatever and then we'll be back at square 1, or Drupal 7.
The problem with narrowly focused features is that they are just that, "narrowly focused features", and that is not how you get great frameworks to build upon, but more often just more lines of code.

sun’s picture

There are other configuration system issues for which metadata might be the only viable solution.

One of them is #1653026: [META] Use properly typed values in module configuration — which would essentially mean that we'd leverage the metadata for every Config::save() operation; not only to switch from the enforced casting to strings to an enforced type-casting to the expected/defined data type, but potentially also to add a [pre-]storage-level validation of values. (Configuration is rarely written, so this is feasible.)

I additionally want to point out that the combined purpose of #1648930: Introduce configuration schema and use for translation and #1763640: Introduce config context to make original config and different overrides accessible is a base building block for multilingual (!= translatable) configuration; i.e., allowing more than translatable strings to diverge per language.

Anonymous’s picture

re #299 - data types: well, then lets use XML. when we chose yml, the CMI team was explicitly rejecting schema for our config. if we're going to do this, then lets use the right tools for the job (xml), and not invent our own more complex, less powerful schema handling. just to be clear, that's not my preferred option. however, i'd prefer that to yml plus our own, unique to Drupal, less powerful than xml-schema thing.

re #299 - multilingual (!= translatable) configuration: how does that relate to schema for our config? (not a rhetorical question.) other than having different values for config keys based on language, what does that mean, and how does it require a schema for our config data?

Gábor Hojtsy’s picture

@beejeebus: re multilingual != translatable, the subtle differentiation we make there is where most of the discussion on #1851498: Polish file format of configuration metadata (for translation) happens. We consider translatable are the things that are *translations* of each other. Such as how Montag is a German translation of Monday. Multilingual is where you use different values per language but they are not translations of each other, such as when you use a single "work in progress" node as a landing page for your Japanese site but a full blown panel with numerous panes as your French front page. These two are not translations of each other, just different things used for different languages. Both of them are the setting value for your default front page path but apply to different languages.

The translatables we need to identify first and foremost through the .yml files because they live in "exported configuration" that works as part of the software. Eg. Views and user notification email subjects and bodies shipped with Drupal core should be translatable as part of the software on localize.drupal.org. That gives us the requirement to have a format that lets us statically parse the .yml file for all the translatable pieces. This has no need for types or labels, it is just binary information about cherry-picked pieces of config.

Then to support the multilingual use case (and more importantly to be able to generate translation forms for anything that is not shipped configuration, such as a view you create on your site), we need to reproduce all the pieces of the configuration that could be multilingual in a sensible settings form. Eg. if you look at the account settings screen on your site, you want to translate the Anonymous user name and all the email subjects and bodies. BUT you don't want to let translators change the admin role or change signup security settings for your site. That would be disastrous. So we need to be able to build a highly simplified version of the form that does not take its data from the original config, does not save to the original config, does not fire the regular hooks for the forms, and so on. In essence we need some kind of representation as to how a translation form would look like for this data (view, account settings, etc). That needs type info, labels, etc. This is the most contentious part. For example, here @catch argued we should support the full force of Form API and @chx argues we should support as little as possible, while others suggest we should use XML (Schema), although it is not clear how would XML Schema provide us with features such as types, labels (descriptions?) that we can map to Form API forms.

Damien Tournoud’s picture

Title: Introduce configuration metadata and use for translation » Introduce configuration schema and use for translation
Status: Reviewed & tested by the community » Needs review

My point all the way back in #13 is still valid. We are talking about introducing a schema for configuration.

The format itself seems limiting and inconsistent. Some examples:

  • It doesn't support marking properties as required
  • It doesn't support arrays/list
  • It calls "list" what is actually a "map"
  • It relies on magic prefixes ("."), which are a pile of pain (see the Form API use of #)
  • It doesn't seem to support complex data types properly (the handling of that seems at odd with the rest of the Typed Data API)

Also, the schema is currently defined in two places (at least for configuration entities):

  • In the schema
  • In the entity class

Fago #117 was never properly replied to. And really, it all boils down to: why would we re-write the Typed Data API instead of just using it? We already have Typed Data information for configurable entities, we really just need to add a way to declare the same for other CMI data, and there is really no need to do that in YAML, we can and should do that directly in PHP.

Gábor Hojtsy’s picture

@Damien Tournoud: how do you imagine that being defined directly in PHP and still processable statically (without actually running the code). Yes, we can put that in PHP if all coders of that PHP code follow strict guidelines, so we can parse said PHP code statically without running it, which means it needs to be in fact a declarative description, which is better done in a declarative format to aid that poor developer AFAIS.

Gábor Hojtsy’s picture

Status: Needs review » Reviewed & tested by the community

Based on an IRC discussion with @DamZ, his proposal is we adopt the typedData properties/structure as-is (as in http://api.drupal.org/api/drupal/core%21modules%21system%21tests%21modul...) to describe config structures (eg. types would be like 'string_field' and 'integer_field' and 'translatable' would be its own property). The plugin based inclusion system would still be needed that is introduced in the patch and DamZ understood we need this in a statically processable format (so we can actually extract translatable strings without running the code), so it all ends up in a YML file and would need the separations and inclusions introduced as well as the file naming, locale mapping and other subsystems introduced by this patch.

Given that, his concern seems to be tightly related to reworking the file format with yet another proposal that is not yet brought up in other followups. So his concerns would be resolvable fully in #1851498: Polish file format of configuration metadata (for translation).

Moving back to RTBC with that.

Jose Reyero’s picture

Status: Reviewed & tested by the community » Needs work

Congratulations! We got the issue paged on the issue tracker, which makes it still harder to follow :-(

I can understand people not liking the format but the last ones.... So first we trim it down hardly (we don't need forms, validation, etc, bla, bla) to let it at the very minimum we need for translation, and now the complain is it doesn't provide enough information, it's not a real schema, etc.. ?

This is what I think we can do:
- Drop this one. I won't cry, I promise.
- Add real schema data: http://www.kuwata-lab.com/kwalify/ruby/users-guide.01.html#schema
(That would be a minimum api patch)
- Add in next steps, small patches: validation, form generation, translation, plugin support, etc, etc..

This is what I can do with Kwalify:
Content of contact.category.%.schema.yml

type: map
class: ContactCategory
mapping:
  "id":
    type: str
  "label":
    type: str
  "recipients":
    type: seq
    sequence:
      - type: str
  "reply":
    type: str
  "weight":
    type: int

Generated class:

kwalify -a genclass-php -tf contact.category.%.schema.yml > ContactCategory.php

/// 
class ContactCategory {
    var $id;         // str
    var $label;      // str
    var $recipients; // seq
    var $reply;      // str
    var $weight;     // int
    function __construct($arr=NULL) {
        if (! $arr) return;
        $this->id         = @$arr['id'];
        $this->label      = @$arr['label'];
        $this->recipients = @$arr['recipients'];
        $this->reply      = @$arr['reply'];
        $this->weight     = @$arr['weight'];
    }
}

Now for anyone concerned, the DX for this is harder to learn, but at least we would be having some real schema for yaml, that can be used with external tools (parsers, validators, class generation for a few languages not only PHP, etc...)

We don't even need a patch to get started, just a simple statement and documentation about where it should be placed: "We use Kwalify schema definitions for configuration files and they need to be placed in ... "

Kwalify exists as a debian package btw.

Gábor Hojtsy’s picture

Erm, how would that make it possible to generate forms? Damien pointed out above that we szould mirror the TypedData available to entities in general, so having a separate type/schema system here was a major drawback in terms of DX. He indicates building the format pure and simple on TypedData would be better DX. I am sure we keep could keep discussing this for as long as we want, months on end.

Gábor Hojtsy’s picture

Issue tags: +Avoid commit conflicts

Adding tag since this keeps not being applyable all the time...

Dries’s picture

I spent some more time on this patch. It's really hard and have been breaking my head on it for a while. After having spent 3+ hours on it, I still don't feel that I fully grok this approach. As a result, I'm struggling to provide good direction for the time being.

Here is my current stream of thoughts: I understand that this is important but I'm nervous about introducing a new custom file format that is pretty hard to grok. I still struggle with the fact that it only marginally improves the translator experience and that it is still an incomplete solution. I'm also not interested in having core automatically generate simple forms from the meta files; I don't want to introduce another form API, at least not in Drupal 8. It's too late for that.

It feels like we should have some time to continue to figure this out. I encourage people to explore simpler solution similar to what chx is working on (just fixing the translation regression) as well as effulgentsia (simplified file format). I think both are interesting evolvements that give us a point of comparison.

I'm not giving up on this issue. I'll continue to study this issue and will think about it more. In the mean time, I encourage people to continue to work on this issue and the other issues listed above.

Jose Reyero’s picture

@Gábor #309,
Config entities don't have any type schema at all they're just adhoc php code. So they don't use the TypedData API AFAIK.
Field types in the TypedData API are multidimensional arrays of things. They're not reusable at all for anything that are not our multlingual-multi-everything-things entities.
Overall the TypedData API has caused more trouble than benefit -I mean for the purposes of this patch-. And if anyone pretends to use Entities for all this I'd dare them to write up the code for any simplest case that has a translatable string. (Which just means I am clueless about it, I don't buy the "entities are good for everything" and I'm not going to understand the explaination either, so please just show me the code)

Oh, about how to generate forms from schema data (=data type + validation rules), if you add a label to it, I think that's pretty trivial, see initial versions of this very same patch :-)

Jose Reyero’s picture

Status: Needs work » Postponed

@Dries,
Thanks for reviewing this one.

Now, not replying to anyone, just some thoughts on how to move on.

I'm not sure anyone will want to continue with this one at least without some important buy-in on the idea of adding metadata, certainly not me. So I'm marking this as potponed at least until we see what happens with the other patches trying to address similar issues.

Or if ayone wants to reroll it, just in case, there are things that could be better / simpler / cleaner:
- The TypedData API has been more trouble than benefit, we really don't need it here, I'd drop it if I ever do a reroll (which doesn't mean we cannot use the type names like 'string' or 'integer' from TypedData, just trying to map the whole thing, as config_elements are typed data themselves, has been exhausting)
- Bundling the translation feature into this only made sense with the timing of the feature freeze, so for any future move my advice would be splitting it in two patches.
- Forget about plugins for now, since modules using plugins don't seem to have any clean pattern for adding plugin data into configuration yet, nor figuring out dependencies, so we should wait at least until this exists. (Or at least wait for node types to be converted to configuration to see how they look like)

Gábor Hojtsy’s picture

I'm also not interested in having core automatically generate simple forms from the meta files; I don't want to introduce another form API, at least not in Drupal 8. It's too late for that.

All right, we have been working on this issue since the summer (and had g.d.o discussion pieces and drupalcon/camp meetings of the underlying ideas since 2011 summer), but have indeed had a hard time to get reviewers and people interested to support it. Looks like the form generation task will be fully on contrib then. With the form generation use case discarded though, I don't think D8MI has any reason to push for a metadata of any richness (eg. labels, types), because we will need to do it all over in contrib (either by duplicating Form API structures as-is or introducing some kind of metadata). Then contrib modules will either choose to not be multilingual with their config or support that contrib system. (In short, if this is an ugly system but we don't have better ideas, contribs will need to cope with this either way. If we have better ideas then we'll be able to experiment in contrib. The goal of making Drupal 8 more multilingual for configuration is hardly achieved.)

So the core use case is back down to the "simple" problem of identifying strings for localize.drupal.org, so "exported configuration" as part of the software can be translated and translations can be applied to installations of Drupal and its distributions. A huge chunk of this patch deals with that (importing those strings to the locale system, exporting translations from the locale system to .yml files, etc), so that will need to be carried over to whatever format patch we have for identifying those strings. Also, if we keep the assumption that we need to identify those strings statically, then we also need the inclusion and plugin support in the metadata format. So in fact all we loose if the type and label from the metafile (as thus the TypedData connection altogether), the rest of the complexity for describing nested structures with plugins to identify their pieces to extract is needed. So we will need to salvage that piece of this code as well.

I think the conclusion is that core will be simpler, contrib (wanting to support config translation) will be much more complex but the UX will benefit from it. Contrib wanting to implement multilingual for config was not an easy ride in Drupal 7 either, we'll not have a huge step forward.

Gábor Hojtsy’s picture

Untagging with things not relevant anymore.

plach’s picture

Disclaimer: this is no personal attack to anyone, just my thoughts in this moment.

I think D8 cannot ship without a solid API to support multilingual configuration. It's a hard task but this is the time to achieve it. Jose and Gabor have done a tremendous work here, which is going to bring us many advantages besides what is strictly required by the D8MI.

A scenario where we provide a similar API in contrib is simply desolating: we will end-up like with the Variable API which, despite being a sensible solution to many problems we've been having for years now, is leveraged by few projects besides i18n.

Moreover, I'd be tempted to say that if we fail to provide a system allowing us to support multilingual configuration, the D8MI cannot be considered successful. And we should stop considering this a minor problem: our usage statistics tell us that i18n in installed on more or less the 10% of the Drupal installations out there, and this is not counting very simple multilingual websites not needing it. In addition we should consider that doing multilingual websites in Drupal is still a very hard task, which is potentially steering away many potential Drupal adopters. With a full-fledged i18n system in core and a cohesive API allowing contrib to implement i18n easily (DX is crucial here), we will see the above percentage raise upper and upper. Let's stop thinking about multilingual sites as a minority that can be safely ignored, please: at the BadCamp keynote @Dries defined i18n a key factor for Dupal's success in Europe. Key factor.

I'm not diving into implementations details, I didn't follow this issue closely enough to say something relevant, besides the fact that the raised DX concerns seem to be addressable to my eyes. I just want to point out that, from a project management perspective, rescoping this issue to just being able to parse translatable strings in configuration is pure madness.

I can be biased but IMO it's time to put an end to this: we cannot ship with yet another version of Drupal which can do amazing things by just installing modules and clicking through their UIs and, as soon you have to add a language, you have to revisit most of your site-building process, apply tens of patches, write lots of LOC and hope for the best.

That said, I think we should close this issue and open a new one, copy the issue summary and update it with any relevant change needed, because starting again with this after 300 comments will just not work.

plach’s picture

sigh

Gábor Hojtsy’s picture

@plach: well, the reality is that multilingual needs various things:

1. it needs a collection of all translatable things to be exported on localize.drupal.org
2. it needs a collection of all things translatable on a site (includes config that was not shipped with the software), so translation status can be calculated, translation can be QA-ed, etc., this is wider than 1.
3. it need translation UIs that make sense to each piece of configuration and is not contextual only (ie. serious sites will not allow people to go edit a view or edit account settings, so they find a translation tab there, we need to be able to generate translation experiences that are separate from the original forms)

This patch wanted to attack all three and it did an admittedly sub-par job at 3 with the intention to experiment with it and *expand* on it. Now reviewers actually found it already very extensive and dangerous for DX, so no possibility in sight to even expand on it later. (If you go to your site account settings or go edit a view's header text, you'll see lots of supporting UX, descriptions, input formats, tokens, etc. that this patch did not support but an *ideal* UI would). So Dries is right this patch does not further the cause in terms of actually providing a better UX. This was not supposed to be a be all and end all, it was supposed to be a patch to set up the base system which we can experiment and work with. Based on reviews, people don't want to do that :)

I don't think we can skip solving (1) for core in Drupal 8, given that most useful things will be in CMI and if Drupal 8 translators cannot translate shipped content type names, view titles, field labels, and so on, that is going to be a huge problem. Then you could try installing in a foreign language but never be successful with it :) Solving 2 and 3 in core looks like requires such complexity that whoever came in to review or think about it got their "head hurt" and neither did they support the solutions proposed or came up with better ones that others would have supported. The matter of fact is the problem does not go away, it is that it will need to be solved in contrib, and as you rightly pointed out, only those contribs who care about it will work with it (and will likely have a harder time doing it). There is only so much one can do in one core release.

It is also not like problems for CMI/multilingual are at an end in Drupal 8, we still nee to solve (1) and make config accessible in different languages: #1861640: Provide config metadata solely for translation and #1763640: Introduce config context to make original config and different overrides accessible. And then we have problems with the existing override system in #1859110: Circular dependencies and hidden bugs with configuration overrides. Those issues lack people collaborating as well :) Contributions welcome.

plach’s picture

I perfectly understand the scope and context of this patch, thanks, I read though all of its comments, albeit I didn't have the time to look at the code. I realize it's a really hard task, probably far more complex than the ones we have to face while working on the entity stuff: just the fact that we are dealing with arbitrarily complex data-structures instead of predictable ones makes me want to cry.

And I understand the use-cases we want to support here, but it seems to me that by rescoping this we are losing a formidable opportunity to solve in core a number of problems that in contrib will simply explode. And none will have the brainpower, the dedication or simply the time to fix them all, then.

This is why I'm talking about PM and not about solutions: we need to change our mindset here, IMHO this really should be a critical release blocker. No deadlines. We just need to do this. Personally I've not been able to help here because I've never been particularly involved with the CMI from the very beginning. I've tons of stuff that have a deadline to work on. But this one IMO is one of the most important changes in D8, and the fact that very few people actually worked on it is what makes me think we need a change in our mindset. If my help can be useful in any way, I'll just complete just the stuff that strictly needs to be done before feature completion on the entity side and then focus on this.

I'd be totally ok with having a UI providing both multilingual config forms and stripped-down translation forms in contrib (I have some experience with this approach ;). This is exactly what we are trying to achieve in #1807776: Support both simple and editorial workflows for translating entities, and having core support the same for config is a unique chance to unify or make more consistent at least our patterns if not our APIs. But the plumbing must be in place and it has to be complete.

Heck, we need this metadata stuff :)

effulgentsia’s picture

@plach: per my comment in #1861640-14: Provide config metadata solely for translation, I agree that extensible metadata, and not just translatable string identification, would be really awesome plumbing to have, and a sad missed opportunity to not have. That's why I'm happy this issue is still prioritized as critical, even if it's also postponed for the time being. I disagree about it being release blocking: it is a feature request after all, so IMO, should be subject to the Feb. 18th feature completion deadline. I think we can still solve it by then.

I think some more tightly scoped exploration in #1861640: Provide config metadata solely for translation is a good thing. I'm curious to see what patches show up there.

I continue to be interested in pursuing this full issue though. I agree with most of Jose's suggestions in #312 and his reason for postponing this issue, along with your suggestion in #315 to start new issues for doing the things Jose recommends in smaller chunks.

However, I disagree with Jose's claim that the TypedData integration should be abandoned. I'm currently delving into that to see if I can come up with a patch that demonstrates my understanding and reasonings, and maybe that will help me convince others why TypedData integration is worth continuing, or will help others convince me why it isn't. It'll take a little time for me to do that though, so if others in the meantime want to pursue some other ideas, please do. Some parallel explorations would not be a bad thing here: it'll get more people understanding the problem space from different perspectives, and then maybe we can converge on something that works for all those perspectives.

And #317 lists other very important related issues too, so there's no shortage of things that people can help on right now without feeling like everything is stuck on this issue.

fago’s picture

Coming back to this issue after a while.

I'm also not interested in having core automatically generate simple forms from the meta files; I don't want to introduce another form API, at least not in Drupal 8. It's too late for that.

I'm not sure it would be the right solution UX wise, but I would not agree this re-invents form API. Field API widgets do not re-invent form API either, it's a layer on top of it.

But given the issue summary states multiple good reasons for having the metadata, this does not seem to be the real problem here. I guess it's more the new file format:

I understand that this is important but I'm nervous about introducing a new custom file format that is pretty hard to grok. I still struggle with the fact that it only marginally improves the translator experience and that it is still an incomplete solution.

I'm not sure why we went away from a straight php2yaml typed data defintion + an easy way to nest things to mixing .type and .definition. Maybe, we could re-consider a slightly simplified approach that takes typed data definitions from php really 1:1 to yaml? That would mean it would be more work to integrated nested structures, as you'd have to expose a separate "type" for each level. But it would work and once we've more experience with nested structures we know how it could be simplified best.

1. it needs a collection of all translatable things to be exported on localize.drupal.org

After diving more into translations with validation violations translations I must say I'm not sure that our current approach of extracting translations just magically with potx is right. I'm not sure it's reasonable to think about alternatives that late for d8, but what about making that more explicit?
In addition to supporting potx for t(), we could support an optional file of to-be-translated strings. So if you need to pass something dynamically to t(), you can still register it there. The config system could embrance the metadata to determine translatable strings and write it to the file, once config gets updated. Yes this duplicates some strings, but is that really a problem?

Once we'd have that file written, we do not need a way to statically parse config schema any more. Thus we could embrace the regular typed data API way of adding metadata. Right now, this uses hook discovery, but once switched to class based discovery this should be reasonable simple - dx++

Given #1845546: Implement validation for the TypedData API I really think we should embrace the type data API for config - either from PHP or from yaml. In addition to the must have of config-string-translations, having config being properly validated *independently from form submissions would be a big plus to me.

effulgentsia’s picture

As promised in #319, I posted my initial work in #1865300: Introduce configuration definition file format and API for reading it. Early feedback there would be welcome and appreciated.

I'm not sure why we went away from a straight php2yaml typed data defintion

That issue brings us back to that.

Gábor Hojtsy’s picture

The config system could embrance the metadata to determine translatable strings and write it to the file, once config gets updated. Yes this duplicates some strings, but is that really a problem?

That is funny you are getting us back to there, and I've been discussing "going back to the Barcelona discussions" with heyrocker yesterday, and in fact this very idea (again) with chx as well. One of the options early on (eg. in the Barcelona discussions and earlier) indeed was to either have syntax *in the data yml file* that would identify the translatables, some creative thing like:

key: String
tobetranslated%: Some other string

OR
tobetranslated: t(Some other string)

OR
tobetranslated: Some other string #t

Where obviously %= or t() or #t (or any other kind of syntax) would be syntactical markers in the file to mark items to translate and would not end up as part of the value. Then another one of our ideas was to just do what Views does in Drupal 7 and export a list of translatables at the end of the file, something like:

key: String
tobetranslated: Some other string
thistoo: One more strings
.translatables:
 - Some other string
 - One more string

Both of these proposals were deemed against the simplicity goal of CMI (as well as making the files hardly hand-editable and result in magic/broken side effects). Obviously both of these would be limited to *the localize.drupal.org use case only* and would not let us display translation forms or figure out if an element needs an input format / text editor or should be validated as a number or email address.

It is hard to argue the resulting metadata system that was developed in light of the various times repeated arguments to *keeping the CMI simplicity* and thus not touching the actual data files might in fact be more complex, although encoding the identification of the translatable pieces in the runtime PHP code might not in fact be simpler, and even more importantly it would need the CMI storage system to know which items are translatable (or each implementation in the application layer to have custom logic for it), which also goes against the design goal of the metadata system that it would only be needed if you actually need to identify the translatables, and therefore not put a baggage on regular use of CMI.

So coupling the CMI requirements with the D8MI requirements, we need a system that does not affect the regular CMI runtime operations and/or complicate the simplicity of the CMI files (both requirements from the CMI team) but still let us statically identify the translatable pieces in the files (requirement from D8MI), which all result in a big fat pointer to have a declarative external description format.

Since we need this anyway, and its use or weight does not affect normal CMI operation, we wanted to couple it with a solution for the form generation problem which needs a similar description of the data that lets us generate translation editing forms. That needs a *lot* more information to generate useful forms, and the patch proposed here would not be enough to have useful forms (eg. for a simple example: Views header text which both has an input format associated (and could use WYSIWYG editors) and has tokens offered). So we would need to expand on the original format based on further experience. We cannot really solve everything right away, especially if a simpler format/system needed 300 comments to figure out.

plach’s picture

Can at least some power user delete the system messages so that we get under the 300 threshold?

alexpott’s picture

My 2 cents on this issue.

Q: Why don't people like this patch?
A: This is a complex problem.
A: People don't like the need for a schema and metadata format.
A: XML does it better than YAML.
A: TypedData has proved difficult to use.

So given that we need this complexity - and Gabor has outlined why many times...

Q: Do you we need a schema?
A: Yes. Why? Gabor's comment #322

Q: Do we need all the metadata keys or can this be simplified?
A: If we want to provide a sensible interface with which to translate the values then yes.

Q: XML vs YAML? And is this important?
A: I think not and I think XML will buy us more pain than it worth. And I'm not alone... see http://drupal.org/node/1861640#comment-6831596 but we could swap this out at a later date anyway.

Q: Should we use TypedData?
A: Well we have a TypedData interface and this is typed data so we need a good reason why not to use it. So... Yes see @effulgentsia's work #1865300: Introduce configuration definition file format and API for reading it

So will #1861640: Provide config metadata solely for translation actually lead to a simpler patch? I think not.

The solution in this patch managed to maintain the simplicity of CMI interface. Even with this patch a module does not have to declare metadata and the essential config api is untouched.

A way forward?

  1. Get the metadata format implementation solved in #1865300: Introduce configuration definition file format and API for reading it
  2. Extract the config batch translation from this patch and implement in a new issue.
  3. Continue the context work in #1763640: Introduce config context to make original config and different overrides accessible

Or...

  1. Commit this patch
  2. Finish #1851498: Polish file format of configuration metadata (for translation) with input from #1865300: Introduce configuration definition file format and API for reading it
  3. Continue the context work in #1763640: Introduce config context to make original config and different overrides accessible
gdd’s picture

It has been interesting watching this issue go round and round for the last two years (no matter what we do I suspect it will circle back around again in the D9 cycle.) I was actually never actually 100% against including the metadata in the configuration files themselves, I just don't think we ever found a good way to do it. I remember talking this over a lot with various people in Barcelona a year ago. Other than that I largely agree with Gabor's summary of CMI's goals and priorities.

I have long been a major proponent of making the metadata files simpler from a DX perspective, because module devs are going to have to create them by hand. However, my concern has always been around the wildcard nesting, which is by far the most complicated piece of this. Since none of the counterproposals remove this need, I don't think we're gaining much simplicity out of them in the end. We are making the DX very slightly easier, but we are removing the potential for a very powerful new feature for D8MI that will offer users a greatly enhanced experience. That doesn't seem like sensible ROI to me.

I'm sort of torn on the TypedData question, I haven't reviewed effulgentsia's new patch and this is also the part of the current patch I am least familiar with. I do know that there is a huge ton of abstraction going on there and I'd be interested in what the patch would look like without it. I suspect it would be a lot easier to grok.

We are not switching back to XML. The issues around it are well documented and still wouldn't have solutions, and we absolutely do not have to time to re-re-convert everything. Everyone stop talking about it.

So in summary I guess I largely agree with what alex and gabor are saying above, with the caveat that I'm more open on the TypedData question and need to research it a bit more to understand the pros/cons.

Jose Reyero’s picture

Surprisingly, there's some wider and stronger support for the "we need schema/metadata" idea than I thought before :-)

However, I have to agree current patch and specially the schema format is not beautiful, not to say it's ugly, so I can understand @Dries concerns about it. But also, on the other follow up issues, I don't think it is looking much better and I'm really concerned that after one more month we get stuck at the same point... :-(

So this is what I think: Let's find a good powerful real one. The bad news though is that it's not going to be too simple because we do need to describe complex data. But really I'd rather settle for something powerful than for something simple. So if it's not going to be simple, let's make it powerful!!

And this is (yet one more) schema format based on Kwalify (which is the only viable option I've found out of here, Rx looking too limited to me), and extended to meet Drupal needs.
Also I have to step back from my previous statements about TypedData not being any help here, though I tried to build without it, ended up (once again) reusing TypedDataInterface (thought not the whole thing).

So this builds on both TypedData and Kwalify and really, the schema itself is contained in the schema, so it at least starts simple... :-)

This is it #1866610: Introduce Kwalify-inspired schema format for configuration

Gábor Hojtsy’s picture

All right, we have three options now with three people working in parallel on different schema approaches. If that does not say there is an agreement we need some kind of schema, I don't know what would :)

  1. Using kwalify and typed data: #1866610: Introduce Kwalify-inspired schema format for configuration
  2. Using typed data only: #1865300: Introduce configuration definition file format and API for reading it
  3. Not even using typed data: #1861640: Provide config metadata solely for translation

Follow these and provide review feedback please as appropriate! (The last one has the representative priority of the task, given that Dries postponed this, so not all three show up as critical).

fago’s picture

So coupling the CMI requirements with the D8MI requirements, we need a system that does not affect the regular CMI runtime operations and/or complicate the simplicity of the CMI files (both requirements from the CMI team) but still let us statically identify the translatable pieces in the files (requirement from D8MI), which all result in a big fat pointer to have a declarative external description format.

Yes, but if we export all translatable strings to a separate file on save we do not need any extern description format - thus that would save us from inventing a new file format! On the PHP side we could just re-use what we have (typed data) and make sure that's simple. Writing that separate translation file could happen on config-entity save, for non-entity config a simple helper could do it. Wouldn't that simplify things and make it easier to grok what can be translated? Just look up the file?

Thus, we could keep it out from CMI also. Yes, if you are editing the config file from hand the translation file would become out of sync. But honestly, I don't see a big problem with that. The manual config update would need a CMI reload anyway, what probably a drush command or so would take care of. It can take care of updating the translations file also.

Gábor Hojtsy’s picture

@fago: my understanding was that there is resistance in the CMI team to tie this into the storage layer, like you suggest, but for D8MI, exporting the items on config save (and re-export after hand edit) works as well. That would mean there is a dynamic description layer of the data somehow and nesting of things like image effects with styles is somehow resolved by either plugins or the application layer. Currently the base CMI system attempts to limit itself to being a generic key-value store, your suggestion would essentially dress up the storage layer with types, nesting, inheritance and other metadata pieces in my understanding.

Gábor Hojtsy’s picture

Had a fruitful discussion with @fago about this in IRC which looks like resulted in a very good argument against a declarative metadata format. Think of the inclusions and data wrapping that we want to model properly. As @yched pointed out, parts of config cannot be sure how they are wrapped in other config, while the wrapper cannot be sure what it wraps. The resulting solution of cross-referencing CMI meta-definition files solves the issue however it means that the metadata can only be evaluated with the context of all dependent modules involved (some of which might be contrib), and there is also a versioning component where this metadata can change, so even if localize.drupal.org could pull in the metadata from the dependent modules (downloading and considering those as well), it would not know the version of the module/metadata to use for sure. In short the declarative metadata does not in fact solve the identification problem of the strings because of cross-dependencies and versioning issues. This sounds like a very good argument for actually running the metadata processing in the Drupal runtime and statically store the list of strings identified in files, which then entirely nullifies the need for the metadata itself to be declarative/static. It does put baggage on all CMI save operations though, so direct feedback from CMI leads would be important.

(Clarified: it does not matter if the metadata is static or not, because we need to evaluate it in the context of the .yml file being created on the actual Drupal site. If we don't like static metadata, we can move it into the runtime.)

The full discussion:

[5:09pm] fago: GaborHojtsy, hey! What do you think about exporting translatable strings in a separate file?
[5:09pm] GaborHojtsy: fago: *I* would absolutely be fine with that it makes the identification of them way simpler for multilingual
[5:09pm] GaborHojtsy: fago: my understanding was it was an absolute no-go from CMI
[5:09pm] GaborHojtsy: fago: pinged alexpott and heyrocker about it
[5:10pm] fago: GaborHojtsy, but we could do that on top of CMI, couldn't we?
[5:10pm] GaborHojtsy: fago: it would need the full type, nesting, etc. understanding in CMI when the data is saved
[5:11pm] fago: GaborHojtsy, well, assuming we'd have type data metadata we could use that to find translatable strings and export them to a file - beside of saving the config to CMI. That way, cmi would not have to know?
[5:11pm] GaborHojtsy: fago: well, anytime the data is saved, the data structure would need to be figured out, and it would need to be part of the base system
[5:12pm] GaborHojtsy: fago: eg. if you create drupal commerce, a distro in English, it would still need to have these extracted strings
[5:12pm] GaborHojtsy: fago: so it would need to maintain the functionality on an English only site
[5:12pm] GaborHojtsy: fago: which is effectively adding a type/nesting/inheritance discovery and application step on all config saved
[5:12pm] GaborHojtsy: saves
[5:12pm] GaborHojtsy: fago: on all sites
[5:13pm] fago: GaborHojtsy, but if it's a separate operation we could simply skip that on english sites?
[5:13pm] fago: GaborHojtsy, ah I see your point is we need that even for English sites.
[5:15pm] fago: GaborHojtsy, I guess it could be functionality that you can turn on somehow, so it could be disabled for English sites by default. But if you do a distro and want it, you'd have to enable it
[5:15pm] GaborHojtsy: fago: so even if it is not in CMI per say, it would need to run all the time
[5:15pm] GaborHojtsy: fago: meh
[5:15pm] fago: GaborHojtsy, yes
[5:15pm] GaborHojtsy: fago: so it would then be on users to submit an issue for each module "can you please export your translatable strings in this module"
[5:15pm] fago: GaborHojtsy, writing configuration is not performance critical, who cares whether it writes one or two files?
[5:15pm] GaborHojtsy: fago: because the module maintainer included 2 views exports, but no exported strings
[5:16pm] GaborHojtsy: fago: *i* don't care
[5:16pm] fago: GaborHojtsy, true, that sucks. So it would have to be always on - yep
[5:16pm] GaborHojtsy: fago: I care for finding a solution within the constraints, and I took this as a constraint
[5:17pm] GaborHojtsy: fago: if CMI folks say this is not a constraint, then we opened up for new opportunities
[5:17pm] fago: GaborHojtsy, yes, but it make things a lot harder. So I'm not sure it's the right approach.
[5:17pm] GaborHojtsy: fago: yeah, well
[5:18pm] GaborHojtsy: fago: we have seen where we can get here
[5:18pm] GaborHojtsy: fago: for some reason neither chx, not effulgentsia nor reyero broke out of this assumption, so I think we need feedback from "up"
[5:18pm] GaborHojtsy: as in CMI die-hards
[5:18pm] GaborHojtsy: alexpott might be reading us here
[5:18pm] fago: GaborHojtsy, you know for the translation includes to work out, you'd have to statically find other modules to source the right include? That sounds like a nightmare to do with multiple drupal versions, module versions, etc. I'd even say it's impossible due to versioning issues.
[5:19pm] GaborHojtsy: fago: well, we assume config structure is according to the current module descriptions
[5:19pm] alexpott: GaborHojtsy fago: I'm reading… making mental notes to think about later :)… I'm unfortunately busy.
[5:19pm] GaborHojtsy: fago: if it is defined via code in modules, it will be the same situation
[5:20pm] GaborHojtsy: fago: I assume when modules are updated, the exported strings would need to be regenerated, because they might be affected
[5:20pm] GaborHojtsy: fago: or at least when module updates cause CMI updates
[5:20pm] fago: GaborHojtsy, no it's not, because localize.drupal.org does not know the module versions you were running when exporting the config.
[5:20pm] GaborHojtsy: fago: yeah well THAT is a very good point, it will not be able to take the other contrib modules possibility involved and merge their metadata
[5:21pm] GaborHojtsy: damn
[5:21pm] fago: GaborHojtsy, yeah, but this is what it would have to do for "includes" to work out
[5:21pm] fago: -> better do it in PHP
[5:22pm] GaborHojtsy: fago: eh
[5:22pm] GaborHojtsy: #facepalm
[5:22pm] • GaborHojtsy bangs head to desk
[5:22pm] GaborHojtsy: yeah
[5:22pm] fago: GaborHojtsy, doing it in PHP would be reliable. you'D have a simple file of translatable strings. you can look into it to verify it has found something, so you know it's translatable. no magic involved -> dx++
[5:23pm] GaborHojtsy: fago: yeah I know
[5:23pm] GaborHojtsy: fago: it definitely seems to go against the CMI requirement of keeping the base functionality simple
[5:23pm] GaborHojtsy: fago: however, the inclusionn problem, well, I don't know how I missed that
[5:24pm] fago: GaborHojtsy, keeping which base functionality simple? I think this "simplicity" requirement creates lots of complexity, so it's wrong.
[5:24pm] GaborHojtsy: fago: yeah, well

Jose Reyero’s picture

@fago, @Gábor Hojtsy,
Yeah, ok, so the translatable strings need to be identified by the runtime system exporting the configuration.

However, as I see it, it only shifts requirements a little bit for the schema system. We still need need a way to identify that translatable strings / elements. And it may be the schema very well.

The option, as I understand is "everything in PHP", for which I fail to see how introducing yet another hook is better DX than adding a yml file. Actually each complex hook we add is one more "file format" to learn.

Still, please, don't dare tell me the 'pure' PHP solution is better until such solution exists.

"Existence" is a pre-requisite for "being better".

yched’s picture

I don't think I grasp why localize.drupal.org is a problem with the "declarative metadata files" approach ?
How is localize.drupal.org involved here ? Could you clarify that ? (sorry to ask, I'm not too familiar with that area)

Jose Reyero’s picture

@yched,
Localize.drupal.org uses pre-extracted lists of strings and it gets them from the module's code using potx extractor. That's done on a package by package basis.
Now if module A depends on module B and A contains configuration for a plugin provided by B that contains translatable strings we need both modules (with the right versions) to be able to tell which are the strings.
So though theoretically possible (you would need to checkout all the dependencies with the right versions for generating the strings for each package), it looks like a hardly workable approach.

gdd’s picture

my understanding was that there is resistance in the CMI team to tie this into the storage layer

I actually don't remember expressing this, although it is possible I did back around Barcelona, when I was fighting against the entire concept of metadata to begin with. However we've come a long way since then, and I would be willing to consider all options that can move us forward at this point. Yesterday I talked some with effulgentsia, xjm, beejeebus, and chx about this. An idea I've been toying with is defining the metadata in PHP (either in annotations, or in TypedData, or whatever) and then actually writing that data out to metadata files on import/install/update. We would need a way for new plugins being installed to trigger a rewrite of the metdata they affect (or possibly just all of it, this is a rare operation and not really a choke point.) This could then get to a world where we have one metadata file per config file. If a plugin changes, you trigger a rewrite and it all gets redone. No more wildcard nesting.

While it does add metadata-specific code into the cmi storage engines, I think the tradeoff could be worth it in terms of DX. I don't have a good picture in my head of how the metadata would look or be declared or anything, as I said before my knowledge of a lot of these new subsystems is pretty limited. Note this is just based on like a day of thoughts rattling around in my head, so there's likely lots of things I'm not thinking through. I'm going to try and find time this weekend to see if I can't work through a proof of concept of some sort.

yched’s picture

@reyero: thanks, makes sense.

Agreed that regenerating the metadata tree associated to a given config item without being on the system that generated the config item itself sounds like asking for trouble. Generating the associated metadata while writing the config file itself does seem much more robust.

Gábor Hojtsy’s picture

All right, so seems like we'd either need to compile a CMI .yml-file specific metadata file in the Drupal runtime (in storage) or actually execute that metadata on the data and extract the translatable strings. My understanding is @heyrocker is pursuing the former route and will hopefully come back with a patch.

Jose Reyero’s picture

While this may mean some workflow change, we still have the need to identify these translatable strings in the configuration for which having schema data is still a solution. Actually there's no other solution anywhere so far.

So I really hope people advocating "other better solutions" are busy working on them. Otherwise I'm afraid Drupal 8 will ship without configuration strings being translatable at all.

yched’s picture

Hm. Possible issue with generating metadata files on write of the config item : config items written during hook_update_N.
We're not supposed to call api funcs in there, just work on raw data. config() read & write have been deemed ok so far (they are the equivalent of the direct db read & write on config tables in D7), but the mechanisms to fetch metadata snippets from various mudules / plugins are probably not kosher in hook_update.

yched’s picture

#338 is the "on config write" part of the problem.
Not strictly part of this exact issue, but we probably have another problem on config read: if the lowest level API we have, config()->get(), already bakes in switching values to the ones assigned to the current language, then how do we access the very-very-raw data in hook_update_N ?
hook_update_N is the place where you'd modify the values or structure of existing config items.
It needs to be able to do so on the raw, untranslated values ?

Gábor Hojtsy’s picture

@yched: the loading of originals problem had a solution in previous patches in #1763640: Introduce config context to make original config and different overrides accessible, although beejeebus did not cover that in his current proposals. We hoped to fix it via the same mechanism, where you say what kind of override you want to have working or ignored.

yched’s picture

@Gabor: Ack.

So, "just" the issue that we're not sure we'll be able to compute metadata trees for config() writes in hook_update_N()...

Gábor Hojtsy’s picture

@yched: @beejeebus made sure we have an issue to track (the raw data inaccessibility bug) at #1868028: Raw (original) config data is not accessible.

fago’s picture

So, "just" the issue that we're not sure we'll be able to compute metadata trees for config() writes in hook_update_N()...

We could have an upper-level, metadata enabled config writing routine which takes advantage of the metadata to update translation also + an lower level config writing routine that writes to the config files only. Writing the translation file should have a similar lower level API then, such that the update functions could leverage both lower level APIs without depending on any metadata or system state.

I'd assume the lower level config API is what CMI is now, but I'm not sure how the upper-level API could like. We could do it as part of writing config-entities, but that won't cut it as we need to take care of single-config forms also. We could add it as another function call there.

Or, as an alternative we could add TranslationFileManager in Config and invoke it from save(), but move it to an injectable dependency. Then, in update functions we inject a null TranslationFileManager and care about translation file writing on our own (if necessary)?

Gábor Hojtsy’s picture

Gosh, if only there would be a way to get people review the forks of metadata formats, and somehow narrow the extent that we get people occupied here (and make some progress):

  1. Using kwalify and typed data: #1866610: Introduce Kwalify-inspired schema format for configuration
  2. Using typed data only: #1865300: Introduce configuration definition file format and API for reading it
  3. Using typed data as well (see patches above)
  4. Not even using typed data: #1861640: Provide config metadata solely for translation

So my understanding is that @heyrocker (and maybe @fago?) is advocating possibly not sourcing the metadata from .yml files but generating them out into .yml files, so there would be two metadata formats, one in annotations and PHP and one in .yml files, and the .yml would be generated. *The two largely different metadata formats are proposed to improve DX*. Sounds like the generated metadata format needs a system as well (one of the above?).

Everybody else seems to assume the metadata would be sourced from .yml files (see @effulgentsia's recent comment at http://drupal.org/node/1865300#comment-6857916). There is still some uncertainty there that maybe their inclusion/merging rules would be sourced from PHP somehow (see @effulgentsia again).

Then there is the above patch, the kwalify patch and @chx's patch where the inclusion is assumed to be happening in the metadata .yml level.

Are we clarifying anything?! (I'm personally amazed that this many people care so much yet cannot get to a common ground).

Gábor Hojtsy’s picture

Category: feature » task
Issue tags: +Regression

Refiling as a task. While we might not agree yet on the implementation path, it would be a serious regression if this is not implemented. "Hey, we added views! But, well, you cannot translate views to other languages. Well, you could do that in Drupal 7? Yes, not anymore." is not a discussion point we'd want to use.

#1861640: Provide config metadata solely for translation was revoked, so there are two (or three) proposals for metadata formats (see above), depending on whether we count the patches above in this issue or only the two remaining forks. There are no issues or alternate proposals for anything against the above patch but the format and how the metadata files are located and discovered.

gdd’s picture

I would like to establish a deadline for deciding how we're going to proceed with this. If we don't, we are going to go back and forth on these decisions in a circle until feature freeze comes and then we're screwed. My feeling is we will probably need about a month to implement any new proposal and get it to RTBC, so I'd like to lay down January 15 as our deadline or deciding what to do. If we haven't agreed on any new proposal, then we reroll the original RTBC patch from #285 and get it in. We have to get this done, and right now I don't see any light at the end of the tunnel which scares the crap out of me.

So there is just under a month to work out the new proposals and establish consensus. Lets do it or admit its not getting done and go with what we've got.

Gábor Hojtsy’s picture

@heyrocker: sounds like a good plan! Looking forward to continued feedback and patches :)

svettes’s picture

Fabulous plan @heyrocker, @gabor seems on board too -- but no movement since :) Hols are keeping everyone busy I think! Anyone care to chime in?
Happy New Year everybodeh!

Gábor Hojtsy’s picture

Posted this call for feedback on the Core group (also going to Drupal Planet): http://groups.drupal.org/node/275008

Gábor Hojtsy’s picture

Update! The action is now in #1866610: Introduce Kwalify-inspired schema format for configuration.

@chx revoked #1861640: Provide config metadata solely for translation in favor of @effulgentsia's approach and @effulgentsia revoked #1865300: Introduce configuration definition file format and API for reading it in favor of @reyero's proposal in #1866610: Introduce Kwalify-inspired schema format for configuration.

It is the most verbose format of all of the proposals, however it is based on (and extends) a pre-existing format.

Although we now have a "replacement" issue for this, I'm NOT marking this a duplicate, since there are a *lot* of integration pieces in the proposed patches in this issue that are not covered by just introducing the format and will need extraction and rerolling to integrate with the other format.

Gábor Hojtsy’s picture

Issue summary: View changes

fix typo

gdd’s picture

So here we are at January 15. After several attempts and some long conversations with people I haven't been able to find a way to make the declarative method work, so I guess it is tabled.

In the meantime, we are waiting for feedback from Dries on the Kwalify file format. Both Gabor and I agree that we want to make sure Dries is on board with the plan before proceeding, as we don't want to spend a bunch of time unnecessarily with so little time to go. Hopefully that will happen this week. If not we will re-evaluate again on Friday.

Gábor Hojtsy’s picture

Need reviews on the kwalify-inspired schema that is the only proposal now and is supported by Dries: https://drupal.org/node/1866610#comment-6973166 (ignore comment #40, that was an interdiff).

Gábor Hojtsy’s picture

#1866610: Introduce Kwalify-inspired schema format for configuration just got committed. Woot! Instead of reopening this for the locale integration (that we need to resurrect from the latest patches here), I think opening a new issue for that would be beneficial. I'll look into that tomorrow if nobody beats me to it :)

Gábor Hojtsy’s picture

Assigned: Dries » Unassigned
Status: Postponed » Closed (duplicate)

All right, opened #1905152: Integrate config schema with locale, so shipped configuration is translated to resurrect all the missing functionality that was not added in #1866610: Introduce Kwalify-inspired schema format for configuration but was part of the above patches. Since that functionality had no debates in this issue either, I think its fine to leave these 320+ comments behind.

That said, marking duplicate of #1866610: Introduce Kwalify-inspired schema format for configuration.

Gábor Hojtsy’s picture

Gábor Hojtsy’s picture

Also opened #1952394: Add configuration translation user interface module in core just for the heck of it, given that Dries opened the possibility of features getting in later (http://buytaert.net/code-freeze-and-thresholds). Although given the number of critical/major bugs and Dries' negative reviews above on even the concept, I don't have a lot of hope for it to land in core for Drupal 8. (The contrib module is progressing nicely btw).

yched’s picture

Hand waving for #1975112: Config schema mechanisms cannot reflect alterable config trees - "the elephant has officially (re-)entered the room"

Gábor Hojtsy’s picture

Re #330 - the versioning problem, I just opened #1977498: Add version tracking for configuration schema and data where we can discuss that now that the locale module integration landed. @fago, et. al. your feedback is very welcome there. We need to resolve that to support module updates in Drupal 8 going forward as well as to make it possible for localize.drupal.org to figure out data/schema versions as they relate to each other.

Gábor Hojtsy’s picture

Issue summary: View changes

Updated issue summary.

Leeteq’s picture