YAML discovery incompatible with translations

#2065571-44: Add YAML Plugin discovery @pwolanin:

Well, we do have a set structure for each plugin, just not as a general feature of this type of discovery.

In another issue we are talking about improving plugin documentation by documentation a defaultConfiguration() method that would list the possible keys. Is that a place we could e.g. have a doc annotation and maybe something that tells you what yaml file pattern to look for?

You are suggesting parsing a doc annotation to get a structure to parse another YML file? Yet one more way to define/explain a YML structure?!?

It is safe for all plugins to assume that 'title'/'label'/'description' are going to have values that need to be / should be translatable? And there are not going to be other keys?

So far we don't have direct specification of what is translatable in a .yml file. We looked at it, but it seemed counter-intuitive and wrong for DX. If you need to repeat in each .info.yml file that the description is translatable, it is painful if we can just easily codify that somewhere else. So far all .yml files outside of the config/* dirs had defined structures, so we knew the structure of all of them, so we can code PHP to extract the strings that we knew we need. So far only .yml files in config/* had dynamic data structures, and that is where we have the config schema system to explain the structure, which can be used to discover all translatable strings. So you only need to define the node type config structure for example once but all node type .yml files can then be parsed for translatable names, labels, default values, etc.

I think one option would be to somehow make the schema apply to these .yml files even though they are outside of config/* but that would be a glaring and confusing exception. I'm not sure making up yet another way to define what is translatable in .yml is a good idea :/

Just so I'm sure I understand this correctly, we're discussing the discovering the strings that need to be translated so they can be listed in translation UI's and the like?

Exactly!

For discovering what?

I'd like to reiterate that IMHO having very different solutions for certain .yml files (routing.yml, info.yml, etc), annotations, config .yml files and then plugin YML files for translation sounds like a *HUGE* mess. It is already complex enough with t(), $this->t(), automated string discoveries and config schemas. It would be great to be able to map this to one of the existing solutions instead of making up yet one more parallel system that core would need to support.

I think **purely from the point of extraction**, having local markers in YML files for each translatable string is a godsend. Localize.drupal.org would easily parse all config .yml files and any other .yml file without needing any special code. (Right now parsing of the config .yml files is still ahead of us and going to be an uphill battle that may take a lot of magic and possibly bugos behaviours on the way as accepted side effects due to lack of config versioning).

HOWEVER from a human DX perspective, writing .info.yml files like:

name: !translation Menu
type: module
description: !translation 'Allows administrators to customize the site navigation menu.'
package: !translation Core
version: VERSION
core: 8.x
configure: admin/structure/menu
dependencies:
  - menu_link

sounds like a definite problem. If Drupal already knows those are going to be translatable, why make me work so much to write those obscure things? Also, for Views and all the config entities and every config handling code to be able to tell when saving a config .yml file which one to save with this marker would currently require that saving always go through the config schema and the translatability information is retrieved from there and put into the yml verbatim like this. So lots more upfront work on the .yml generator side as well for all the config entities and regular config.

I believe we considered this "!translation" option in Barcelona last summer and discarded it for the (a) lack of Symfony support and (b) DX problem with having this all around and needing to deal with it all the time.

What would it help with though is:

- it would make the locale code which parses default config for translatable strings *way* easier, it can ignore looking at the schema
- it would make the localize.drupal.org code to parse .yml files much smaller than it already is and make it effortlessly possible to parse all config .yml as well as any plugin .yml file

It would be on the expense of this ugly thing spreading everywhere though. So far we avoided this and kept nicer DX by encoding known patterns about .yml files to the parser as well as providing a description format (the config schema) for locale to parse the files (and config translation to generate forms).

The introduction of !translation would not make schemas useless, since (a) you would need schemas to be able to tell where to export !translation before values in the first place (or write even more code for each config entity / config specifically) and (b) the config schemas are still needed to generate a UI for translation.

Maybe bring this to the CMI and/or plugin discussion in Prague? It would definitely need much more of a consensus.

@EclipseGC: currently the schema is used only for locale module's parsing so it can identify the translatable strings without explicit local markers like !t or !translate. Several people on the config initiative feel strong about not requiring schemas for basic config operation. Requiring schemas would also help with #1653026: [META] Use properly typed values in module configuration where several people have been pushing back on applying schemas for that. So if we'd use schemas for dumping .yml that would be a definite change. Schemas are not needed to read .yml in any way, except if you need to introspect the structure which is in locale module (and the coming config_translation module which builds forms based on the schema).

@pwolanin: yeah you may have a French-Spanish site, you create some views in French and then need to translate to Spanish. We need to identify the translatable strings in the French view. Locale actually does not interfere there because you custom-created those views, but config_translation needs this info (it gets it from the schema). *However* all the views, content types, fields, etc. that are shipped with modules and packaged on d.o will be originally in English and need those strings extracted and identified on localize.drupal.org. Currently, since there is no local marker for translatable strings, localize.drupal.org would need to parse the .yml files through the schema (which is pretty darn hard (but not impossible) due to module dependencies, plugins, etc). The localize.drupal.org parsing is a yet unresolved problem.

Once again I think this should be opened up as a larger scale DX question. If people are not freaked out with !t showing up in their YML and we can painlessly dump that in .yml generators, then it may gain acceptance. So far my understanding was people don't want this in their YML files :/

The schema is currently only used in core by locale module to find all the translatable strings. It needs to traverse arbitrary nested containers like a view .yml which has arbitrary plugins to find all their translatable strings. https://drupal.org/project/config_inspector shows a few potential ways the schema can be used. Eg. its useful to inspect your config data (debug, find issues). Labels help find what your stuff is about there. It also shows how the schema can be used to generate a form from your data. This may be useful for quick input solutions or where otherwise the data only has a web service based entry point or to generate form scaffolding that can be further tweaked. Schemas may also be used to validate the data that comes in through a web service or migration and tell you where errors are based on labels and nesting information. None of this is happening in core or contrib (yet).

https://drupal.org/project/config_translation is a module that actually uses the config schemas to generate translation forms for *every* configuration in Drupal that may have translatable things (think site name through filter formats, roles and shortcut sets to views). It knows how to build a relatively useful form out of the nested structures using the labels and data types loaded from the schema. So if you have a path field or an integer field, it can know which simple widget to use for that.

The main driver behind the config schemas was (a) lack of another way to build such translation forms (b) lack of another way to figure out translatable things in .yml files with *static parsing*. So it was all multilingual driven yeah. But as you can see there are tons of potential uses outside of languages.

We could have decided that each configuration screen need to be implemented twice, once for the main editing screen and once for the translation screen. Eg. Views would have needed to build an alternate form building pipeline for the plugins to build such a translation form, there would be two versions of the site info settings form, etc. We looked at automatically generating translation forms out of even simple forms like account settings or more complex forms like editor/input format settings or views, but we found no practical way to do that and ensure there is going to be no privilege escalation, etc. Also solution for forms would not have led to any solution for the static parsing of translatable strings, so *then* using !t or !translate would have been mandatory and all config save pipelines would need to know about which values are translatable and mark them appropriately at all times. Currently config save does not need to know about this (and is therefore more speedy for the default case). (There was another alternate route discussed where all config would be able to tell their translatables and export it elsewhere as well, so when you commit config exports to d.o, you'd include translatable text as copies in another file as well. People did not like that for the extra build step and again for the needed built-in know-how that may slow down config in regular operations).

Configuration schemas are an optional component at the moment and you don't need to implement them if you don't care about potential contrib advantages or working in a foreign language / multilingual environment at all.

In the end configuration schemas may be more complex than they need to be so the base config is simpler and can serve the base use case without more complexity in a self-contained way.

Bingo, in fact config schemas map to typed data types (although the integration is not perfect): #1905230: Improve the typed data API usage of configuration schema

@neclimdul: so if tabs are plugins, where would tab labels go? Tab labels are inherently human readable text and will be in a specific human language. (For d.o hosted modules in English).

@neclimdul: re #25: how would your option (a) work if the source code of your d.o contributed project is dropped into a 3rd party system for parsing out strings (that cannot run your code at all). I think the only way it would work is (a) happens *before* committing to d.o and results in an exported list, so you commit the exported string list as a separate thing. IMHO one of four things may to happen:

1. The strings are marked explicitly in the files. This is how t(), $this->t(), @Translate(), etc. operate and !t could be similar.
2. The file structure is well-known so strings can be extracted based on those rules. This is how *.info.yml, *.routing.yml, hook_menu(), etc. works.
3. A parseable structure definition is included, so the data can be parsed with that structure to extract the strings. Config has schema for this.
4. The runtime code can identify the translatables in some way and exports in a static format. Then this method falls back on (2) since that static format would have a known structure to parse. No core examples for this because people seemed to despise "build systems" like this.

Once again the key is that we cannot run from the code in question that has the strings, because we need to run it through a 3rd party systems (and cannot just get all the dependencies, etc. involved) to identify the strings. So if there is nothing better than running the code, then it needs to be run on the dev environment and the strings exported ahead of commit to the community (4). Otherwise we need a static way to know how to find the strings, which is what 1-3 are about. Either local markers, or well known formats or an external description of structure.

@EclipseGC: I don't contest that these .yml files may be candidates for local markers like !t because the structure does not need to be explained necessarily. I want to point out that people will not see the reason why info.yml would not need this while the plugin .yml needs it. So far we managed to follow common rules around how .yml files should be structured.

@pwolanin: take a views or field instance YML and try to autogenerate a translation form based solely on the structure and knowledge about which items are translatable or not and you'll quickly see why we added schemas with types and labels. As said above an alternative would have been that all plugins would provide their own translation forms separate from their editing form and that all save operations would know about translatable strings, so they would export local markets at all times.. Definitely more work vs. writing a declarative structure definition.

Also, once again the schema may be useful for data validation, in migration especially or with web services. It may easily allow to create a web service endpoint which even automatically documents itself :) So lots of potential side benefits that would not have come with writing more form api arrays.

Nowadays schema is used for many more things, eg. it is used to cast values so they get proper typed values and are deployment safe. One option to resolve this is we introduce schema for the plugin YAML files, but that would be quite some twisting of the structure, and we may not be able to fully define schema for all plugin YML structures. That would mean we should apply schema for core plugin YML files as well.

Yeah that is a fine option with me. Just make sure to introduce this to all plugin YAMLs then including menu items, local tasks, etc. for consistency. Less ways to do the same thing the better. Let's not make it magically work in some cases and not in most others.

@herom: I think the good news is we can start using this NOW in potx, we can put this file into .potx and support it loading any other files from contrib projects as being parsed. We should just come up with a file naming pattern. I'm not happy about "yaml_translatables.yml" but we can iterate on that :D Should we move this issue to the potx queue or open yet one more issue?

#2322839: Unify YAML translation extraction, and allow extension by contrib. resolves this in potx module. Contribs will be able to define their own files. It now uses the $your_module/yaml_translation_patterns.yml file name. Better suggestions welcome :)