#2309105: Support for Drupal 8 shipped configuration translatables without external schema dependency solved extracting default config strings which do not have external dependencies (only use types that they define). That would in practice only work for core. Any contrib module would not work. We still need to solve the problem of parsing configuration where some schema is defined outside of the project being parsed (ie. in external dependencies). Because schema is not versioned, potx will need to store some version(s) of the schema of those projects and parse dependencies of projects to be able to handle configuration parsing.

More prior discussion of this at http://drupal.org/node/1648930#comment-6845334 and above.

An overview of the code is given here.

  • At potx start, all available modules are found in the local system, based on Drupal 8's directory structure.
  • During yaml parsing, some metadata is gathered about modules, including module dependency list, config schema files, ...
  • Parsing shipped configs happens at a different pass, and works as follows:
  • First, process the schemas in /core/config/schema. All data types and base definitions exist there.
  • Then, process the schemas for the given module, which includes finding its dependencies and processing them, recursively.
  • Each config file could include its own dependencies. In that case, process them too, but keep it isolated so that it doesn't affect other configs' parsing.
  • For optional config, the whole processed schema should be rebuilt for each config file, since their owning module might be different.
  • Finding the shipped config translatables itself isn't modified here, and involves traversing the config file based on its matching schema.

List of new defined globals:

  • global $potx_callbacks contains the list of functions that should be overriden by l10n_server to allow using a database, and not require extracting the dependency modules themselves.
  • global $_potx_module_metadata contains metadata about the modules that are being parsed by potx. Right now, it includes the module's dependencies, and the list of config files (schema, install, optional) in the module. This might later be used for similar dependency resolution in yaml translation extraction too.
  • global $_potx_processed_schema contains all the processed schema required for parsing a shipped config file. This might include schema from the module, its dependencies, the real owner (in case of optional config), and the module dependencies in the shipped config itself.
  • global $_potx_processed_modules tracks the list of processed modules, so that we don't process a module's schema multiple times.
  • global $_potx_module_schema contains the processed schema for a single module. This can be cached, or stored in the database, for using later.

The following globals are defined in potx.local.inc:

  • global $_potx_found_modules: when parsing a module locally, potx should try to find its dependencies in the local file system, based on the Drupal 8 directory structure. The global $_potx_found_modules global holds the list of main directories that can contain modules, e.g. core/modules, themes/, sites/all/modules.
  • global $_potx_schema_lookup: When parsing optional config in a module, the module itself might not contain the config's schema. Instead, potx should look through all modules to find a matching schema. To makes this faster (and possible), a lookup table is created from all config matching keys in schema (e.g. block.block.*) and their corresponding modules.
  • global $_potx_reverse_lookup_built: Since creating the lookup table is expensive, it isn't built until an actual optional config is parsed. This global is used to track when it's built. Come to think of it, I can probably get rid of this ...
#32 shipped-config-contrib-1933988-32.patch55.24 KBherom
PASSED: [[SimpleTest]]: [MySQL] 267 pass(es). View
#28 interdiff.txt338 bytesgarphy
#28 shipped-config-contrib-1933988-28.patch45.42 KBgarphy
PASSED: [[SimpleTest]]: [MySQL] 252 pass(es). View
#25 shipped-config-contrib-1933988-25.patch45.26 KBherom
PASSED: [[SimpleTest]]: [MySQL] 252 pass(es). View
#25 interdiff-1933988-23-25.txt21.05 KBherom
#23 shipped-config-contrib-1933988-23.patch25.76 KBherom
FAILED: [[SimpleTest]]: [MySQL] 237 pass(es), 16 fail(s), and 4 exception(s). View
#22 shipped-config-contrib-1933988-22.patch13.05 KBherom
#13 shipped-config-contrib-1933988-13.patch6.01 KBherom
Members fund testing for the Drupal project. Drupal Association Learn more


Gábor Hojtsy’s picture

Priority: Normal » Major
Gábor Hojtsy’s picture

Issue tags: +Drupal 8 compatibility

Also tag with this unique tag for findability.

Gábor Hojtsy’s picture

Version: 7.x-2.x-dev » 6.x-3.x-dev
Priority: Major » Critical
Issue summary: View changes

Moving to the right version and marking with proper status given this is critical for D8 translatability.

drupalshrek’s picture

I just came here from your interesting blog post http://hojtsy.hu/blog/2014-may-19/drupal-8-multilingual-tidbits-15-confi...

So, based on comment #3, this issue has nothing to do with Drupal 8, but only Drupal 6?! I'm sure I'm missing something.

Gábor Hojtsy’s picture

This issue is about parsing the configuration schema structure on localize.drupal.org. Localize.drupal.org runs on Drupal 6. So we need the Drupal 6 code of the potx module to parse the configuration schema in Drupal 8 yes.

drupalshrek’s picture

An alternative, I suppose, logically at least, would be to upgrade localize.drupal.org to Drupal 7?

Gábor Hojtsy’s picture

That would not really help with needing to resolve this issue custom on localize.drupal.org or not. Just we'd need to implement the functionality on Drupal 7. It is not available there either. So it is not different at all from the perspective of this issue if we need to build it on Drupal 7 or 6.

herom’s picture

Started working on this in #2309105: Support for Drupal 8 shipped configuration translatables without external schema dependency. That should support extracting shipped configuration translatables from Drupal 8 core.

YesCT’s picture

adding "infrastructure blocker for Drupal 8.0.0" tag, since this blocks the drupal 8 core issue: #2267715: [meta] Drupal.org (websites/infra) blockers to a Drupal 8 RC1, so that the infrastructure blockers to d8 release (in issue queues outside of the core queue) is accurate. Remove the blocker tag when this issue is fixed, and update 2267715.

webchick’s picture

Trying to get a handle on release blockers prior to 8.0.0 and what (if anything) the community can do to help things along in preparation of https://events.drupal.org/losangeles2015/sessions/plain-drupal-english-g...

Is this still an issue anymore given herom's work in #2309105: Support for Drupal 8 shipped configuration translatables without external schema dependency? If so, could the issue summary be updated with an outline of where to start?

Gábor Hojtsy’s picture

Title: Support for Drupal 8 shipped configuration translatables » Support for Drupal 8 shipped configuration translatables with external dependencies (in effect contrib)
Issue summary: View changes

Updated the summary. This is still needed.

japerry’s picture

Version: 6.x-3.x-dev » 7.x-3.x-dev

Marking 7.x due to localize.drupal.org getting upgraded to d7.

herom’s picture

Assigned: Unassigned » herom
6.01 KB

I just started working on this again.
Here's a WIP patch. It's not ready for review, but it does handle external dependencies for core/contrib modules on a local drupal install.

The remaining work is handling optional dependencies, allowing l.d.o to change part of its behavior (on fetching dependencies), adding comments, and writing tests, hopefully for next week.

Gábor Hojtsy’s picture

@herom: do you envision l.d.o would fetch the dependency tar.gz's and extract those too? I think a possibly much faster parsing method would be to store the schemas from projects in a database table and take them from there. It would be a fine dance definitely to align that with potx and l10n_server.

herom’s picture

Adding a database is a good idea. The problem is that while parsing all modules on l.d.o, a module might be parsed before its dependencies. So, the dependecy's schema in database might be out-of-date. In that case, l10n_server needs to fetch the dependecy's tar.gz and update the schema in database.

Gábor Hojtsy’s picture

If there is already schema in the DB, how do we tell if its outdated and why would it be?

herom’s picture

Here's a simple scenario:
Module B depends on Module A. Module A has previously released v8.0-1.0, and its schema is extracted and stored in the DB.
Now, both Module A and Module B release a new version. If l10n_server requests parsing Module B before Module A, then when loading Module A's schema as a dependency from DB, the schema would be outdated.

I agree that we need a database, but it should also store version information, and when loading dependecies' schema should check whether a new version is released and update it if necessary.

Gábor Hojtsy’s picture

I don't think modules would necessarily provide version dependency information for each other, they usually just provide the name as dependency? I can see how taking the latest schema there would be good, but also I can see how downloading all dependent projects all the time in case they have new releases would be painful.

herom’s picture

Well, there is definitely a trade-off between correctness and speed. Unfortunately I don't have any numbers to know how big the trade-off would be (since we are going to extract and update Module A's schema anyway, only later in the process).
Assuming we don't check for latest schema for dependencies, once a module's schema gets updated, some of its dependent modules might not see the new schema and miss a few strings on the first cron run after that. But that will be fixed/propagated on the second cron run.

So, for the sake of moving forward, I will leave that part out; No need for talkback with l10n_server or fetching dependencies' tar.gz files.

Gábor Hojtsy’s picture

Its hard to tell what is exactly going on in the patch. Can you explain the proposed resolution? Also, you have lots of tabs used for indentation, so fixing those may be nice to get it easier to be reviewed. Other than that:

  1. +++ b/potx.inc
    @@ -1985,34 +1985,51 @@ function _potx_load_yaml_translation_patterns($path) {
    +  	    'deps' => isset($yaml['dependencies']) ? $yaml['dependencies'] : array()

    I would not abbreviate this if possible :)

  2. +++ b/potx.inc
    @@ -2357,11 +2362,37 @@ function _potx_replace_variable($value, $data) {
    +      $_potx_processed_schema = array(
    +          'translatables' => array('label', 'text', 'date_format'),

    This had a todo up there, removed here, but the concern does not seem to be resolved?

Gábor Hojtsy’s picture

Status: Active » Needs work
herom’s picture

Updated the patch to use the database for storing and fetching dependency schemas, which allows simpler integration with l10n_server, per Gábor's suggestion.

There are a few bits of work remaining:

  • Handling optional dependencies.
  • Single module extraction. Right now, the code needs to find all the dependencies in the given path (basically, the drupal root). So, if the path to a module is given, the shipped config extraction doesn't work.
  • Adding comments and explaining the code.
  • A few tests.

The code is not ready for review yet. But about the points in #20:
indentations fixed. bad IDE config.
1. fixed.
2. The comment meant that without handling dependencies, there needed to be some basic defaults. Now that dependency handling is implemented, the todo is fixed.

herom’s picture

Status: Needs work » Needs review
25.76 KB
FAILED: [[SimpleTest]]: [MySQL] 237 pass(es), 16 fail(s), and 4 exception(s). View

Here's a new patch. Finally ready for review.

Changes since last patch:

  • Fixed handling dependencies defined in config files themselves.
  • Extracting translatables from a single module using drush is fixed: drush finds other possible locations for the dependency modules in the drupal folder hierarchy.
  • Tests are not added, but it's been tested on 50+ real-word cases (core modules), and it's working correctly. Hopefully, someone else can come up with a full test-case for this.
  • Fixed handling optional config.

Handling optional config turned out to be harder than I imagined. I added another table (potx_schema_reverse_lookup) to help with that. Since the real owner/provider of an optional config is not explicitly defined, dependency resolution cannot be used there. The real owner module should have been already processed in an earlier run of potx (even locally). Than means, for example to extract all translatable from core, drush potx should be called twice on the /path/to/drupal/core

Status: Needs review » Needs work

The last submitted patch, 23: shipped-config-contrib-1933988-23.patch, failed testing.

herom’s picture

Assigned: herom » Unassigned
Status: Needs work » Needs review
21.05 KB
45.26 KB
PASSED: [[SimpleTest]]: [MySQL] 252 pass(es). View

Fixing test fails. They were mostly from the tests themselves: Bad sequence definition in test schema files, missing .info.yml for test module, missing base /core/config/schema files since default translatable definitions were removed, and a single fail (translatable string "1") which was actually related to yaml translatable extraction (unrelated here) that I disabled.

garphy’s picture

Status: Needs review » Needs work

I successfully achieved to extract labels of custom ECK entities from their exported configuration like this example :

Filename: eck.eck_entity_type.a_sample_eck_type.yml

id: a_sample_eck_type
uuid: 92c9047a-b4bb-468d-8000-e4376b95a7be
label: 'A sample ECK type'

But, I think we're missing hook_update_N() for the newly added tables.

garphy’s picture

Assigned: Unassigned » garphy
garphy’s picture

Status: Needs work » Needs review
45.42 KB
PASSED: [[SimpleTest]]: [MySQL] 252 pass(es). View
338 bytes

Added hook_update_N()

Gábor Hojtsy’s picture

Status: Needs review » Needs work

I spent about an hour trying to review this code.

1. Looks like you only look for config schema/files in modules? Why not in profiles/themes? Core provides them there too and contrib will too.

2. Looks like this patch makes potx require a database backend, while before it was not the case. I think it would be nice to keep potx as a possible stand-alone thing that people can use with drush, etc. to parse stand-alone things. Now it looks like this runs into a db error in those cases no? I'd rather potx say it does not have sufficient schema for those cases.

3. Generally its very hard to review this code given you did not provide an overview of the changes in the proposed parsing structure / process. It would be amazing to see why you did what in a high level overview (AKA issue summary) so I don't need to stumble around trying to figure it out. Eg. how does this plan to work with l10n_server?

herom’s picture

1. That code is only for dependency resolution. I agree that themes should be added too, but can anything depend on a profile?

2. The database requirement wasn't necessary; it was only added so that integration with l10n_server would be a lot easier (and I incorrectly thought this was something you asked for, hence my comment in #22). The dependecy modules still need to be found though, since config files aren't standalone very much (I assume they are all going to rely on core modules' schemas to some degree).

So, I can remove the database, but add a --dependencies-path=/path/to/modules command-line argument, or require that modules should be parsed inside a working drupal installation with all dependencies to extract their shipped config translatables.

Gábor Hojtsy’s picture

@herom: I think it would be great if the parser would work without a database but it could use said database if its there. Can you update the issue summary with an overview of what is going on in the patch?

herom’s picture

Assigned: garphy » Unassigned
Status: Needs work » Needs review
55.24 KB
PASSED: [[SimpleTest]]: [MySQL] 267 pass(es). View

Here is a new patch. It addresses the points in #29, I think.
The database tables are removed, and the way it's going to work with l10n_server is a lot more clear (6 functions to override).

I'll add some information, including what each new global does, and a high-level explanation of how the code works, in 1-2 hours.

I rewrote a lot of the patch, and failed to rebase the old patch on HEAD, so, no interdiffs, sorry.

herom’s picture

Issue summary: View changes

Added explanation about the new globals to the issue summary. high-level explanation of the code is next ...

herom’s picture

Issue summary: View changes

Ok, done. Ready for review now.

Gábor Hojtsy’s picture

Status: Needs review » Reviewed & tested by the community

That seems to be separating the concerns well. The local inc file is not very elegant but the existing extension capabilities ($string_callback, etc) are on par :D Ideally sometime we'll have a good month to refactor potx. (One can wish :D). Let's get this in and then integrate with l10n_server.

  • herom committed 9657a23 on 7.x-3.x
    Issue #1933988 by herom, Gábor Hojtsy, garphy: Support for Drupal 8...
herom’s picture

Status: Reviewed & tested by the community » Fixed

Committed. Yay!

Gábor Hojtsy’s picture

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.