Problem/Motivation

When importing .po files, the integration with Javascript translation file creation and configuration translation file creation is broken. The batch running these operations only recreates JS translation files for the first 100 strings (if any config strings are found in the first 100 which is likely) and only for the first config name found, regardless of how many strings were imported. A normal core .po import would contain 5k+ translations and should affect over a hundred config translation files. Without this fix it only updates one, so the imported translations are almost entirely useless as far as configuration is concerned.

The reason for this is a badly implemented batch, where once the first configuration name was identified for the imported strings, the batch step finishes and never comes back for the rest.

Proposed resolution

1. Instead of not setting a 'finished' value when processing a config key to be updated as a result of translation import, we should set one. Since this is a sub-process of the string handling, we cannot count progress locally here, so opted to "inherit" the status of the "parent" batch process.
2. Add tests ensuring that the configuration files are actually created.
3. Because this made the configuration file updates run as long as half the batch (because now they actually run :D), they are visible for a long time, we need to add messaging to what is going on so the batch is not void of information for 5+ seconds.

Remaining tasks

Review. Commit.

User interface changes

Instead of not telling the user anything about what is going on for 5+ seconds, we now tell them configuration translation updates are happening.

API changes

None.

Original report by @jhodgdon

I was testing #2310735: Advanced search form doesn't translate languages and node type names today for Drupal 8.

I enabled the Language, Interface Translation, and Config Translation modules, added Spanish language, and imported the Drupal 8 alpha-13 translation from localize.drupal.org (it didn't import automatically, I am assuming because I was testing in 8.x not 8.x-alpha-13? Anyway, for whatever reason I had to go to localize.drupal.org and import the translation manually).

The user interface translations are working, after this import. But there is a lot of text in Core that is configuration rather than being t() user interface text.

For example:
- The name of the Language Switcher block that I enabled.
- The names of the Basic Page and Article content types that the Standard profile created.
- The names of the English and Spanish languages that are active on my site.

These items were all part of the PO file I imported. But they are not being displayed -- for instance, if I go to node/add, or use the advanced search form, I do not see the translated names of the Standard Profile content types.

This seems to be a major oversight in the whole idea of being able to import translations from localize.drupal.org. The config text strings are there and have been translated, but they are not being recognized by the config translation module, unless I use the config translation interface and translate them manually.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

jhodgdon’s picture

Gábor Hojtsy’s picture

Status: Active » Closed (duplicate)
Issue tags: +language-ui

@jhogdon: yeah, we are tracking this in the critical release blocker at #2267715: [meta] Drupal.org (websites/infra) blockers to a Drupal 8 RC1, so we should consider closing this in favor of that one IMHO. While the criteria is not very spelled out there, that is the umbrella issue for these things, among other infra blockers. The concrete potx/localize.drupal.org issue for this is at #1933988: Support for Drupal 8 shipped configuration translatables with external dependencies (in effect contrib) and @herom is working on a more limited use case now there which seems to be working out well.

If we want to reopen this, then we should discuss what is our process for tracking external dependencies like this :) I think the overall infra blocker may not express the complexities hidden there (given it is only one issue and does not summarize all the problems to make it findable/searchable) but that is our process so far.

jhodgdon’s picture

Status: Closed (duplicate) » Active

Ummm....

But in this case, the strings being translated *are* already on localize.d.o, and if I look at my User Interface translation admin page on my site, the strings are there and were imported.

The problem I was trying to report is that if I import a PO file with these translations, independent of where the translation file comes from, the translations are not actually being used to translate core shipped configuration on my site. I think this is a separate issue from the infra problem that may not even exist (it does look like strings like the language names and the names of the "standard" profile content types like "Basic page" are somehow getting onto localize.d.o already).

So I'm temporarily reopening this... sorry if I am misunderstanding something? Maybe I need to test again and verify this assertion, but I'm pretty sure that is what is happening, and it's not purely an infra or potx issue.

Gábor Hojtsy’s picture

Status: Active » Needs review
Issue tags: +sprint, +Needs tests
FileSize
661 bytes
1.39 KB

Oh, very good find! So the problem is the following:

- The strings are imported nicely to the database.
- Then the batch moves on to locale_translate_batch_refresh() which is implemented weird.
-> Its "first" step is it unifies the list of strings to be handled
-> Its "second" step is it picks the first (up to) 100 strings and refreshes any JS translations related to them and looks up config names
-> Its "third" step is it updates translations for the config names one by one of those hundred

Then the "third" step runs for all the config names. Once that is exhausted, the "second" step comes back for the next 100 strings, then the "third" step for all those, etc.

The problem is that the "third" step sets no batch finished percentile. Batch steps are invoked with the assumption that they finish, so if we don't set a percentile, then it will assume it is ready. So we have a full list of the affected strings, then a full list of the affected names for the first 100 strings, and then only ever process the very first of those. That is why you see at the end of the import that only one config file is updated. If we return a percentile that makes sense, then further config files are processed and you get 123(!) files updated, at least with the Hungarian translation that is pretty extensive.

It is true that the source strings are mostly not present on the server side, but those that are somehow detected via tests and other means I guess, not the config files per say will this way end up imported.

I attached a minimal and a more elaborate fix. The creation/update of the config translation files takes considerable time, so we need to provide some feedback. The minimal patch makes the batch jump back and forth, so the better fix remembers the string step status and uses that in the name step.

For a test IMHO we need the following:

- a .po file which has a translation for at least 2 config files
- import it
- look at feedback on screen that it updated 2 files (would already fail without the patch)

Gábor Hojtsy’s picture

Title: Translation of core configuration via localize.d.o doesn't work » Configuration update does not work for .po translation import
Gábor Hojtsy’s picture

Looked at writing a test for this. BUT there is already a test for this at LocaleImportFunctionalTest::testConfigPoFile(). That updates two config file translations based on a .po file source, checks the translations were imported in locale and saved into config. So I could not do much else here either. AFAIS the reason the tests pass even though they should not is the timing factor, typical of batches. :/

Some more minor changes to the messages. Now that all config is updated, this is about half the processing time for core translations, so pretty important to have messages from there.

Gábor Hojtsy’s picture

Title: Configuration update does not work for .po translation import » Configuration and Javascript translation not created/updated in .po translation import
Gábor Hojtsy’s picture

It did not let me rest that this may not be due to the batching stuff. And indeed. The test is bogus because it tests loading the translations via the locale manager, which will in fact load the override from the locale database, so not really testing if the override was *saved*. It was testing whether the override was in the locale system, but the prior translation verifications already do that.

If we instead directly look at the translation overrides, the test will fail without the fix because it stops right after updating the first config file. Yay!

I think this is a complete fix/test now. IMHO only needs issue summary update and then good to go :)

The last submitted patch, 8: 2311859-interdiff-and-test.patch, failed testing.

Gábor Hojtsy’s picture

Issue summary: View changes
Issue tags: -Needs issue summary update

Also updated issue summary, please review :)

Gábor Hojtsy’s picture

Issue summary: View changes
herom’s picture

Tested and worked perfectly. Just a small nit.

Value NULL is equal to value 'Névtelen felhasználó'.

The test assert could use a human-friendly message.

Gábor Hojtsy’s picture

I would repeat the dynamic value in the "User friendly" message anyway because we are iterating over multiple items, so the line number will be no indication which one fails. So I don't think it will get better than that. Then its easier to rely on what is already provided IMHO, because less pain same gain :)

herom’s picture

Status: Needs review » Reviewed & tested by the community

ok. the documentation in the test does explain it, so I guess that should be enough.
no more issues.

webchick’s picture

Status: Reviewed & tested by the community » Fixed

Nice catch.

Committed and pushed to 8.x. Thanks!

  • webchick committed c42156c on 8.0.x
    Issue #2311859 by Gábor Hojtsy, jhodgdon: Fixed Configuration and...
Gábor Hojtsy’s picture

Issue tags: -sprint

Yayayayay, thanks.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.