Problem/Motivation
Steps to reproduce
- Fresh install (in english)
- Enable the locale and language modules.
- Add a language (dutch in my case)
- Make the newly added language standard language
- Enable a module which provides default config (book module for example)
Note that this bug only occurs if the default language or the fallback language is not English.
Result
In the UI, you will see some English instead of Dutch on various pages that load configuration.
In the database:
- The base config items in the config table now have a langcode element in them, telling Drupal they are in Dutch not English.
- So when you try to load config, the base config thinks it is Dutch (even though the text is English), and the config overrides (which are still in the config database) are not being loaded to translate the English into Dutch.
Expected result
The English configuration is interpreted as English config, not Dutch, and the config translation overrides are used when viewing the site in Dutch.
Related issues
An issue which does a great job of explaining the issue, including a reference to core issue this was introduced. It also provides proposals: #3150540: Configuration langcode is forced to site default language
This issue seems fairly similar to #2910353: Prevent saving config entities when configuration overrides are applied .
Also, UI text gets overridden when modules are installed. This issue is about config being mangled; the UI text problem is a separate issue: #2408549: Display status message on configuration forms when there are overridden values and solved #2806009: Installing a module causes translations to be overwritten.
From comment #72 on, broader work on the configuration system was proposed instead. This/related work has been taken up (partly via 'duplicate' #3568743: Convert / unify all default configs to the site default language) in #3337864: Introduce a dedicated "Configuration default language" different from "Site default language". That might resolve this issue, or it likely needs to be re-tested after..
Proposed resolution
There was some debugging done; probably the cause is described in #6 / #11 / #14
There was a test-only patch in #5 but it's really for the related UI text issue, not the config text issue.
Remaining tasks
Make a patch, including a test.
User interface changes
Configuration language will not be changed to an inappropriate language.
API changes
None.
Data model changes
None.
| Comment | File | Size | Author |
|---|---|---|---|
| #83 | english_default_configs_updb.txt | 10.27 KB | kriboogh |
Issue fork drupal-2905295
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
geertvd commentedI see the same behaviour when I do the install in a foreign language. During site installation everything is fine, but if you enable a module after that the same thing happens as described in the IS
Comment #3
cilefen commentedThis is critical if reproducible.
Comment #4
geertvd commentedThis patch should fail and prove the issue.
Comment #5
geertvd commentedNow one that isn't empty.. :)
Comment #6
geertvd commentedAfter some debugging I found out
LocaleConfigSubscriber::onConfigSave()is triggered when a module is enabled, the passed config object contains the langcode of the default language rather then falling back on "en".Comment #9
star-szrDiscussed with @xjm @catch @effulgentsia @lauriii and we agreed this is a critical issue (data loss, if I'm not mistaken).
Comment #10
jian he commentedThis issue is annoy. We have to doing this after module install:
vendor/bin/drush langimp --langcode=zh-hans sites/default/files/translations/drupal-8.4.x-zh-hans.po
Comment #11
sutharsan commentedDuplicating isssue found in #2806009: Installing a module causes translations to be overwritten with steps to reproduce by @jhodgdon and debbuging results by @Sutharsan.
Further up the call stack from
LocaleConfigSubscriber::onConfigSave, mentioned by @geertvd, inlocale_system_set_config_langcodes()it is written:This is exactly what happens, it only changes the langcode, but does not change the translatable strings to match the langcode.
Comment #12
jhodgdonNote different steps to reproduce in the other issue, which was filed a year earlier and not really noticed by anyone until now... it's not quite the same issue maybe, but probably identical cause.
Comment #14
ckaotikThanks @Sutharsan for the pointer to
locale_system_set_config_langcodes. That code (seemingly) blindly sets the config language to, but only the langcode, not the actual values. So in my case, I ended up with English config values being stored as
de.I wonder how this should best be handled, because I susoect that it's only an issue if using the
config_translationmodule.Comment #15
sutharsan commentedNo, it is an issue with two or more languages with Language and Locale module only. And having untranslated strings that are used in config. For example exporting config before importing interface translations.
The latter happens easily because Drush does not download/import translations when enabling a module (due to a bug).
Comment #16
mpp commented@Sutharsan, could you add the reference to the drush issue?
Comment #17
sutharsan commented"D8 Run batch process during pm-enable when needed" https://github.com/drush-ops/drush/pull/3445
The equivalent Drush 9 pull-request is already resolved.
Comment #18
jhodgdonSo... It seems like the process discussed in #11 / #14 is only part of the problem. That code is:
https://api.drupal.org/api/drupal/core%21modules%21locale%21locale.modul...
For some unknown reason, it is going through all stored configuration, and updating each config item's langcode to the default site language.
So this would account for configuration being overwritten with English configuration. But it doesn't account for other translations in the UI translation (Locale module) database being overwritten, which also happens.
Comment #19
jhodgdonWhat I mean is, if you look at the Steps to Reproduce in the issue summary of #2806009: Installing a module causes translations to be overwritten, which has been closed as a duplicate of this issue, you can see that regular UI strings like "Content" also have their translations overridden, presumably due to what is happening in
https://api.drupal.org/api/drupal/core%21modules%21locale%21locale.modul...
I'm not sure why that would be, but it doesn't have anything to do with configuration (which is also most likely corrupted).
Comment #20
jhodgdonSo should we have two separate issues, one for config being overridden (which I am not exactly sure is actually happening?? But I think it probably is), and one for UI strings?
Comment #21
jhodgdonOK. There are two different, but both serious, bugs here for sure.
The issue summary here is about configuration translations being overwritten, and I have just confirmed this is true. This is caused by what is described in #6 / #11 / #14. The result in the database is that:
- The base config items in the config table now have a langcode: 'du' element in them.
- So when you try to load config, the base config thinks it is Dutch (even though the text is English), and the config overrides (which are still in the config database) are not being loaded to translate the English into Dutch.
The other issue I reported in #2806009: Installing a module causes translations to be overwritten is kind of similar: if you enable a module, translations for user interface text, which are stored in the locales_source / locales_target tables, are also being overwritten. This is NOT caused by what is described in #6 / #11 / #14 , I think. The patch in #5 here provides a valid test that fails and illustrates this bug. It does not illustrate the config bug that is described here.
So.... I think what we need to do is:
- Reopen the related issue. It's not quite a duplicate.
- Move the test-only patch from #5 here over to that issue, because it's a great test of that problem.
- Write a new test-only patch for this issue, that illustrates the config override problem.
I'm also setting the status of this issue back to Active, and am updating the summary.
Comment #22
jhodgdonHiding the file that belongs on the other issue.
Comment #23
berdirDidn't read through the whole issue yet, but this sounds a lot like a duplicate or at least very much related to #2910353: Prevent saving config entities when configuration overrides are applied
Comment #24
jhodgdonFixing title
Comment #25
catchComment #27
mmrares commentedI have the same problem when a module gets installed but we managed it by doing a config import to restore the correct language (en) until we used ignored configuration using config_ignore module. Since the configuration is ignored, a new config import doesn't do anything to that configuration and the language is stuck to default site language.
And at this point there is no way around that.
Comment #28
jcnventuraOr you run this on drush sqlc after each module install:
Where XX is the langcode of the site's default language. It's an ugly kludge. But I can verify this is working during code review of config commits.
Comment #29
mpp commentedThis also happens when the site default language is English but the default fallback language isn't English.
Comment #30
andypostComment #31
mpp commentedComment #32
jhodgdonI don't think it happens unless the default language or fallback language is not English, so I am not sure about the title change. Adding that clarification to the issue summary.
Comment #34
yonailoAs it is said in #18, I really do not understand why 'locale_system_set_config_langcodes()' has to modify all configuration files languages. If I understand well the code in that function, having no language key is treated the same as being 'en', then language is changed to the "default_langcode" which is not 'en'.
But a configuration file is always in a given language, so what's the point of changing the language this way ? It seems totally useless for me.
Comment #35
johnzzonEncountered this today after installing EU Cookie Compliance on a site with Swedish as default language.
Installing and configuring the module first time is fine. But when you reinstall from existing config, it overwrites my custom config with the Swedish translations from
translations://eu_cookie_compliance-8.x-1.8.sv.po.I expect the strings in my configuration in config/sync to remain. We heavily rely on reinstall for our workflow and it's quite annoying to have the config diff all the time. Can you block importing translations somehow? That'd be a workaround until this is fixed.
Comment #36
mpp commented@jhodgdon, all of our sites (me, @geertvd, @mmrares) have English as the default language, we often do have set Dutch as the default language negotiator (/ => /nl).
Comment #38
stephane aimar commentedHi evrybody,
I got the same issue, here is my site setup:
We manage all common configurations with Features module, all shipped with the common english langcode.
So here is the moment we got the issue, each time we provide a new module installed with a hook_update_n all configuration langcode are falling back at each differents website's default language code.
And there is the moment the translations related issue come too.
I was thinking of a short and smart way to prevent the current default language override. I thaught that there is no reason that shipped configuration language has to be overridden if the shipped langcode exist allready in the website, default or not.
Do you think It's a good way to operate?
Comment #40
rar9 commentedAnother 2 Months have passed since my ticket was closed - https://www.drupal.org/project/drupal/issues/3161643#comment-13765390
Drupal 9.1 is approaching soon. Will this CORE bug be resolved in the coming release?
Will any of the above patches fix my Maintenance bug?
Comment #41
jhodgdonThis kind of comment is not actually helpful or constructive -- complaining will not get this issue fixed sooner. Please read
https://www.drupal.org/docs/develop/issues/issue-procedures-and-etiquett...
and the two pages listed under Related Content.
Comment #42
askibinski commentedComment #44
rar9 commentedIs this patch perhaps helping to fix my Issue?
https://www.drupal.org/project/drupal/issues/3161643
Comment #46
stephencamilo commentedComment #47
simgui8 commented@ stephencamilo: any reason you are closing this?
This is not fixed and IMO it should stay open.
Comment #48
poker10 commentedReverting status, see: #3276540: Block (temporarily?) stephencamilo user for modifying many issues to be "won't fix"
Comment #50
quietone commentedI tested this on 9.5.x and was not able to reproduce this error. I followed the steps given in the Issue Summary, although I installed Italian as the second language. I installed book via drush and tried some other via the UI.
However, when describing the error the IS states
Various pages is too vague. What specific pages are having this problem?
I also looked at the config table in the database and the 'base config' (if I understand that correctly) was always without a langcode and additional 'it' and 'en' versions were added for the translations.
Comment #51
xjmComment #52
andriy khomych commentedI can confirm the above issue.
It will work with the above steps, or in my case, it is caused by the custom importer command which is reimporting every time configs from a specific folder.
Moreover, it ends up with 3 DB records of config collections and later incorrect data in the configuration.
While I fixed incorrect data by explicitly calling locale_system_set_config_langcodes() function at the end of the importer command the overwriting of the configuration still exists. As a workaround I used configuration ignore and ignores those configs.
As an idea, we can allow the existing logic of the locale module but use the translatable attribute and based on it merge prev and imported configurations, and skips the overwriting of some translatable parts. E.g. from here - https://www.drupal.org/docs/drupal-apis/configuration-api/configuration-...
Comment #54
andypostUpdated summary with remaining UI related #2408549: Display status message on configuration forms when there are overridden values
and replaced fixed issue with #2910353: Prevent saving config entities when configuration overrides are applied
PS: #2468767: English config source strings are saved as custom translations in locale in foreign install
Comment #56
shalini_jha commentedHello everyone,
I attempted to reproduce this issue on my local environment, and I successfully replicated it by following these steps:
step 1) Freshly installed Drupal.
step 2) Installed essential multilingual modules such as locale, content_translation, config_translation, and language.
step 3) Added a new language, Dutch, through the admin interface (admin/config/regional/language),and added dutch as a default.
step 4) downloaded Dutch translations from localize.drupal.org.
step 5) Went to /admin/config/regional/translate/import and imported the translation file (.nl.po extension).
step 6) Examined the configuration of a content type (in this case, the Article content type) and confirmed that all values were stored in Dutch (see screenshot: article content type config.png).
step 7) Installed a module with default configurations; for example, I tried the Book module.
Step 8)After installing the Book module, I tested the configuration of the Book content type and noticed that it forcefully overrides the configuration language to Dutch (see screenshot: book content type configuration.png).
Upon investigation, I found that the issue is related to the locale_system_set_config_langcodes function in local.module. The function calls updateDefaultConfigLangcodes, which contains the following code:
This code forcefully sets the configuration language to the default site selected language whenever a module or theme is installed
This behavior seems to be affecting the Book module, where the configuration language code is 'en' as actual language.
I have added a logger in tracing the execution flow and identifying the specific language code being added to the Book module configuration. Notably, the logger revealed that the Book module's configuration is assigned the language code 'en' (see screenshot: logger-message.png).
Comment #57
shalini_jha commentedComment #58
shalini_jha commentedComment #59
shalini_jha commentedI have taken reference from the #38 , as it prevent the config overrides to default language to default config file of module. as i have tested this book module only after applying this patch book module default config value is not overrides with dutch language its in English only.
i have added a patch against 11.x .
Please review.
Comment #61
shalini_jha commentedi have added a MR with test. please review.
Comment #62
smustgrave commentedMR appears to have a test failure.
Added some nitpicky comments to the MR that should be addressed too while tests are updated.
Comment #63
shalini_jha commentedAddressed the mentioned comment, please review.
Comment #64
joachim commentedI wonder if the test be a kernel test rather than a browser test? I'm going to experiment with converting it :)
Comment #65
joachim commented(ignore, false alarm)
Comment #66
joachim commentedI can't work out how the translated node type label is supposed to show when you go to de/admin/structure/types/manage/page -- it doesn't work manually for me at all.
So I can't figure out how the actual test part of the test could be done using a kernel test, as I can't work out what's happening when it goes wrong. How does the translated node type label get loaded to be shown in the form?
In the meantime, I'm streamlining the Functional test with code from the kernel test I started writing.
I'm also removing the mentions of locale_test_translate -- that's not being enabled in this test, so it looks like frankencode comments to me :)
Comment #67
joachim commentedI wonder whether the kernel tests in LocaleConfigSubscriberForeignTest are relevant here -- they seem to be testing the same sort of things.
Comment #68
joachim commentedOk I am now fairly sure that test coverage for this should be added to LocaleConfigSubscriberForeignTest. But I am very confused about how to make that work:
My problem is that I don't understand the underlying architecture of how config is translated to understand what all the test helpers do and what the services like localeConfigManager / LocaleTranslation etc do. I suspect this is the case with most people and that the translation system has a very low bus factor!
Comment #69
joachim commentedI'm digging some more in this.
I'm confused by the fix in the MR being in code that says this:
'prior shipped configuration' means config that was installed from a module *before* the current install operation. So I don't see how that's going to help with the bug, which is about something that goes wrong with config from a module *as it's being installed*.
But then I'm not sure what updateDefaultConfigLangcodes() is trying to do anyway -- what does 'default' mean in this context? In getDefaultConfigLangcode(), 'default' means 'shipped with a module's code in config/install'. But here we're updating config, so it's not the shipped version is it?
Comment #70
shalini_jha commentedWhenever a module with configuration settings is installed, the configuration is consistently overridden by the default language of the site if it is not set to English. The updateDefaultConfigLangcodes function is tasked with updating the language code of the configuration provided by that module. To mitigate this issue, I have implemented a check to verify whether the language code of the current configuration contains 'en' (English) and if 'en' is already included in the available language codes. This check prevents the configuration from being overridden by the default selected language.
Comment #71
joachim commentedIt would be really good if the code comment explained that a bit more, and more precisely, explained the circumstances under which the language code should be updated.
I think this logic could be made clearer:
In the case that $langcode is empty, then it's obviously not in the array. So we only need to check if it's $available_langcodes if it's 'en', AFAICT?
So for example, this would be clearer:
And while I am generally REALLY in favour of splitting up complex conditionals, I'm not sure breaking out the check for $is_available_langcode is good for readability here, and it's not efficient, as it's checking even if $langcode is empty, or not 'en'.
I'm still not sure how we test the patch using the API.
I've installed book when 'fr' is the default language, with and without the fix, and looked at the {config} table.
Without fix:
With fix:
I can see there's a difference, obviously -- only one row rather than two!
But shouldn't I be seeing translated text in there?
I also don't understand what the API reports.
I've got this debug code:
WITH the fix, I get FALSE, TRUE
WITHOUT the fix, I get TRUE, TRUE
I don't understand how BOTH are reporting they're new without the fix.
Comment #72
kriboogh commentedI think the whole config translation system should be simplified and could be solved if:
- "default" config (the yml config files shipped with modules), is always in langcode "en" and contains English strings (obviously).
- All module translations are loaded and imported before the config is imported, so string translation is available.
- For each default English config, a language override is created when installing a site in a different language or when additional languages are added. So a site with languages NL and FR, with FR the default language, has English configs and two overridden configs in Dutch and French.
That way you can switch default site language whenever you want, as much as you want, add languages, delete them. Without default config ever needed to be updated.
Comment #73
kristiaanvandeneyndeHeavy +1 for what @kriboogh said in #72. It's basically what I also said in #2806009-51: Installing a module causes translations to be overwritten. That issue was fixed to address the symptoms at hand, but the root cause was never really addressed for all I can tell.
We require all projects on DO to ship their config in English, so why don't we make use of that and keep the config/sync folder English at all times? This would make everything so much easier from a code perspective too as we then know that the main config folder is always English and the language-specific folders are always for said language.
From my comment above the one I linked:
Comment #74
anybodyI also agree with #72 / #73. Thanks!
Comment #75
catchI was trying to think what the effect of this would be on a monolingual non-English site. I think in that case, if you really, really didn't want to enter English for some config, you could just add it in whatever language (let's say French), the system would store your French text in config/sync as 'English', and then you'd have to translate from 'English' to French (copy and paste the same thing or leave it untranslated).
And even though that would be a bit of a workaround, it seems like it'd be really predictable what would happen, so still OK overall. And then all the other situations would be simplified and make a lot more sense. I only deal with one multilingual site and even that site is not fully multilingual, but with that caveat, +1 from me too.
Comment #76
aleksip+1 for default config always being in English.
Comment #77
kristiaanvandeneyndeSo what would be the next steps to fix this? I've found a handful of issues that all boil down to the same root cause described in #72 / #73. If we want to switch to a way where the main config folder is always EN, which steps can we already take without breaking BC or with a solid update path?
Comment #78
jcnventuraWhat I'd like to see here would be a new entry to the system.site.yml file that would have the following value:
config_langcode: en
This would be set to English if the site is installed in English, or to whatever other language is the site's install language.
At the moment, there would be no provision to change this value in the UI, but if we ever figure out a way to do that, while not messing up a lot of translations, that could change in the future.
Comment #79
kriboogh commentedI think that by allowing the config_langcode to be configurable like that, you're gonna end up in the same mess if someone changes that config setting while the site already has config in the first language.
By having a default config in EN always and have the language config overrides system (which already is in place and working) deal with the translations, everything should basically work out of the box (sort of). The only problem is "fixing" existing websites when we introduce this fixed EN config langcode.
We have done this in a current project. Basically you need to get all default config that was stored in the current default language (for example NL). Change the langcode of the default config to EN. WE now have a EN default config with NL data. So look if a EN config override exists. If there is, swap all data from the EN override into the new EN default. If there isn't an EN override, look for the original EN source strings of the data in the NL data and copy that into the new default EN config. To create a NL override, we can either let core create it (by looking up the translations through locale, or copy the old default translatable NL keys (labels, text,...) from the original NL default config into the new NL override.
Comment #80
catchThis seems sufficiently complex that we might not be able to provide an update hook for it in core.
I'm not sure exactly how (maybe a flag in settings or a container parameter), but can we look at forcing English for default config on new installs, which would fix the bug for new sites, and then we could at least provide instructions (and maybe a contrib module) to help existing sites convert based on the above?
Comment #81
kristiaanvandeneyndeIf we force it on new sites only, then the code base also needs to take care of both scenarios: old and new. I can already sense the amount of bug reports we'd be getting then. If we provide an update hook, then the code base only needs to take the new scenario into account.
@kriboogh would you mind sharing your update code here? Perhaps it might be simpler than it sounds.
Comment #82
kriboogh commented@kristiaanvandeneynde sure, it's not pretty though :D
We had to do some specific stuff with system.site config, which might have been only related to our project. But I'll keep it in.
Disclaimer: To anyone else reading this, this is specific project code given as an example, do not run this blindly on your production site !!!
Comment #83
kriboogh commentedFollow up: We were still having issues on a site by this. The problem was the site has only one language (NL) which is also the default language. I'm now trying to also have English installed and set EN as the default language. This will force config to be EN. I updated the updb hook above to swap and fix config translations (see attached file). The only downside now is that what used to be a single language website, now has two languages (NL and EN). We installed disable_language module to hide EN in the site. Also this module needs a patch to also hide the disabled language in translation overview.
Again disclaimer: To anyone else reading this, this is specific project code given as an example, do not run this blindly on your production site !!!
Comment #84
nmudgal commentedThanks to everyone for all the work on this thread. I’ve read through the discussion, and it seems like setting English as the default
langcodefor configurations could be a straightforward way to avoid language conflicts during module installs. Here’s a quick summary and a few ideas on how we might combine some of the suggestions:langcode: en):Setting
langcodeto English across all module configurations would help avoid conflicts and keep things consistent, especially on multilingual sites. This way, English serves as the base, with other languages applied as overrides as needed.For older sites, a contrib module could help automate the migration to this new standard. It could convert non-English
langcodevalues to English and move existing language settings into overrides. This module could also support language overrides and protect configurations during updates.config_ignore):While
config_ignoreand similar modules are useful right now, moving to a standardizedlangcodeapproach would eventually make these workarounds unnecessary, simplifying language management.Comment #87
xjmAdding issue triage credit as per #9.
Comment #89
roderikAdded some references to the issue summary.
Larger work (pertaining to the discussion in #72 onward) is currently being done in #3337864: Introduce a dedicated "Configuration default language" different from "Site default language". I'm not sure if this issue (which is more specific than that) should be 'duplicate' or 'postponed (re-test and/or re-check the automated test when that lands)'. So I'm not touching status.
The MR has a merge conflict since #3427564: Require `langcode: …` only for simple config that contains translatable values landed. I don't know if adding the included test to Core will still be useful when it's rebased, but... that probably also needs adjustment because it uses the not-in-Core-anymore 'book' module.