Problem/Motivation
As you create configuration on a Drupal site, it may be created in one of 4 languages:
- Current request language
- Configuration created on the UI that does not have a language selector (eg. content types, block types, etc.). This means if an admin browses the site in a foreign language, if they create a new content type, it will be created in that foreign language.
- Configuration created through config actions (eg. in recipes). This means that users of project browser will have configuration created in recipes with config actions created in the language they happened to use in their browser.
- Language explicitly selected on the UI
- Very few config entities have an explicit language selector (in core: menus, views, taxonomy vocabularies, date formats). This means that even if your request language is X, you can still create that config in Y, Z, etc. languages using the selector. This is the only way to create in "Not specified" and "Not applicable", which are two options offered.
- Configuration language directly in a YAML file, even if the language does not exist on the site at the time(!) (or if missing assumed as English)
- Configuration shipped modules, when the module is installed if locale module is not yet installed, regardless of the site default language!. This means that if your site default is German but locale module is not installed, module config will still be imported in English. Even if the English language was deleted earlier.
- Configuration shipped in recipes, when the recipe is applied regardless of the site default language or if locale is installed or not!. Even if locale module is installed recipe shipped config is always installed in the language it is in, regardless of whether that language is configured on the site or not.
- Site default language
- Configuration created through extension installation if locale module was previously installed already on the site. When a module is installed, its config is created in the site default language at the time.
So you already have config in various languages. Then Drupal tries to keep the config language up to date but that only ever happens with extension installed config. But!
- Extension config language update only happens when extensions get installed. If you modify the site default language, you need to THEN install an extension for the config of all previous extensions too to get updated.
- The update only ever happens if the config was still English and was now switched from English to something else. If it was not English, than a site default language modification does not result in config language modification even later when extensions get installed.
- Once again this modification only applies to extension installed config, not manually created config or recipe installed config or config action created config.
See locale_modules_installed() and core/modules/locale/src/LocaleConfigManager.php for this logic.
Steps to reproduce
- Install Drupal standard in English
- Enable the language module and add German on
admin/config/regional/language. - Create a new content type. This will be English. Existing installed config will be English.
- Create a new menu. Note that you can pick English or German for this menu. Create it in German. Config export will show the config entity got created as German.
- Go back to create a new content type. Add a language prefix (
/defor German) to the URL and load that page. While it is not apparent in this UI, this will create the content type in German. Verify with config export. - Add another language (for example French) on
admin/config/regional/language - Change the default language to French that you just added.
- Install Media module. The installed config will be English even though your site default is now French.
Another variant of the steps to use interface translation from the start for different dramatic effect:
- Install Drupal standard in English
- Enable the language module and Interface translation modules. Add French on
admin/config/regional/language. - Create a content type. This will be English. Verify with config export.
- Change the default language to French that you just added.
- Create a content type. This will be French as the URL has
/frin it because modifying the site default to French brought you there. Verify with config export. - Note that all of your config other than the French menu is still English. Verify with config export.
- Now install Media module. A big config update will happen that updates extension installed config to be French. Your English content type remains English. Verify with config export.
- Now switch the site default language to German. Nothing happens to config. Verify with config export.
- Install search module. This will still NOT rewrite your existing config to be German but search module related config will be installed in German now. Verify with config export.
- Now you have English, French and German config, most config is French.
Also, if you then switch site default language back to English, the configuration language for existing config also never switches back, not even when you install more modules. (This is true at any point after the site default language was once a foreign language).
Also, if you delete either of the 3 languages, which can be done without issue, you are left with configuration in that language Drupal does not know about.
Proposed resolution
The site's configured default language (system.site:default_langcode) is the single source of truth for what langcode new or existing configuration without a langcode gets. A new event lets every installed subsystem contribute to that question, with the most specific answer winning.
The ConfigLangcodeResolveEvent event
Drupal\Core\Config\ConfigLangcodeResolveEvent is dispatched to determine a configuration's langcode. It is only dispatched when the system determines it is necessary. This depends on whether the configuration has a langcode already or if config is being created by the ConfigInstaller. It carries no information about the object being saved. A subscriber calls setLangcode(). The first call wins and stops propagation, so a higher-priority, more specific subscriber takes precedence over a more general one, without inspecting what any other subscriber decided. This allows contrib or custom code to completely override what Drupal core does. getLangcode() never returns null. Core always registers a subscriber that answers unconditionally.
Dispatched by:
Config::save(), for simple configuration that is translatable and has no langcode.ConfigEntityStorage::doCreate(), for an entity created without an explicit langcode.ConfigInstaller::createConfiguration(), once per call to createConfiguration() for the default collection, for extension-shipped configuration with 'en' or no langcode.
How this works for Drupal core
During site installation: ConfigLangcodeInstallerSubscriber responds with the language chosen for that installation - at the highest priority, so shipped configuration installs in the chosen language before system.site itself has been set.
After site installation: ConfigLangcodeSiteDefaultSubscriber resolves to system.site:default_langcode. Unless the language module has been installed this will be 'en'. If the language module has been installed it will be whatever the user has selected as the site default language.
ConfigInstaller::createConfiguration() treats a missing langcode as English by default and resolves it through the same event. An explicit, non-English langcode is left alone. Recipes install configuration through the same code path (RecipeConfigInstaller extends ConfigInstaller), so this applies equally to recipe-shipped configuration.
What happens when the site's default langcode is changed
This can only occur if the language module is installed. The language module triggers a batch controlled by the Drupal\language\DefaultLangcodeConfigUpdater service. It updates the default configuration langcodes and preserves translations in config overrides for the new langcode before it removes them. It also adds config override for the old default language.
If locale is installed, it does two things during the default langcode change. First, LocaleConfigSubscriber ignores config and override saves while DefaultLangcodeConfigUpdater::isReconciling() returns TRUE. This stops those saves from being stored in locale as customized translations. Second, locale subscribes to DefaultLangcodeConfigReconcileEvent. The event is dispatched before each config object is saved. Locale translates the data into the new default language. Values translated by the old default, but not by the new default, go back to the English source. The config is saved once. So installing German and then adding French and making it the default language gives the same result as installing French and then adding German.
Previous solution
We tried to address this problem by adding a new setting for a default config langcode but this hit snags because we still had to deal with the site's default langcode changing. See the hidden MR for how this approach worked https://git.drupalcode.org/project/drupal/-/merge_requests/16280
Remaining tasks
- Code review
- Create any necessary change records
User interface changes
New batch triggered when changing the default langcode
API changes
- New:
ConfigLangcodeResolveEventevent. It is dispatched when a config object's language is unknown or when it is English and the config is being created by the ConfigInstaller. - New:
Drupal\language\DefaultLangcodeConfigUpdaterservice - creates a batch to update config when the default langcode changes. This uses the new interfaceDrupal\language\DefaultLangcodeConfigUpdaterInterface - New:
Drupal\language\Event\DefaultLangcodeConfigReconcileEvent. It is dispatched before each config object is saved. Subscribers can change the data with::setData()or skip the config with::skip(). - New:
Drupal\locale\LocaleConfigManager::getDataForDefaultLangcodeChange(). Returns config data for the new default langcode, without saving. Keys still in the old default language fall back to English. Locale translations for the new langcode are applied on top. - Changed:
LocaleConfigSubscriber::updateLocaleStorage()and::processTranslatableData()each take a new$original_configparameter. - Changed:
LocaleConfigSubscriber::__construct()takes new$configInstallerand$defaultLangcodeConfigUpdaterparameters. Calling it without them is deprecated. - Deprecated, no replacement:
LocaleConfigManager::updateDefaultConfigLangcodes(). - Deprecated, no replacement:
LocaleConfigBatch::batchUpdateDefaultConfigLangcodes()and the$update_default_config_langcodesparameter ofLocaleConfigBatch::buildBatch().
Data model changes
Config::save()adds a langcode key when config becomes translatable and none is present, and removes langcode when config stops being translatable.system_post_update_reconcile_translatable_config_langcodeis a new update that ensures simple configuration has a langcode if it is translatable and does not have a langcode if it is not.
AI Usage
alexpott used AI to review code, propose solutions, write tests, implement ideas and help write the issue summary. alexpott made every commit, and reviewed and tested each change before committing it. Design decisions are alexpott's.
Release notes snippet
Configuration installed by modules, themes and recipes now gets the site's default language as its langcode. This happens with or without the Interface Translation module. Configuration entities created without an explicit language, for example a new content type, also get the site's default language instead of the current interface language.
When the site's default language is changed on the language overview page, a batch updates all configuration in the old default language to the new default language. The previous values are stored as translations in the old language.
This batch does not run when the default language is changed outside a form, for example with Drush.
| Comment | File | Size | Author |
|---|
Issue fork drupal-3337864
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
tsotoodeh commentedThe same issue observed in 9.5.x, it is a core design issue, I opened a multilingual support issue:
Multilingual support of strings in second language website
I Tried using configuration update manager module to adjust langcode from default language to primary language, time consuming process! Although the content is translated but it would not shown correctly in second language. Same issue here!
Comment #3
jose reyero commentedComment #4
jose reyero commentedUpdated task description to account for the fact that when you switch default language back to English, nothing happens, existing config language is never updated.
I guess we need to make locale_config_batch_set_config_langcodes() behave the same for all languages and not to make an exception if default language is "en", or maybe keep track of the last language set for configuration, and whether the default langcode has been changed or not...
Comment #5
tsotoodeh commentedWell, as far as I can see the Drupal multilingual efforts has come a long way from the time that i18n or internationalization and localization modules was once were.
To be truly multilingual the configuration translation and interface translation should be taken a second thought.
As you said, the the language switch should be respected both in terms of configuration of the elements of the site and also the contents. An architectural plan should be devised for this task. Currently the configuration in consistency after switching the language from default language are many to account.
Comment #7
almador commentedI don't know if this is the right topic to share my issue, but it's also related to switching the original language.
What I did - while upgrading Drupal 8 to Drupal 9 by mistake I changed my default language from English to Spanish. When I realized this (after the upgrade) I switched my default language back to English from this page:
admin/config/regional/languageBut now, when I'm trying to edit any of my translations related things it still says "Español (original)".
Is there any way to switch every translation option back to English (original)?
Comment #8
gábor hojtsyMade the title more specific.
Comment #9
gábor hojtsyLooking at why this happens. When a module is installed after the default language is changed:
All in all I think the logic protects the existing language changes in active config if they were made prior to the default language change. So if you already have a Swedish site let's say, you switch to Finnish, the active config items that were previously Swedish will remain Swedish. Newly installed modules will get their config in Finnish in active config, so you end up with a mix of config items with Swedish and Finnish. But the integrity of your config is maintained.
If you built out a part of the website in English and then switch the default langcode, this indeed ends up doing more dramatic changes later because English is considered special. (When the default language is set back to English, I think the data protection logic does what it was supposed to do).
Now as to what happens when default language is changed, I'll post another comment, this is already quite long :D
Comment #10
gábor hojtsyDefault language change on the UI is implemented in LanguageListBuilder::submitForm(), and does this essentially:
Language module reacts on the config save of the default langcode in its ConfigSubscriber but that of course is not related to interface translation.
Finally locale has its own LocaleConfigSubscriber but that does not consider a default language code change as something to care about.
Comment #11
gábor hojtsyI'm a bit torn as to how to resolve this. For one, we want to make default language changing as easy as possible. The intent of the code is to consider config similar to content, where the existing language and overrides of configs is kept, except in the English special case that it assumes belongs to the default config that needs changing. There is no such logic for content entities either to swap their default language and translations when a site default language changes.
At the same time, users likely expect to edit configuration in the site's default language by default, which the current logic attempts to ensure, at least as long as that default is English and after the first transition from English to non-English. It does not maintain that user experience once you want to make the second language transition, whether back to English or to something else. It does leave your config intact but it does not maintain the assumed user experience. Also, if the first language transition happens later in the site's life, a massive config change happens then. That in itself is expected, but not when you install a new module :D So the problems are:
Comment #12
drupaldope commentedIt seems that someone has been changing some things in how languages work.
I am now unable to rename a certain language (D 10.2.4).
I wanted to name English en and German de.
The YML configuration is correct, but when the page is shown in the default original language German, it will just say "Deutsch" no matter what.
I guess the new code changes fail to retrieve and display the correct label.
Comment #13
drupaldope commentedActually I had it working correctly on a site where I did the following :
Installed the site in English
Added German
Added British English
Deleted English
works.
Comment #14
drupaldope commentedthere is actually more to this issue.
languages need a translatable label, which would resolve the present issue and a dozen more related issues I have found today when going down this rabbit hole.
BUT
languages also need editable labels that will be displayed in translations, verbose translations and abbreviated translations, URLs ...
So for example, a site could have installed British English and it would display the following way :
"en-gb" (non translatable) in any technical-related stuff such as hreflang, html, etc.
"British English" (translatable) in any configuration dialog in the admin interface, translation forms, etc.
"en" (translatable) as a label for language switcher links
and potentially, the site owner would want something else (translatable) to be displayed in the URL, such as /eng/ or /english/ or ... whatever
so the short ISO language code would be used by Drupal to identify the language and source translations, and would be untranslatable.
and then there should be 3 translatable labels on top.
Comment #15
drupaldope commentedI can now also confirm the workaround to correct this issue on running sites.
First, install an additional language.
Then change the language of all localized content to that new language you just installed.
That may be nodes, blocks, media, redirects and URLs, etc.
Then delete the language of which you can't change the name (be careful, this will delete all content you forgot to switch to the new language).
Purge the cache and then re-install the language you just deleted.
Edit the name of that language to what you wanted and then change the contents' language back to what it was.
So it seems like something screws up the language settings when installing the site or when declaring a language default.
Comment #16
drupaldope commentednever mind the workaround, one of the language labels reverted back to original after doing something else on the site.
Comment #17
claudiu.cristeaI don't know whether is related to this. I set the default language to other than English. Then I've exported the config (drush cex). Reinstalled the site from config but the English language is gone. Checked the config sync directory, I can see thelanguage.entity.en.ymlfile. Checked the{config}table, nolanguage.entity.enat all. It just vanished.EDIT: Ignore this comment. I just discovered that I need to set
keep_english: truein the installed profile. Maybe this deserves to be better documented, even by showing a warning message to the site builder when they are changing the default language?EDIT2: I have created #3545306: Installation from config removes English if the installed profile does not have keep_english for this issue
Comment #18
anybodyNice, just found this and I think it's very similar to this one: #3568743: Convert / unify all default configs to the site default language (which has a possible solution).
For the future I'd vote to always store the default config in English by design, so we can't have such cases any more and treat any other language as translations in their respective collection / folder!
That will also simplify importing (typically English) config from modules etc...
Comment #20
dqdAdded to the "Refine settings, flexibility and coherences between translatable areas" outline on git.drupalcode.org. Thoughts appreciated.
Comment #21
ressaConnecting related issue and adding "Update Install a language doc page" in Issue Summary as a task.
Comment #22
gábor hojtsy@anybody: in some European studies (eg a study of 100k websites of in the EU, 4-5k sites per country) it was found that 80% of the sites are not English single language. Why would they need to create content types, views, etc. in English? That would be bad :) Drupal already takes some effort to keep the default config in your language, we need to fix the missing use cases.
I wrote out some plans above 2 years ago in #9, #10 and #11. Since then I realized we probably don't need to deal with this in config import as that would assume you are already deploying an updated set of config. So we need to do this on the form. Since that already happens later when you install a module, we should at first bring the same behaviour forward to the default language change IMHO as a first step.
Comment #23
anybody@gábor hojtsy thanks. Yes I understand that. My recurring issue in the role of a site builder is, that again and again I have to think about in which language I'm acting and how I should name even the machine names.
In 60% of our cases the projects are in German, some of them are ONLY in German and some of them are translatable into other languages. But we love the configs to be consistently in English as base language..
From our perspective it would be more appropriate to be able to rely on English as base language and use translation for anything else.
You know a lot more about the internal concepts and I think you discussed this 100000x more than me and have good reasons. So this comment is not meant to tell what should be done, just to let you know about the background and our daily pain with the "mix" we're experiencing.
Still my solution would not save us from having clients that then create entity types with German names ;) So there's no perfect. Most importantly the bug needs to be fixed, as you wrote. For the other things we'll still have to find a better / best practice.
Thank you so much! :)
Comment #24
gábor hojtsyI'm working on implementing this now :)
Comment #25
gábor hojtsy@andybody: for your base English use case you can install the site in English and keep site default in English, why not do that?
Comment #27
gábor hojtsyWhile I pushed a complete lifecycle solution above in an MR, I decided that this will not be the best route :D Instead I think there is a lot of merit in trying to cater more for what @anybody is saying about consistent source language AND decouple the config default language from site default after that (so changing config default language is a conscious choice separate from site default). For the first part of this, I opened #3608533: Introduce "Configuration language" settings page to control langcode visibility. Once that lands, I think we can return here and introduce the Config default language and adjust the code to use that when dealing with config.
I'll work on #3608533: Introduce "Configuration language" settings page to control langcode visibility tomorrow, marking this postponed on that. I already posted the rationale and high level plan there though :)
Comment #28
anybody@gábor hojtsy re #25 we do that in many cases, but it doesn't feel perfect because then you have the English language also in many Content (related) areas. It just doesn't feel natural and correct to me. And you still have to be very careful to not accidentally use the wrong language somewhere.
#27 sounds like you came to a similar conclusion. Thank you so much for all your work put into this.
Comment #29
gábor hojtsy@anybody: yeah I did came to the conclusion that we need to decouple the "Configuration default language" from the "Site default language", which is another way to approach "enforce a config language". I also came to the conclusion that the few core changeable config languages are in the way sort of :D So I created #3608533: Introduce "Configuration language" settings page to control langcode visibility and proposed a solution there. That introduces explicit intent about language of those config, so after that as we want to I think introduce a dedicated "Configuration default language" here, we can rely on that intent / configuration to not change things that site owner explicitly wanted to be in other languages but change the rest of the config as needed. (Also to not change config language on site default language change ever). Let's reposition this issue for that, but we need to resolve #3608533: Introduce "Configuration language" settings page to control langcode visibility so we have data on the intent of the config that is currently potentially in other languages. That also helps enforce a consistent language if you don't configure those entities to be able to pick a language which is the new core default as proposed there.
Please review that one to push this cause forward :)
Comment #30
gábor hojtsyCreated a parent issue to group related problems and show how this leads to resolving config language issues once and for all hopefully :D
Comment #31
gábor hojtsyRewrote the issue summary. Will of course need more expansion but this is a good start IMHO.
Comment #32
gábor hojtsyFix table HTML.
Comment #33
gábor hojtsyFix one more HTML issue and reorder the events in the HTML to make it easier to follow.
Comment #34
gábor hojtsyMore elaborate items in the table to point out the problems better.
Comment #35
gábor hojtsyFix list markup.
Comment #36
gábor hojtsyComment #37
gábor hojtsyComment #38
gábor hojtsyMassive update to issue summary based on recent discoveries. I found 3 new problems since the above, so getting more convinced that we want to have one unified "Configuration default language".
Comment #39
gábor hojtsyClarified a few items.
Comment #42
gábor hojtsyComment #43
roderikThank you for this :-)
I was tracing a situation that I wasn't sure is 'my' bug or not, and I haven't seen it mentioned anywhere:
The samlauth module starts out with very minimal installed config, with no label values and no language code. But there are labels in the config object, and as soon as you set a label value... Configuration Inspector reports a missing language code. #3611776: samlauth.authentication config object must specify a language code
It felt like either I was missing how exactly I should solve this... or the only way to solve that reliably would be adding code into
Config::save(). (Which felt like too big a thing to propose for someone without intricate knowledge of the config system...)Luckily, I read your latest MR's code and it does just that! So... I believe I bumped into yet another issue that is solved by your MR!
Comment #44
gábor hojtsy@roderik: I have a targeted issue for that problem at #3600904: Shipped configuration entities that do not have translatable elements are not set to the site default langcode as that is also reproducible with core only :)
Comment #45
gábor hojtsyI've been experimenting how far this needs to be stretched in the past few days and ended up building https://www.drupal.org/project/config_language_lock which has about 900 lines of PHP code and almost 2x the amount of that in docs and 4x the amount that in tests to prove how core works vs how the module works. People following this issue would be welcome to test that (in dev environments for now, it is very early stages). That said the docs and tests and SUPER thorough, so I'm fairly confident it is fine. Read the very extensive docs at https://project.pages.drupalcode.org/config_language_lock/
Comment #46
roderikD'oh! I failed to read this issue's meta parent and its children.
Reviewed #3600904: Shipped configuration entities that do not have translatable elements are not set to the site default langcode. The issue I stumbled into is not currently solved by it though (because it's simple config), so curious for the next step.
Comment #47
alexpottI changed the MR to result in less changes by ensuring we're only adding a langcode to config with translatable parts. I've also enforced the 'en' default in code which makes the change less disruptive to kernel testing where the system module might not be installed.
Comment #48
catchIt would be useful for installer refactoring if we were able to move the default language configuration out of system module to core/
This is because if you try to install system module along with any other modules at the same time, it's impossible to set the system default language in between system module being installed and the other modules getting installed, and that information comes from the request.
Currently working around this in #3614153: Install system module alongside other modules in the installer where you can see having the config outside system would make it a lot easier.
Comment #49
alexpott@catch sure we can introduce a core.language and open a follow-up to move the site default langcode there too. Sometimes I wish we could move this to container parameters instead of config.
We also need to prevent the language that is the default config langcode from being deleted.
Comment #50
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".
This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #51
alexpottI spoke with @Gábor Hojtsy about this issue. Two things came out of that conversation:
1. A new blocking issue is needed:
2. Test coverage and scope for this issue:
Comment #52
gábor hojtsyOther than "changing langcode", there is also applying langcode for config actions (can happen through ECA for example too?) and UI creation of config, which should use the same config language by default. But not general write of all of config as we agreed to keep the existing flexibility if people really want to use it.
Comment #53
alexpottSo I worked on improving the current approach - moving away form core.language as we only need the config when the language module was installed and introducing an event architecture to resolve the config langcode - this allows us to do different things when the installer is in play or the language module and also allows us to know when we;re resolving a langcode during an extension install and when it is regular runtime. I also added an exception when you delete the language that is the config default. And this is where I hit a big block - we have a number of tests that expect to be able to change a site default language and delete the old one - for example \Drupal\Tests\config\Functional\ConfigLanguageOverrideWebTest::testSiteNameTranslation.
This has led me to the conclusion that we need to allow the config default langcode to be changed. And once we think this then we actually do not need to introduce a dedicated configuration default language but we should use the site default langcode and support (and test extensively changing it). I've done this and in my next comment I'll detail all the changes to support this. It builds off much of the work done on this issue and think we should re-frame this issue to do take this approach. I've updated the existing MR https://git.drupalcode.org/project/drupal/-/merge_requests/16280 0 will all my work to get to this point so if we pivot again back to a separate config default langcode setting then we're nearer completing.
Comment #55
alexpottI've created a new MR https://git.drupalcode.org/project/drupal/-/merge_requests/16913 that does not introduce a new config default langcode - but rather uses the system.site default_langcode and fixes and tests the behaviours around changing it.
Here's a summary of the changes:
ConfigEvents::RESOLVE_LANGCODE, replacing ad hoc default-langcode logic across Config::save(), ConfigInstaller, and config entity creation.Config::save()only adds or removes a langcode when a config object's translatability changes. It never overwrites an existing langcode. This fixes explicit English configuration being rewritten to the site default on every save. Config entities are always considered translatable.system_install()now sets system.site:default_langcode from the language chosen during installation.ConfigInstallerrelabels extension-shipped configuration with no langcode, or en, to the site's default. It rejects configuration shipped with an unsupported explicit langcode. Without language module installed, only English is supported.LocaleConfigManager::updateDefaultConfigLangcodes()or its batch operation now triggers a deprecation notice; there is no replacement.Comment #56
alexpottComment #57
alexpott@Gábor Hojtsy pointed me towards https://git.drupalcode.org/project/config_language_lock/-/work_items/361... and initially I thought that this puts a big hole in my new approach in #55 but after working with Claude on the issue I've realised that actually if change the
ConfigEvents::RESOLVE_LANGCODEevent so that only the first resolved langcode wins we actually provide a pretty easy way for https://www.drupal.org/project/config_language_lock to support this exact use-case as long as we file a follow-up to improve config entity editting when the site is multi-lingual.Comment #58
svendecabooterWe have a site in a messed up configuration-language state, which this issue is aiming to resolve.
I'm aiming for the following setup: site default language Dutch, English enabled but not the default, and all configuration authored and shipped as
langcode: en.Since we use recipes for adding extra features regularly, I tested this MR against that scenario to see how a recipe carrying
enconfig behaves when the site language isnl.The result is that a recipe delivers configuration by two routes that now disagree with each other. Setting the site default to
nland applying a recipe that (a) shipslangcode: enconfig in its own config/ directory and (b) installs a module shippinglangcode: enconfig:- the recipe's own config/ file resolves to
nl- correct- the config shipped by the module the recipe installed stays
en- the same module installed directly, not through a recipe, resolves to
nlSo the same module's configuration ends up in a different language depending on whether a recipe installed it or not.
The cause looks like the syncing gate.
RecipeRunner::installModules()andinstallTheme()wrap the install inconfig.installer->setSyncing(TRUE), and the resolve inConfigInstaller::createConfiguration()sits inside, so it never runs for those. The recipe's own config goes through a separately constructed RecipeConfigInstaller that is not syncing, so it does resolve.
Comment #59
alexpott@svendecabooter firstly thanks for taking the time out to test code on this issue. Amazing!
I would like to clarify something: It would say this issue is aiming to help resolve. I think your use-case of having the site default and config default langcodes as different is special and does not necessarily need to be fixed here.
Your feedback on Recipes and sync status is very helpful - I agree that we need to fix that. I'm not sure whether this issue should fix it (it is currently broken after all) or attempt to resolve that here to. I'm unsure if we need to resolve #3613607: Add RecipeRunner::isApplying() to determine if a recipe is being applied, without confusing it with a config sync before attempting this.
Comment #60
alexpott@svendecabooter one thing the MR is introducing is an event that will allow your site to set a default config langcode to something other than the site's default langcode. Maybe, if we manage to land this issue then that's what https://www.drupal.org/project/config_language_lock can provide.
Comment #61
alexpottOkay I've merged the solution from #3613607: Add RecipeRunner::isApplying() to determine if a recipe is being applied, without confusing it with a config sync into the MR and now recipes should correctly using the default langcode too. This issue is blocked on #3613607 but I want to continue working on this and hopefully that one will land quickly.
Comment #62
alexpottI've added recipe testing to the new MR. @svendecabooter if you could repeat your tests that'd be amazing. Note your site will need to implement a subscriber to ConfigLangcodeResolveEvent and ensure that it has a high priorty and set the langcode to 'en'.
My next task is to update the issue summary and title to properly reflect the new MR and what is being changed by the new MR.
Comment #63
alexpottI've updated the issue summary with the help of an LLM to reflect the latest MR. I'm going to hide the old MR that introduced a new setting for default config language as I no longer think that is the best way to fix the problems outlined in the issue summary.
Next I'm going to add an 11.x branch as it much easier to reproduce the problems in 11.x as locale can download translations from drupal.org there.
Comment #65
alexpottComment #66
alexpottComment #68
alexpottComment #69
quietone commentedTrying for a concise title. that is still correct. For myself, less wordy title are easier to skim both in the issue queue and it git logs.
Comment #70
alexpottComment #71
svendecabooterThanks for merging in #3613607: Add RecipeRunner::isApplying() to determine if a recipe is being applied, without confusing it with a config sync and the further updates.
I was indeed aware of the config_language_lock module, which is how I ended up in this issue...
I have rerun my earlier test on the latest code state, and now the recipe scenario also provides the same consistent result as the others, so that appears to have fixed that issue.
So if I understand the current state of this MR correctly, it fixes (almost) all of the issues that config_language_lock describes, and "locks" the language always to the default site language (with no configuration option to choose another language).
The reason config_language_lock would then exist is to allow more advanced use cases like the one I described, where the site default language would be Dutch for example, but we'd want all config to be stored as English. Then we'd install that module and configure it to lock to English - it would do so be reacting to the new event that core submits.
One thing that remains unclear to me is how this would behave in the following scenario that has bit us a few times already in the past:
A multilingual site gets built say in English as default language. Mid-way the project the client decides they want the default language to be Dutch instead. The DefaultLangcodeConfigUpdater would then still change all config langcodes from "en" to "nl" automatically? Also when a module like config_language_lock would be configured to react to the event, as described above?
If so, that might still get us in a situation where a bunch of English strings are suddenly marked as being "Dutch", confusing website translators...
Comment #72
alexpott@svendecabooter thanks for testing - it's great to know that the recipe system is now consistent - nice!.
Yes you are correctly that config_language_lock will be able to drop a lot of code once this lands and become something that solves your advanced case. And yes it will need to alter the behaviour of events that respond to system default langcode change. Actually this brings up a great point about the language config subscriber - we need to make this easier to override. I'm going to move it into it's own event so it is easier for config_language_lock to override this event and replace it with it's own logic about when to fire.
To answer your last question, if you change the default language from english to dutch this MR now ensures that all the current config values become english translations and Dutch translations are applied to the default config. The scenarios about changing the default langcode from the issue summary now behave as expected. This is quite easy to test with the 11.x branch where locale will be able to get translations from localise.drupal.org. And english strings should not be marked as Dutch.
Comment #73
alexpottI've separated out the event that responds to a change in system.site:default_langcode so that it is easy for contrib and custom.
Comment #74
alexpottComment #75
alexpottI've removed the skipping on unknown langcode from the installer - it's a different bugfix that can be handled in a follow-up. I'll create the follow-up later. I've updated the issue summary to match this change.
Comment #76
alexpottComment #77
alexpottComment #78
gábor hojtsyI think this is almost there! Left a lot of review comments :) Also the issue summary needs updating for recent MR updates.
Comment #79
alexpottI'm updating the issue summary to make it more correct. I've also removed the test coverage section as it is hard to maintain.
Comment #80
alexpottImprove the API documentation.
Comment #81
alexpottComment #82
alexpottI've tried to update the issue summary to make it clearer.
Comment #83
alexpottComment #84
alexpottComment #85
alexpottUpdated the API change section with the details of the change I've just made to support the case where there are no overrides or a partial override when you change the default langcode of a site.
Comment #87
alexpottI've closed the 11.x MR as now that 12.0.x exist we have translations for Drupal 12. Nice.
Comment #88
alexpott#87 was premature you still don't get a full locale download on 12.x or main... I'll bring back the 11.x MR when I have time as it make testing this much easier.
Comment #89
alexpottCreated a very small self-contained issue to improve \Drupal\Core\Config\Schema\ArrayElement::hasTranslatableElements() performance - see #3627544: \Drupal\Core\Config\Schema\ArrayElement::hasTranslatableElements() creates unnecessary objects
Comment #91
alexpottUpdate issue summary for the latest changes - I've managed to remove one of the events this issue was adding - LanguageDefaultLangcodeChangeEvent - locale's requirements to add to what language does all happen in the decorator.
Comment #92
alexpottComment #93
alexpottThis is ready for another round of review! Quite a few changes since the last round.
Comment #94
nicxvan commentedI got part of the way through reviewing this again, one question that I have after reading through this, is this changing how simple configuration works related to translation or is this just fixing the mixups that happen.
It's not clear to me if this is fixing a bug, adding new capabilities or both. I suspect it's the latter.
Comment #95
alexpott@nicxvan in my mind this issue is 100% bug fixing. We already provide the capability to change your default language in core. This issue fixes bugs that occur when you do that. In order to fix the bug we've had to introduce new API around changing langcodes in config because that currently is in locale BUT you don't need locale installed to change your default language - you only need the language module. So code for managing that has moved there. Similarly, in order to set the config langcode at install time we've needed to add a new API as again it was locale that was changing configuration langcodes after module install.
Comment #96
alexpottJust run an interesting test on 11.x (because you get translation files from l.d.o) ...
French translations are applied to the config in the default config supplied y modules and en overrides are create as expected. It works great. Note this would work on HEAD but my point is that we've not broken anything. There are somethings that are different. With this MR the configuration language is changed when set french to be the default language rather by a batch triggered by the locale module install.
Comment #97
gábor hojtsyRe-reviewed my previous review notes. What I found is the early return in
LocaleConfigSubscriber::onConfigSave()skips new configuration with_core.default_config_hash.I worked with my LLM to propose a test for this scenario that would fail with your MR.
Comment #98
gábor hojtsyMinor updates to issue summary. Adding a simple release notes snippet. Not sure what else to add, probably need to keep it short for anyone to read but need to be informative enough :)
Comment #99
alexpottUpdate issue summary for latest changes:
DefaultLangcodeConfigReconcileEventevent triggered by\Drupal\language\DefaultLangcodeConfigUpdater::reconcileConfigLangcodeOperationand \Drupal\language\DefaultLangcodeConfigUpdater::isReconcilinghook_language_preserve_config_langcode()withDefaultLangcodeConfigReconcileEvent::skip()Comment #100
alexpottI;ve also fixed the code to pass the test @gábor hojtsy added in #97
Comment #101
alexpottComment #102
alexpottFixed issues with the issue summary.
Comment #103
alexpottAdded an AI usage section to the issue summary.
Comment #104
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".
This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #105
gábor hojtsyI looked at this with LLM help which I then manually reviewed :)
1. A recipe module install replaces existing locale translations with English.
This one is a new problem that was caused by the
!isSyncing()check that was added for the import case I raised.English testtranslated toHungarian test, not customized.Locale string after the install:
English test, customized=1.Hungarian test, customized=0.Hungarian test, customized=0.This seems to be cause by the following:
RecipeRunner::installModules()setts syncing to TRUE. The early return inLocaleConfigSubscriber::onConfigSave()requires syncing to be FALSE, so it does not apply. ThenConfigInstallersets the langcode tohubecauseRecipeRunner::isApplying()returns TRUE. But the value is still English. So the value is saved as the Hungarian translation because it differs from the existing translation.2. A customized config entity (different text from the default) is not stored in locale.
Not a recently introduced problem with the MR, but is a situation where I think it behaves different from how I would expect it (IMHO unintentionally changes behaviour from main).
Steps:
Default.Custom(which is different from the default valueDefault).updateConfigTranslations()runs.Results on the MR:
hu, then the existing translation replaces the label, even though it was not customized and should actually be overwritten. Main keepsCustomand stores it in locale as customized (which I would expect).huand the file langcode isenor missing, then the translation replaces the label.Customno longer exists in storage. Main keepsCustomin active config and creates ahuoverride for the translation. I think in this case the English was intended to beCustomshould end up in a translation override, no?en, then the result is the same as main.The reason for this one is that
ConfigInstallerdoes not create config entities while syncing is TRUE.RecipeConfigInstallercreates the entity after syncing returns to FALSE. The configuration is new and contains_core.default_config_hash. All conditions of the early return inLocaleConfigSubscriber::onConfigSave()are met, so nothing is written to locale.I think this is hard/impossible to fix in the current design as
ConfigInstallerchangesentohubefore the config is saved. By the time locale handles the save,Customwithlangcode: hulooks the same whether the file was really Hungarian or English. The first case should become a Hungarian customized translation, and the second anenoverride I think. A fix would either need to keep the file’s original langcode longer, or handle this before the relabel.Changing the default language replace edited English value with the shipped value if there was no translation.
Steps:
My edited English value.My edited English valuetoo as it does not have a translation override.Results:
langcode: fr, value is the original shipped config text (wrong).enoverride:test: My edited English value(good).extractUntranslatedKeys()returns all translatable keys that are not in the override for the new default. This includes values in English when the old default is English.getSourceStrings()returns the shipped source for these keys.getDataForDefaultLangcodeChange()merges the source over the edited value, so values that the site did not even use anywhere unexpectedly show up.Comment #106
alexpottI've addressed and added test coverage for all the issues raised in #105.
For point 1 - I incorporated a check of whether a recipe is being applied into the LocaleConfigSubscriber - https://git.drupalcode.org/project/drupal/-/merge_requests/16913/diffs?c...
For point 2 - I've added to the original langcode to the _core config info if it is not english - and then used this information in LocalConfigSubscriber to work out what to do - https://git.drupalcode.org/project/drupal/-/merge_requests/16913/diffs?c...
For point 3 - I no longer supplied the untranslated keys for en config - https://git.drupalcode.org/project/drupal/-/merge_requests/16913/diffs?c...
Comment #107
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. The merge request has merge conflicts and cannot be merged. Therefore, this issue status is now "Needs work".
This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #108
alexpott