Problem/Motivation

Steps to reproduce

  • Fresh install (in english)
  • Enable the locale and language modules.
  • Add a language (dutch in my case)
  • Make the newly added language standard language
  • Enable a module which provides default config (book module for example)

Note that this bug only occurs if the default language or the fallback language is not English.

Result

In the UI, you will see some English instead of Dutch on various pages that load configuration.

In the database:
- The base config items in the config table now have a langcode element in them, telling Drupal they are in Dutch not English.
- So when you try to load config, the base config thinks it is Dutch (even though the text is English), and the config overrides (which are still in the config database) are not being loaded to translate the English into Dutch.

Expected result

The English configuration is interpreted as English config, not Dutch, and the config translation overrides are used when viewing the site in Dutch.

Related issues

An issue which does a great job of explaining the issue, including a reference to core issue this was introduced. It also provides proposals: #3150540: Configuration langcode is forced to site default language

This issue seems fairly similar to #2910353: Prevent saving config entities when configuration overrides are applied .

Also, UI text gets overridden when modules are installed. This issue is about config being mangled; the UI text problem is a separate issue: #2408549: Display status message on configuration forms when there are overridden values and solved #2806009: Installing a module causes translations to be overwritten.

From comment #72 on, broader work on the configuration system was proposed instead. This/related work has been taken up (partly via 'duplicate' #3568743: Convert / unify all default configs to the site default language) in #3337864: Introduce a dedicated "Configuration default language" different from "Site default language". That might resolve this issue, or it likely needs to be re-tested after..

Proposed resolution

There was some debugging done; probably the cause is described in #6 / #11 / #14

There was a test-only patch in #5 but it's really for the related UI text issue, not the config text issue.

Remaining tasks

Make a patch, including a test.

User interface changes

Configuration language will not be changed to an inappropriate language.

API changes

None.

Data model changes

None.

Issue fork drupal-2905295

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

geertvd created an issue. See original summary.

geertvd’s picture

I see the same behaviour when I do the install in a foreign language. During site installation everything is fine, but if you enable a module after that the same thing happens as described in the IS

cilefen’s picture

Priority: Normal » Critical

This is critical if reproducible.

geertvd’s picture

Status: Active » Needs review
StatusFileSize
new0 bytes

This patch should fail and prove the issue.

geertvd’s picture

StatusFileSize
new3.02 KB

Now one that isn't empty.. :)

geertvd’s picture

After some debugging I found out LocaleConfigSubscriber::onConfigSave() is triggered when a module is enabled, the passed config object contains the langcode of the default language rather then falling back on "en".

The last submitted patch, 4: 2905295-4.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 5: 2905295-5.patch, failed testing. View results

star-szr’s picture

Issue tags: +Triaged D8 critical

Discussed with @xjm @catch @effulgentsia @lauriii and we agreed this is a critical issue (data loss, if I'm not mistaken).

jian he’s picture

This issue is annoy. We have to doing this after module install:
vendor/bin/drush langimp --langcode=zh-hans sites/default/files/translations/drupal-8.4.x-zh-hans.po

sutharsan’s picture

Duplicating isssue found in #2806009: Installing a module causes translations to be overwritten with steps to reproduce by @jhodgdon and debbuging results by @Sutharsan.

Further up the call stack from LocaleConfigSubscriber::onConfigSave, mentioned by @geertvd, in locale_system_set_config_langcodes() it is written:

// Need to rewrite some default configuration language codes if the default
// site language is not English.

This is exactly what happens, it only changes the langcode, but does not change the translatable strings to match the langcode.

jhodgdon’s picture

Note different steps to reproduce in the other issue, which was filed a year earlier and not really noticed by anyone until now... it's not quite the same issue maybe, but probably identical cause.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

ckaotik’s picture

Thanks @Sutharsan for the pointer to locale_system_set_config_langcodes. That code (seemingly) blindly sets the config language to

$config->set('langcode', $default_langcode)->save();

, but only the langcode, not the actual values. So in my case, I ended up with English config values being stored as de.

I wonder how this should best be handled, because I susoect that it's only an issue if using the config_translation module.

sutharsan’s picture

I susoect that it's only an issue if using the config_translation module.

No, it is an issue with two or more languages with Language and Locale module only. And having untranslated strings that are used in config. For example exporting config before importing interface translations.

The latter happens easily because Drush does not download/import translations when enabling a module (due to a bug).

mpp’s picture

@Sutharsan, could you add the reference to the drush issue?

The latter happens easily because Drush does not download/import translations when enabling a module (due to a bug).

sutharsan’s picture

"D8 Run batch process during pm-enable when needed" https://github.com/drush-ops/drush/pull/3445
The equivalent Drush 9 pull-request is already resolved.

jhodgdon’s picture

So... It seems like the process discussed in #11 / #14 is only part of the problem. That code is:
https://api.drupal.org/api/drupal/core%21modules%21locale%21locale.modul...

For some unknown reason, it is going through all stored configuration, and updating each config item's langcode to the default site language.

So this would account for configuration being overwritten with English configuration. But it doesn't account for other translations in the UI translation (Locale module) database being overwritten, which also happens.

jhodgdon’s picture

What I mean is, if you look at the Steps to Reproduce in the issue summary of #2806009: Installing a module causes translations to be overwritten, which has been closed as a duplicate of this issue, you can see that regular UI strings like "Content" also have their translations overridden, presumably due to what is happening in
https://api.drupal.org/api/drupal/core%21modules%21locale%21locale.modul...

I'm not sure why that would be, but it doesn't have anything to do with configuration (which is also most likely corrupted).

jhodgdon’s picture

So should we have two separate issues, one for config being overridden (which I am not exactly sure is actually happening?? But I think it probably is), and one for UI strings?

jhodgdon’s picture

Title: Setting default language to any language other than english results in interface translations being overwritten during module install » Interface translations being overwritten during module install if site default language is not English
Issue summary: View changes
Status: Needs work » Active

OK. There are two different, but both serious, bugs here for sure.

The issue summary here is about configuration translations being overwritten, and I have just confirmed this is true. This is caused by what is described in #6 / #11 / #14. The result in the database is that:
- The base config items in the config table now have a langcode: 'du' element in them.
- So when you try to load config, the base config thinks it is Dutch (even though the text is English), and the config overrides (which are still in the config database) are not being loaded to translate the English into Dutch.

The other issue I reported in #2806009: Installing a module causes translations to be overwritten is kind of similar: if you enable a module, translations for user interface text, which are stored in the locales_source / locales_target tables, are also being overwritten. This is NOT caused by what is described in #6 / #11 / #14 , I think. The patch in #5 here provides a valid test that fails and illustrates this bug. It does not illustrate the config bug that is described here.

So.... I think what we need to do is:
- Reopen the related issue. It's not quite a duplicate.
- Move the test-only patch from #5 here over to that issue, because it's a great test of that problem.
- Write a new test-only patch for this issue, that illustrates the config override problem.

I'm also setting the status of this issue back to Active, and am updating the summary.

jhodgdon’s picture

Hiding the file that belongs on the other issue.

berdir’s picture

Didn't read through the whole issue yet, but this sounds a lot like a duplicate or at least very much related to #2910353: Prevent saving config entities when configuration overrides are applied

jhodgdon’s picture

Title: Interface translations being overwritten during module install if site default language is not English » Configuration language being overwritten during module install if site default language is not English

Fixing title

catch’s picture

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mmrares’s picture

I have the same problem when a module gets installed but we managed it by doing a config import to restore the correct language (en) until we used ignored configuration using config_ignore module. Since the configuration is ignored, a new config import doesn't do anything to that configuration and the language is stuck to default site language.
And at this point there is no way around that.

jcnventura’s picture

Or you run this on drush sqlc after each module install:

UPDATE `config` SET data = cast(replace(cast(data AS char), 's:8:"langcode";s:2:"XX"', 's:8:"langcode";s:2:"en"') AS BINARY);

Where XX is the langcode of the site's default language. It's an ugly kludge. But I can verify this is working during code review of config commits.

mpp’s picture

This also happens when the site default language is English but the default fallback language isn't English.

andypost’s picture

Version: 8.6.x-dev » 8.8.x-dev
mpp’s picture

Title: Configuration language being overwritten during module install if site default language is not English » Configuration language being overwritten during module install
jhodgdon’s picture

Issue summary: View changes

I don't think it happens unless the default language or fallback language is not English, so I am not sure about the title change. Adding that clarification to the issue summary.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

yonailo’s picture

As it is said in #18, I really do not understand why 'locale_system_set_config_langcodes()' has to modify all configuration files languages. If I understand well the code in that function, having no language key is treated the same as being 'en', then language is changed to the "default_langcode" which is not 'en'.

But a configuration file is always in a given language, so what's the point of changing the language this way ? It seems totally useless for me.

johnzzon’s picture

Encountered this today after installing EU Cookie Compliance on a site with Swedish as default language.

Installing and configuring the module first time is fine. But when you reinstall from existing config, it overwrites my custom config with the Swedish translations from translations://eu_cookie_compliance-8.x-1.8.sv.po.

I expect the strings in my configuration in config/sync to remain. We heavily rely on reinstall for our workflow and it's quite annoying to have the config diff all the time. Can you block importing translations somehow? That'd be a workaround until this is fixed.

mpp’s picture

I don't think it happens unless the default language or fallback language is not English, so I am not sure about the title change. Adding that clarification to the issue summary.

@jhodgdon, all of our sites (me, @geertvd, @mmrares) have English as the default language, we often do have set Dutch as the default language negotiator (/ => /nl).

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

stephane aimar’s picture

Hi evrybody,

I got the same issue, here is my site setup:

  • Multisite having differents multilinguages by site.
  • All the website have his own default language.
  • English is common to all websites.

We manage all common configurations with Features module, all shipped with the common english langcode.

So here is the moment we got the issue, each time we provide a new module installed with a hook_update_n all configuration langcode are falling back at each differents website's default language code.

And there is the moment the translations related issue come too.

I was thinking of a short and smart way to prevent the current default language override. I thaught that there is no reason that shipped configuration language has to be overridden if the shipped langcode exist allready in the website, default or not.

Do you think It's a good way to operate?

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

rar9’s picture

Another 2 Months have passed since my ticket was closed - https://www.drupal.org/project/drupal/issues/3161643#comment-13765390

Drupal 9.1 is approaching soon. Will this CORE bug be resolved in the coming release?

Will any of the above patches fix my Maintenance bug?

jhodgdon’s picture

This kind of comment is not actually helpful or constructive -- complaining will not get this issue fixed sooner. Please read
https://www.drupal.org/docs/develop/issues/issue-procedures-and-etiquett...
and the two pages listed under Related Content.

askibinski’s picture

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

rar9’s picture

Is this patch perhaps helping to fix my Issue?
https://www.drupal.org/project/drupal/issues/3161643

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

stephencamilo’s picture

Status: Active » Closed (won't fix)
simgui8’s picture

@ stephencamilo: any reason you are closing this?

This is not fixed and IMO it should stay open.

poker10’s picture

Status: Closed (won't fix) » Active

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

Issue tags: +Bug Smash Initiative

I tested this on 9.5.x and was not able to reproduce this error. I followed the steps given in the Issue Summary, although I installed Italian as the second language. I installed book via drush and tried some other via the UI.

However, when describing the error the IS states

In the UI, you will see some English instead of Dutch on various pages that load configuration.

Various pages is too vague. What specific pages are having this problem?

I also looked at the config table in the database and the 'base config' (if I understand that correctly) was always without a langcode and additional 'it' and 'en' versions were added for the translations.

xjm’s picture

andriy khomych’s picture

I can confirm the above issue.
It will work with the above steps, or in my case, it is caused by the custom importer command which is reimporting every time configs from a specific folder.
Moreover, it ends up with 3 DB records of config collections and later incorrect data in the configuration.
While I fixed incorrect data by explicitly calling locale_system_set_config_langcodes() function at the end of the importer command the overwriting of the configuration still exists. As a workaround I used configuration ignore and ignores those configs.
As an idea, we can allow the existing logic of the locale module but use the translatable attribute and based on it merge prev and imported configurations, and skips the overwriting of some translatable parts. E.g. from here - https://www.drupal.org/docs/drupal-apis/configuration-api/configuration-...

Translatability
The label, plural_label, date_format and text types are
defined as translatable: true.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

shalini_jha’s picture

Hello everyone,

I attempted to reproduce this issue on my local environment, and I successfully replicated it by following these steps:

step 1) Freshly installed Drupal.
step 2) Installed essential multilingual modules such as locale, content_translation, config_translation, and language.
step 3) Added a new language, Dutch, through the admin interface (admin/config/regional/language),and added dutch as a default.
step 4) downloaded Dutch translations from localize.drupal.org.
step 5) Went to /admin/config/regional/translate/import and imported the translation file (.nl.po extension).
step 6) Examined the configuration of a content type (in this case, the Article content type) and confirmed that all values were stored in Dutch (see screenshot: article content type config.png).
step 7) Installed a module with default configurations; for example, I tried the Book module.
Step 8)After installing the Book module, I tested the configuration of the Book content type and noticed that it forcefully overrides the configuration language to Dutch (see screenshot: book content type configuration.png).

Upon investigation, I found that the issue is related to the locale_system_set_config_langcodes function in local.module. The function calls updateDefaultConfigLangcodes, which contains the following code:

$default_langcode = $this->languageManager->getDefaultLanguage()->getId(); //this will provide selected default language in site.
$langcode = $config->get('langcode'); // this will provide the configuration language like in book module have "en".
if (empty($langcode) || $langcode == 'en') {
   $config->set('langcode', $default_langcode)->save();
}

This code forcefully sets the configuration language to the default site selected language whenever a module or theme is installed
This behavior seems to be affecting the Book module, where the configuration language code is 'en' as actual language.

I have added a logger in tracing the execution flow and identifying the specific language code being added to the Book module configuration. Notably, the logger revealed that the Book module's configuration is assigned the language code 'en' (see screenshot: logger-message.png).

shalini_jha’s picture

shalini_jha’s picture

StatusFileSize
new74.62 KB
shalini_jha’s picture

Status: Active » Needs review
StatusFileSize
new1.5 KB

I have taken reference from the #38 , as it prevent the config overrides to default language to default config file of module. as i have tested this book module only after applying this patch book module default config value is not overrides with dutch language its in English only.
i have added a patch against 11.x .
Please review.

shalini_jha’s picture

i have added a MR with test. please review.

smustgrave’s picture

Status: Needs review » Needs work

MR appears to have a test failure.

Added some nitpicky comments to the MR that should be addressed too while tests are updated.

shalini_jha’s picture

Addressed the mentioned comment, please review.

joachim’s picture

I wonder if the test be a kernel test rather than a browser test? I'm going to experiment with converting it :)

joachim’s picture

(ignore, false alarm)

joachim’s picture

I can't work out how the translated node type label is supposed to show when you go to de/admin/structure/types/manage/page -- it doesn't work manually for me at all.

So I can't figure out how the actual test part of the test could be done using a kernel test, as I can't work out what's happening when it goes wrong. How does the translated node type label get loaded to be shown in the form?

In the meantime, I'm streamlining the Functional test with code from the kernel test I started writing.

I'm also removing the mentions of locale_test_translate -- that's not being enabled in this test, so it looks like frankencode comments to me :)

joachim’s picture

I wonder whether the kernel tests in LocaleConfigSubscriberForeignTest are relevant here -- they seem to be testing the same sort of things.

joachim’s picture

Ok I am now fairly sure that test coverage for this should be added to LocaleConfigSubscriberForeignTest. But I am very confused about how to make that work:

  public function testInstallModuleWithConfiguration() {
    // Install the Language module's configuration so we can use the
    // module_installer service.
    $this->installConfig(['language']);
    $this->container->get('module_installer')->install(['locale_test_translate']);
    $this->installConfig(['locale_test_translate']);


    // Do we need to do this?
    locale_system_set_config_langcodes();
    $langcodes = array_keys(\Drupal::languageManager()->getLanguages());
    $names = Locale::config()->getComponentNames();
    Locale::config()->updateConfigTranslations($names, $langcodes);

    // Not this -- it fails on both 11.x and the feature branch.
    $this->assertEquals('hu', \Drupal::service('locale.config_manager')->getDefaultConfigLangcode('locale_test_translate.settings'));

    // ???
    $this->assertEquals('hu', $this->configFactory->getEditable('locale_test_translate.settings')->get('langcode'));

My problem is that I don't understand the underlying architecture of how config is translated to understand what all the test helpers do and what the services like localeConfigManager / LocaleTranslation etc do. I suspect this is the case with most people and that the translation system has a very low bus factor!

joachim’s picture

I'm digging some more in this.

I'm confused by the fix in the MR being in code that says this:

      // Update active configuration copies of all prior shipped configuration if
      // they are still English. It is not enough to change configuration shipped
      // with the components just installed, because installing a component such
      // as views may bring in default configuration from prior components.

'prior shipped configuration' means config that was installed from a module *before* the current install operation. So I don't see how that's going to help with the bug, which is about something that goes wrong with config from a module *as it's being installed*.

But then I'm not sure what updateDefaultConfigLangcodes() is trying to do anyway -- what does 'default' mean in this context? In getDefaultConfigLangcode(), 'default' means 'shipped with a module's code in config/install'. But here we're updating config, so it's not the shipped version is it?

shalini_jha’s picture

Whenever a module with configuration settings is installed, the configuration is consistently overridden by the default language of the site if it is not set to English. The updateDefaultConfigLangcodes function is tasked with updating the language code of the configuration provided by that module. To mitigate this issue, I have implemented a check to verify whether the language code of the current configuration contains 'en' (English) and if 'en' is already included in the available language codes. This check prevents the configuration from being overridden by the default selected language.

joachim’s picture

It would be really good if the code comment explained that a bit more, and more precisely, explained the circumstances under which the language code should be updated.

I think this logic could be made clearer:

          $langcode = $config->get('langcode');
          $is_available_langcode = in_array($langcode, $available_langcodes);
          if ((empty($langcode) || $langcode == 'en') && !$is_available_langcode) {

In the case that $langcode is empty, then it's obviously not in the array. So we only need to check if it's $available_langcodes if it's 'en', AFAICT?

So for example, this would be clearer:

          if (empty($langcode) || ($langcode == 'en') && !$is_available_langcode) {

And while I am generally REALLY in favour of splitting up complex conditionals, I'm not sure breaking out the check for $is_available_langcode is good for readability here, and it's not efficient, as it's checking even if $langcode is empty, or not 'en'.

I'm still not sure how we test the patch using the API.

I've installed book when 'fr' is the default language, with and without the fix, and looked at the {config} table.

Without fix:

	node.type.book	a:12:{s:4:"uuid";s:36:"10752ba3-9235-4540-a959-c9e9190376ae";s:8:"langcode";s:2:"fr";s:6:"status";b:1;s:12:"dependencies";a:1:{s:8:"enforced";a:1:{s:6:"module";a:1:{i:0;s:4:"book";}}}s:5:"_core";a:1:{s:19:"default_config_hash";s:43:"xvVZ9piiDxh4ziNyl-YqCT5vF8nI7xyupTdQWlN-Hxk";}s:4:"name";s:9:"Book page";s:4:"type";s:4:"book";s:11:"description";s:87:"<em>Books</em> have a built-in hierarchical navigation. Use for handbooks or tutorials.";s:4:"help";N;s:12:"new_revision";b:1;s:12:"preview_mode";i:1;s:17:"display_submitted";b:1;}
language.en	node.type.book	a:2:{s:4:"name";s:9:"Book page";s:11:"description";s:87:"<em>Books</em> have a built-in hierarchical navigation. Use for handbooks or tutorials.";}

With fix:

	node.type.book	a:12:{s:4:"uuid";s:36:"88e05137-2a0e-4a18-840e-60769f3dccd7";s:8:"langcode";s:2:"en";s:6:"status";b:1;s:12:"dependencies";a:1:{s:8:"enforced";a:1:{s:6:"module";a:1:{i:0;s:4:"book";}}}s:5:"_core";a:1:{s:19:"default_config_hash";s:43:"xvVZ9piiDxh4ziNyl-YqCT5vF8nI7xyupTdQWlN-Hxk";}s:4:"name";s:9:"Book page";s:4:"type";s:4:"book";s:11:"description";s:87:"<em>Books</em> have a built-in hierarchical navigation. Use for handbooks or tutorials.";s:4:"help";N;s:12:"new_revision";b:1;s:12:"preview_mode";i:1;s:17:"display_submitted";b:1;}

I can see there's a difference, obviously -- only one row rather than two!

But shouldn't I be seeing translated text in there?

I also don't understand what the API reports.

I've got this debug code:

$config_name = 'node.type.book';
      // enabled but not default language
      $override = $this->languageManager->getLanguageConfigOverride('en', $config_name);
      dump($override->isNew());

      // the default language
      $override = $this->languageManager->getLanguageConfigOverride('fr', $config_name);
      dump($override->isNew());

WITH the fix, I get FALSE, TRUE

WITHOUT the fix, I get TRUE, TRUE

I don't understand how BOTH are reporting they're new without the fix.

kriboogh’s picture

I think the whole config translation system should be simplified and could be solved if:
- "default" config (the yml config files shipped with modules), is always in langcode "en" and contains English strings (obviously).
- All module translations are loaded and imported before the config is imported, so string translation is available.
- For each default English config, a language override is created when installing a site in a different language or when additional languages are added. So a site with languages NL and FR, with FR the default language, has English configs and two overridden configs in Dutch and French.
That way you can switch default site language whenever you want, as much as you want, add languages, delete them. Without default config ever needed to be updated.

kristiaanvandeneynde’s picture

Heavy +1 for what @kriboogh said in #72. It's basically what I also said in #2806009-51: Installing a module causes translations to be overwritten. That issue was fixed to address the symptoms at hand, but the root cause was never really addressed for all I can tell.

We require all projects on DO to ship their config in English, so why don't we make use of that and keep the config/sync folder English at all times? This would make everything so much easier from a code perspective too as we then know that the main config folder is always English and the language-specific folders are always for said language.

From my comment above the one I linked:

We also have the default language set to DE or NL, but need the contents of config/sync to be EN. The reason is simple: All modules ship with EN config, so if we wanted our config/sync folder to be DE or NL, then all modules would not be able to drop their config in config/sync unless it would be in the wrong language for that folder.

anybody’s picture

I also agree with #72 / #73. Thanks!

catch’s picture

We require all projects on DO to ship their config in English, so why don't we make use of that and keep the config/sync folder English at all times?

I was trying to think what the effect of this would be on a monolingual non-English site. I think in that case, if you really, really didn't want to enter English for some config, you could just add it in whatever language (let's say French), the system would store your French text in config/sync as 'English', and then you'd have to translate from 'English' to French (copy and paste the same thing or leave it untranslated).

And even though that would be a bit of a workaround, it seems like it'd be really predictable what would happen, so still OK overall. And then all the other situations would be simplified and make a lot more sense. I only deal with one multilingual site and even that site is not fully multilingual, but with that caveat, +1 from me too.

aleksip’s picture

+1 for default config always being in English.

kristiaanvandeneynde’s picture

So what would be the next steps to fix this? I've found a handful of issues that all boil down to the same root cause described in #72 / #73. If we want to switch to a way where the main config folder is always EN, which steps can we already take without breaking BC or with a solid update path?

jcnventura’s picture

What I'd like to see here would be a new entry to the system.site.yml file that would have the following value:
config_langcode: en
This would be set to English if the site is installed in English, or to whatever other language is the site's install language.

At the moment, there would be no provision to change this value in the UI, but if we ever figure out a way to do that, while not messing up a lot of translations, that could change in the future.

kriboogh’s picture

I think that by allowing the config_langcode to be configurable like that, you're gonna end up in the same mess if someone changes that config setting while the site already has config in the first language.

By having a default config in EN always and have the language config overrides system (which already is in place and working) deal with the translations, everything should basically work out of the box (sort of). The only problem is "fixing" existing websites when we introduce this fixed EN config langcode.

We have done this in a current project. Basically you need to get all default config that was stored in the current default language (for example NL). Change the langcode of the default config to EN. WE now have a EN default config with NL data. So look if a EN config override exists. If there is, swap all data from the EN override into the new EN default. If there isn't an EN override, look for the original EN source strings of the data in the NL data and copy that into the new default EN config. To create a NL override, we can either let core create it (by looking up the translations through locale, or copy the old default translatable NL keys (labels, text,...) from the original NL default config into the new NL override.

catch’s picture

We have done this in a current project. Basically you need to get all default config that was stored in the current default language (for example NL). Change the langcode of the default config to EN. WE now have a EN default config with NL data. So look if a EN config override exists. If there is, swap all data from the EN override into the new EN default. If there isn't an EN override, look for the original EN source strings of the data in the NL data and copy that into the new default EN config. To create a NL override, we can either let core create it (by looking up the translations through locale, or copy the old default translatable NL keys (labels, text,...) from the original NL default config into the new NL override.

This seems sufficiently complex that we might not be able to provide an update hook for it in core.

I'm not sure exactly how (maybe a flag in settings or a container parameter), but can we look at forcing English for default config on new installs, which would fix the bug for new sites, and then we could at least provide instructions (and maybe a contrib module) to help existing sites convert based on the above?

kristiaanvandeneynde’s picture

If we force it on new sites only, then the code base also needs to take care of both scenarios: old and new. I can already sense the amount of bug reports we'd be getting then. If we provide an update hook, then the code base only needs to take the new scenario into account.

@kriboogh would you mind sharing your update code here? Perhaps it might be simpler than it sounds.

kriboogh’s picture

@kristiaanvandeneynde sure, it's not pretty though :D
We had to do some specific stuff with system.site config, which might have been only related to our project. But I'll keep it in.

Disclaimer: To anyone else reading this, this is specific project code given as an example, do not run this blindly on your production site !!!

function my_module_update_9000(&$sandbox) {

  // See patch https://www.drupal.org/project/drupal/issues/3150540.
  \Drupal::configFactory()->getEditable('locale.settings')->set('update_default_config_langcodes', FALSE)->save();

  // 1. Fix system.site config.
  $db = \Drupal::database();
  $db->delete('config')
    ->condition('collection', 'language.en')
    ->condition('name', 'system.site')
    ->execute();

  // Make sure the default config is set up as 'en'.
  $default_langcode = \Drupal::languageManager()->getDefaultLanguage()->getId();
  $default_config = \Drupal::configFactory()->getEditable('system.site');
  if ($default_config->get('langcode') == $default_langcode) {

    // Keep the default langcode name, we need it for the new override.
    $old_default_config_data = $default_config->getRawData();

    // Change the default langcode to english.
    $default_config->set('langcode', 'en');

    // Check if we have an english translation override, copy over its values.
    $config_en_translation = \Drupal::languageManager()->getLanguageConfigOverride('en', 'system.site');
    if (!$config_en_translation->isNew()) {
      $default_config->set('name', $config_en_translation->get('name'));
      $default_config->set('mail', $config_en_translation->get('mail'));
    }
    $default_config->save();

    // Create a new language override for the default langcode and store
    // the old values.
    if (!empty($old_default_config_data['name']) || !empty($old_default_config_data['mail'])) {
      $config_default_langcode_translation = \Drupal::languageManager()->getLanguageConfigOverride($default_langcode, 'system.site');
      if (!empty($old_default_config_data['name'])) {
        $config_default_langcode_translation->set('name', $old_default_config_data['name'] ?? '');
      }
      if (!empty($old_default_config_data['mail'])) {
        $config_default_langcode_translation->set('mail', $old_default_config_data['mail'] ?? '');
      }
      $config_default_langcode_translation->save();
    }
  }

  // Now cleanup the language overrides, they should only contain the site name and mail.
  foreach (\Drupal::languageManager()->getLanguages() as $langcode => $language) {
    if ($default_config->get('langcode') != $langcode) {
      // Only do overrides, skip the default config.
      $config_translation = \Drupal::languageManager()->getLanguageConfigOverride($langcode, 'system.site');
      if (!$config_translation->isNew()) {
        // Ignore non existing overrides.
        $config_translation->clear('page');
        $config_translation->save();
      }
    }
  }

  // 2. Check all default config langcodes and check if a language override
  // exists with the same langcode. Fix that.
  $query_string = <<<MYSQL
    SELECT c1.name, c1.collection
    FROM config c1, (SELECT c.name,
        REGEXP_REPLACE(REGEXP_SUBSTR(c.data, 's:8:"langcode";s:2:"(..)"'), '.*s:2:"(..)"', '\\\\1') as langcode
        FROM config c
        WHERE
          c.collection = ''
        AND
          c.data REGEXP 's:8:"langcode";s:2:".."'
        ) as clang
    WHERE
    c1.name = clang.name
    AND
    c1.collection = CONCAT('language.', clang.langcode)
  MYSQL;

  $rows = $db->query($query_string, [], ['allow_delimiter_in_query' => TRUE])->fetchAllAssoc('name');
  foreach ($rows as $row) {
    $db->delete('config')
      ->condition('collection', $row->collection)
      ->condition('name', $row->name)
      ->execute();
  }

  // 3. Swap default language config with language.en overrides.
  $query_string = <<<MYSQL
    SELECT c1.*
    FROM config c1, (SELECT c2.name FROM config c2 WHERE c2.collection = '' AND c2.data REGEXP 's:8:"langcode";s:2:"{$default_langcode}"') as c2
    WHERE c1.name = c2.name and c1.collection = 'language.en'
  MYSQL;

  $rows = $db->query($query_string, [], ['allow_delimiter_in_query' => TRUE])->fetchAllAssoc('name');
  foreach ($rows as $name => $row) {

    $default_config = \Drupal::configFactory()->getEditable($name);
    $english_config_override = \Drupal::languageManager()->getLanguageConfigOverride('en', $name);

    if (!$english_config_override->isNew() && !$default_config->isNew()) {
      // Only do this if both configs already exist!
      // Copy the english override into the default config and change it
      // to english.
      $english_config_override_data = $english_config_override->get();
      $default_config_data = $default_config->getRawData();
      $english_default_data = array_replace_recursive($default_config_data, $english_config_override_data);
      $default_config->setData($english_default_data);
      $default_config->set('langcode', 'en');
      $default_config->save();

      // Now take the original default config values and create a new
      // language override with the default language. Store only the keys
      // from the default config that where present in the english override.
      $default_config_override = \Drupal::languageManager()->getLanguageConfigOverride($default_langcode, $name);
      $default_config_override_data = array_intersect_key_recursive_values_2($english_config_override_data, $default_config_data);
      $default_config_override->setData($default_config_override_data);
      $default_config_override->save();

      // Delete the english override.
      $english_config_override->delete();
    }
  }

}

/**
 * Do a recursive key intersect but return values from array 2.
 *
 * @param array $array1
 * @param array $array2
 *
 * @return array
 */
function array_intersect_key_recursive_values_2(array $array1, array $array2) : array {

  $intersection = [];

  foreach ($array1 as $key => $value) {
    if (isset($array2[$key])) {
      if (is_array($array1[$key]) && is_array($array2[$key])) {
        $intersection[$key] = array_intersect_key_recursive_values_2($array1[$key], $array2[$key]);
      }
      else {
        $intersection[$key] = $array2[$key];
      }
    }
  }

  return($intersection);
}
kriboogh’s picture

StatusFileSize
new10.27 KB

Follow up: We were still having issues on a site by this. The problem was the site has only one language (NL) which is also the default language. I'm now trying to also have English installed and set EN as the default language. This will force config to be EN. I updated the updb hook above to swap and fix config translations (see attached file). The only downside now is that what used to be a single language website, now has two languages (NL and EN). We installed disable_language module to hide EN in the site. Also this module needs a patch to also hide the disabled language in translation overview.

Again disclaimer: To anyone else reading this, this is specific project code given as an example, do not run this blindly on your production site !!!

nmudgal’s picture

Thanks to everyone for all the work on this thread. I’ve read through the discussion, and it seems like setting English as the default langcode for configurations could be a straightforward way to avoid language conflicts during module installs. Here’s a quick summary and a few ideas on how we might combine some of the suggestions:

  1. Default Configurations to English (langcode: en):

    Setting langcode to English across all module configurations would help avoid conflicts and keep things consistent, especially on multilingual sites. This way, English serves as the base, with other languages applied as overrides as needed.
  2. Combined Contrib Module and Migration Path for Legacy Sites:

    For older sites, a contrib module could help automate the migration to this new standard. It could convert non-English langcode values to English and move existing language settings into overrides. This module could also support language overrides and protect configurations during updates.
  3. Temporary Workarounds (config_ignore):

    While config_ignore and similar modules are useful right now, moving to a standardized langcode approach would eventually make these workarounds unnecessary, simplifying language management.

xjm credited effulgentsia.

xjm credited lauriii.

xjm’s picture

Adding issue triage credit as per #9.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.

roderik’s picture

Added some references to the issue summary.

Larger work (pertaining to the discussion in #72 onward) is currently being done in #3337864: Introduce a dedicated "Configuration default language" different from "Site default language". I'm not sure if this issue (which is more specific than that) should be 'duplicate' or 'postponed (re-test and/or re-check the automated test when that lands)'. So I'm not touching status.

The MR has a merge conflict since #3427564: Require `langcode: …` only for simple config that contains translatable values landed. I don't know if adding the included test to Core will still be useful when it's rebased, but... that probably also needs adjustment because it uses the not-in-Core-anymore 'book' module.