Part of #2557113: Make t() return a TranslationWrapper object to remove reliance on a static, unpredictable safe list

Problem/Motivation

In BooleanItem, we do t('On') and t('Off') when defining the default values for boolean items. The config schema checker has a problem with this. When it is enabled during a test, it fails on some config with TranslationWrapper objects in it, with the message "The configuration property settings.on_label.*string* doesn't exist.", while checking core.base_field_override.node.page.promote during installation of the standard profile.

Proposed resolution

We cast TranslationWrappers in config to string before saving the config objects.

We also turn the labels in the default field settings for booleans into a TranslationWrapper as this is what would happen anyway when #2557113: Make t() return a TranslationWrapper object to remove reliance on a static, unpredictable safe list lands and as we often load these without rendering. This way we also have implicit test coverage (besides the explicit test coverage included in this patch), as contrib modules also might be using t() in values that will end up in config such as field default settings.

Ideally the on_label/off_label should be translatable in schema but this is out of scope for this issue.

Remaining tasks

N/A

User interface changes

N/A

API changes

We now explicitly allow passing TranslationWrappers into config only to cast it to string upon saving. This as contrib may also use t() in config (like core used to in BooleanItem)

Beta phase evaluation

Part of a major issue.

Comments

stefan.r created an issue. See original summary.

stefan.r’s picture

StatusFileSize
new411.61 KB
new91.31 KB

Debug traces for reference:

stefan.r’s picture

Status: Active » Needs review
StatusFileSize
new1020 bytes
new2.09 KB
new1.09 KB

Adding two fail patches to illustrate.

stefan.r’s picture

Issue summary: View changes
joelpittet’s picture

StatusFileSize
new1.85 KB
new3.79 KB

Nice find debugging! Added that same pattern to StorableConfigBase which solve most if not all of the remaining failures.

joelpittet’s picture

StatusFileSize
new2.49 KB
new3.63 KB

Actually I think they should do the cast to string on the $value instead of in the loop on the nested value that way the string on a simple key:value will also be cast appropriately. Does that make sense?

The last submitted patch, 3: 2562155-fail-lots.patch, failed testing.

The last submitted patch, 3: 2562155-fail-less.patch, failed testing.

The last submitted patch, 3: 2562155-3.patch, failed testing.

The last submitted patch, 5: 2562155-5.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 6: 2562155-6.patch, failed testing.

stefan.r’s picture

@joelpittet let's solve that remaining test fail then and rename the issue to reflect the expanded scope here?

joelpittet’s picture

Rename the issue seems fine to me. That last fail seems tricky. It's going into the symfony yaml::dump(). Trying to breakpoint the test

stefan.r’s picture

I doubt the problem is in FileStorage->write() and in the schema checker here, we are reading config from the database with a TranslationWrapper in on_label which seems wrong considering that:

# Schema for the configuration of the Boolean field type.

field.field_settings.boolean:
  label: 'Boolean settings'
  type: mapping
  mapping:
    on_label:
      type: string
      label: 'On label'
    off_label:
      type: string
      label: 'Off label'

So probably that should have been stored as a string instead. I think ModuleInstaller::install() is where we'll want to look, if we solve it there some of the other changes may not be needed.

dawehner’s picture

+++ b/core/lib/Drupal/Core/Field/Plugin/Field/FieldType/BooleanItem.php
@@ -33,8 +34,8 @@ class BooleanItem extends FieldItemBase implements OptionsProviderInterface {
+      'on_label' => new TranslationWrapper('On'),
+      'off_label' => new TranslationWrapper('Off'),

IMHO we should cast it to string before we pass it to config entries ... in order to keep the existing behaviour, IMHO.

stefan.r’s picture

@dawehner yes, this was just for purposes of the test, but if we do #2557113: Make t() return a TranslationWrapper object to remove reliance on a static, unpredictable safe list this is what will happen in contrib modules using the same pattern. So I think we'll need to cater to this and cast these to string in castValue(), just to stick with historical behavior and have less of a BC break.

And then there's the fact that the on_label and off_labels are defined as untranslatable strings in the config schema, they should actually be translatable as well, but I think that would be out of scope for this issue...

stefan.r’s picture

So here's why: #2432791: Skip Config::save schema validation of config data for trusted data

We do $entity->trustData()->save(); in ConfigInstaller::createConfiguration(), which means we skip the typeddata casting :/

stefan.r’s picture

Title: SchemaCheckTrait::checkValue() chokes on TranslationWrapper objects » Cast TranslationWrapper objects in config to string during extension installation
Status: Needs work » Needs review
StatusFileSize
new5.07 KB
stefan.r’s picture

StatusFileSize
new4.76 KB

#18 still had debug code

Status: Needs review » Needs work

The last submitted patch, 19: 2562155-16.patch, failed testing.

The last submitted patch, 18: 2562155-15.patch, failed testing.

stefan.r’s picture

Status: Needs work » Needs review
StatusFileSize
new4.75 KB
new1.18 KB
stefan.r’s picture

This has implicit test coverage through the TranslationWrapper change in BooleanItem, which I think we can actually keep considering the default options are only rendered on the options screen.

joelpittet’s picture

+++ b/core/lib/Drupal/Core/Config/Config.php
@@ -209,18 +209,17 @@ public function save($has_trusted_data = FALSE) {
-    if (!$has_trusted_data) {
-      if ($this->typedConfigManager->hasConfigSchema($this->name)) {
...
+    if (!$has_trusted_data && $this->typedConfigManager->hasConfigSchema($this->name)) {
...
+    else {
+      // We fall back to simple validation.
+      foreach ($this->data as $key => $value) {
+        $this->data[$key] = $this->validateValue($key, $value);
       }

I don't know what has_trusted_data but before it would never do the simple validation on trusted data and now it does because of the moved condition. Is this change needed?

joelpittet’s picture

I like how the simplification has come along on this, it looks way better!

stefan.r’s picture

Trusted data was an optimization (see #2432791: Skip Config::save schema validation of config data for trusted data), we'll still need to do simple casting to string of TranslationWrappers even on trusted data. We could make a 3rd condition that only does that but if we look at what ->validateValue() actually does (in addition to casting to string it does simple recursion through the array and makes sure final values are not NULL or non-scalar), I'm thinking we might as well use that so as to not complicate the code unnecessarily.

Maybe that could use a code comment explaining this...

stefan.r’s picture

StatusFileSize
new6.75 KB

Adding a comment and a test

stefan.r’s picture

Issue summary: View changes
Issue tags: -Needs issue summary update +Performance
StatusFileSize
new2.31 KB

Posting interdiff, updating IS and tagging Performance as we now have to translate 2 less strings on some page loads.

stefan.r’s picture

Issue summary: View changes
stefan.r’s picture

Title: Cast TranslationWrapper objects in config to string during extension installation » Cast TranslationWrapper objects in config to string during Config::save()
fabianx’s picture

+++ b/core/lib/Drupal/Core/Config/Config.php
@@ -209,18 +209,19 @@ public function save($has_trusted_data = FALSE) {
+    if (!$has_trusted_data && $this->typedConfigManager->hasConfigSchema($this->name)) {
+      // Ensure that the schema wrapper has the latest data.

It was exactly that what we wanted to optimize.

Would it not be enough to just use in the original else:

foreach ($this->data as $key => $value) {
        if ($value instanceof SafeStringInterface) {
          $this->data[$key] = (string) $value;
        }
}
stefan.r’s picture

it's a bit of extra complexity but if it gets rid of the performance penalty let's do that instead! Care to roll a quick patch @Fabianx?

stefan.r’s picture

Actually I am looking at this and I am struggling to understand how the current patch could possibly hurt performance? The string casting would need to happen anyway so all we're doing on top of that is a NULL and a scalar check:

+++ b/core/lib/Drupal/Core/Config/StorableConfigBase.php
@@ -155,15 +159,20 @@ protected function getSchemaWrapper() {
     elseif ($value !== NULL && !is_scalar($value)) {
       throw new UnsupportedDataTypeConfigException("Invalid data type for config element {$this->getName()}:$key");
     }

Also #31 would not work as we still need to recursively walk through the array :/

lauriii’s picture

I can profile the different options if someone has an idea of a good use case where that could be tested.

stefan.r’s picture

Double checked with @Fabianx about what the worry was, it may be unfounded as it's about things that are happening in ::castValue() only, which is still only called under the same conditions both before and after the patch (ie when !$has_trusted_data && $this->typedConfigManager->hasConfigSchema($this->name))

<Fabianx-screen> stefan_r: I think the expensive part was getting the schema definition as that needs to build all those caches and ask all the plugin managers (IIRC)

alexpott’s picture

  1. +++ b/core/lib/Drupal/Core/Config/Config.php
    @@ -209,18 +209,19 @@ public function save($has_trusted_data = FALSE) {
    -    if (!$has_trusted_data) {
    -      if ($this->typedConfigManager->hasConfigSchema($this->name)) {
    

    I'm not sure about this change.

  2. +++ b/core/lib/Drupal/Core/Config/Config.php
    @@ -209,18 +209,19 @@ public function save($has_trusted_data = FALSE) {
    -    if (!$has_trusted_data) {
    -      if ($this->typedConfigManager->hasConfigSchema($this->name)) {
    -        // Ensure that the schema wrapper has the latest data.
    -        $this->schemaWrapper = NULL;
    -        foreach ($this->data as $key => $value) {
    -          $this->data[$key] = $this->castValue($key, $value);
    -        }
    +    if (!$has_trusted_data && $this->typedConfigManager->hasConfigSchema($this->name)) {
    +      // Ensure that the schema wrapper has the latest data.
    +      $this->schemaWrapper = NULL;
    +      foreach ($this->data as $key => $value) {
    +        $this->data[$key] = $this->castValue($key, $value);
           }
    -      else {
    -        foreach ($this->data as $key => $value) {
    -          $this->validateValue($key, $value);
    -        }
    +    }
    +    else {
    +      // We fall back to simple validation. Even for trusted data we still want
    +      // to do this, as TranslationWrappers may exist in the configuration
    +      // object which we will need to cast to string.
    +      foreach ($this->data as $key => $value) {
    +        $this->data[$key] = $this->validateValue($key, $value);
           }
         }
    

    I don't like this change for an number of reasons:

    1. Validating config with !$has_trust_data feels wrong - changing that data feels even more wrong.
    2. Having a method called validateValue actually result in a data change is bad.

    There's been a lot of tension throughout the D8 cycle about putting too much logic inside the Config object and its storage classes - personally I think this is a step in the wrong direction for the reasons given above.

  3. +++ b/core/lib/Drupal/Core/Field/Plugin/Field/FieldType/BooleanItem.php
    @@ -33,8 +34,8 @@ class BooleanItem extends FieldItemBase implements OptionsProviderInterface {
    -      'on_label' => t('On'),
    -      'off_label' => t('Off'),
    +      'on_label' => new TranslationWrapper('On'),
    +      'off_label' => new TranslationWrapper('Off'),
         ) + parent::defaultFieldSettings();
    

    If these are the only translated things that end up in config why don't we just cast to string here.

stefan.r’s picture

Actually, looking at the parent issue it seems like there may be more t() strings that end up in config than just this one. I could dig deeper and find out how many of them?

Also the string cast in the default field settings (and wherever else we may use t()) would lead to unnecessary translation calls as it's being loaded on more than just the options screen, which is the only place where we actually need the translation of the default value.

And what about contrib? If they have output from t() that ends up in config they'll have a hard time tracking down these test failures. If we're going to break things by making t() return an object I'd rather we make that transition as painless as possible. So for point 2 I'd rather do a refactoring that makes this look less invasive and convoluted, because in the end all we're doing is a recursive string cast for BC reasons. If we can't do that anywhere it makes me wonder if we don't have a design problem elsewhere?

If the string cast in config/storage truly is still too "magical" and we're going to prevent people from passing objects with __toString() methods into config, let's at least make the BC break explicit and explicitly disallow it, i.e. by failing when they do so?

stefan.r’s picture

StatusFileSize
new4.91 KB
new4.88 KB

So we do have more config items holding output from t() than just the boolean item one, the boolean item is just the only one in core that's in ConfigInstaller::installDefaultConfig(), the others are elsewher.

Talked to @alexpott on IRC and we may want to keep the automatic string cast after all so as to make life easier for developers.

edit:

+++ b/core/lib/Drupal/Core/Config/StorableConfigBase.php
@@ -219,4 +220,29 @@ protected function castValue($key, $value) {
+      foreach (array_values($value) as $nested_value_key => $nested_value) {

this was wrong.. and the extra foreach was unneeded

Status: Needs review » Needs work

The last submitted patch, 38: 2562155-38.patch, failed testing.

alexpott’s picture

Status: Needs work » Needs review
StatusFileSize
new6.07 KB

I think we should cast on set. This means that $config->set('foo', t('bar')); $config->get('foo'); will return the string 'bar' rather than a safe string. To do this I've swapped some logic around in initWithData so we don't cast everything unnecessary when loading config from storage.

alexpott’s picture

I didn't post an interdiff with #40 cause it would have been bigger than the patch.

stefan.r’s picture

If this is green I think this is great too... so we get rid of validate_keys which is only FALSE on InitWithData so we don't have any problem with load()

  1. +++ b/core/lib/Drupal/Core/Config/Config.php
    @@ -8,6 +8,7 @@
    +use Drupal\Component\Utility\SafeStringInterface;
    

    not needed anymore

  2. +++ b/core/lib/Drupal/Core/Config/ConfigBase.php
    @@ -280,4 +277,27 @@ public function getCacheMaxAge() {
    +   *   The data will any safe strings cast to string.
    

    with

alexpott’s picture

StatusFileSize
new1008 bytes
new5.84 KB

Fixed nits from #42.

Status: Needs review » Needs work

The last submitted patch, 43: 2562155.43.patch, failed testing.

stefan.r’s picture

Random test failures in MigrateDrupal6Test and MigrateUserConfigsTest on PIFR and in MigrateUserTest::testUser / MigrateDrupal6Test::testDrupal on CI ... couldn't reproduce them locally.

alexpott’s picture

Well MigrateDrupal6Test runs all the tests including MigrateUserConfigsTest and MigrateUserTest so it's not surprising it has failed too.

I've run MigrateUserConfigsTest 100 times locally - no fails whatsoever and I can't see how this is introduced by this patch.

Status: Needs work » Needs review

alexpott queued 43: 2562155.43.patch for re-testing.

fabianx’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: +Needs change record

RTBC, looks great to me.

We'll need a CR though.

stefan.r’s picture

Title: Cast TranslationWrapper objects in config to string during Config::save() » Cast TranslationWrapper objects in config to string during Config::set()/setData()
stefan.r’s picture

Issue summary: View changes
Issue tags: -Needs change record

  • catch committed 4b78e91 on 8.0.x
    Issue #2562155 by stefan.r, joelpittet, alexpott: Cast...
catch’s picture

Status: Reviewed & tested by the community » Fixed

Committed/pushed to 8.0.x thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

jackbravo’s picture

I know this is closed, but right now I'm getting the same install error:

> PHP message: Uncaught PHP Exception Symfony\Component\Routing\Exception\RouteNotFoundException: "Route "search.view_node_search" does not exist." at core/lib/Drupal/Core/Routing/RouteProvider.php line 191

joelpittet’s picture

@jackbravo that doesn't look like this error at all. I couldn't find references to that route except in tests. Are you sure you have the latest 8.0.1 or dev release? If so try to run /core/rebuild.php or drush cr to rebuild the container.

jackbravo’s picture

Sorry, it probably isn't related. I saw the same error on some tests running for this issue (patch 15): https://www.drupal.org/pift-ci-job/27901

But probably it was because of something else because the next patch did not generate that error (patch 16): https://www.drupal.org/pift-ci-job/27903