Below is the code for the readMultiple method of the CachedStorage class.

  /**
   * {@inheritdoc}
   */
  public function readMultiple(array $names) {
    $data_to_return = [];

    $cache_keys_map = $this->getCacheKeys($names);
    $cache_keys = array_values($cache_keys_map);
    $cached_list = $this->cache->getMultiple($cache_keys);

    if (!empty($cache_keys)) {
      // $cache_keys_map contains the full $name => $cache_key map, while
      // $cache_keys contains just the $cache_key values that weren't found in
      // the cache.
      // @see \Drupal\Core\Cache\CacheBackendInterface::getMultiple()
      $names_to_get = array_keys(array_intersect($cache_keys_map, $cache_keys));
      $list = $this->storage->readMultiple($names_to_get);
      // Cache configuration objects that were loaded from the storage, cache
      // missing configuration objects as an explicit FALSE.
      $items = [];
      foreach ($names_to_get as $name) {
        $data = isset($list[$name]) ? $list[$name] : FALSE;
        $data_to_return[$name] = $data;
        $items[$cache_keys_map[$name]] = ['data' => $data];
      }

      $this->cache->setMultiple($items);
    }

    // Add the configuration objects from the cache to the list.
    $cache_keys_inverse_map = array_flip($cache_keys_map);
    foreach ($cached_list as $cache_key => $cache) {
      $name = $cache_keys_inverse_map[$cache_key];
      $data_to_return[$name] = $cache->data;
    }

    // Ensure that only existing configuration objects are returned, filter out
    // cached information about missing objects.
    return array_filter($data_to_return);
  }

As you can see, it filters out "missing objects". Except it also filters out objects where the data, the valid data, simply evaluates to false as well.
And, naturally, also valid data that just actually IS false.

I'd like to know why? A few lines above, all configuration names that aren't present in the return from $this->storage->readMultiple() is set to FALSE. Fair enough. But shouldn't the array_filter be replaced with a strict evaluation to FALSE then? So that it doesn't throw out empty arrays and empty strings and 0, etc etc? You may say "but we don't support configuration without any keys" except you do. Language collections are full of empty arrays. Empty arrays for days.

Moreover, the read method doesn't filter anything. If read finds an empty array, it'll return an empty array. Unless, of course, readMultiple was called earlier and has cached the lie that the empty array is actually FALSE. This happens because CachedStorage wraps itself before wrapping DatabaseStorage or FileStorage. So the first time, the result, including empty arrays, are cached by $this->cache->setMultiple($items). But then it array_filters, and returns to itself. Suddenly, the empty arrays of before are turned into FALSEs. And those values are then cached. And those values are returned by a later call to read.

The fact that read() can have a different value based on whether readMultiple was called earlier or not is cause for concern. Disabling the cache gets in this class can return wholly different results. That can't be right. A cache can't transform the data it's saving.

This came to light when, using Config Filter, and specifically Config Ignore, I discovered it didn't ignore config translations. There were a bunch of items, and they kept popping up as "Changed". Because CachedStorage->FilteredStorage->DatabaseStorage returns an empty array (which is the actual, correct value also present in the database) but CachedStorage->CachedStorage->DatabaseStorage returns boolean FALSE due to its array_filtering and recursive nature (and caching of the boolean).

Adding an array_filter to FilteredStorage::readMultiple fixes it, but to me that just feels like breaking FilteredStorage instead of fixing CachedStorage.

Comments

Grayle created an issue. See original summary.

cilefen’s picture

Category: Support request » Bug report

I’m marking this a bug because support requests are sometimes ignored.

grayle’s picture

Status: Active » Needs review
StatusFileSize
new754 bytes

Thanks, I'm uploading a patch just to see how much stuff it breaks if we replace it with a strict comparison instead of array_filter. It still doesn't fix the core issue, but reworking how to handle missing configuration will probably break a whole lot more. At least now it should only "mess" with configuration where the value is boolean FALSE, and in that case it'll still return boolean FALSE so.

grayle’s picture

It's come to my attention in the issue in config_filter, that FileStorage also filters its results in ::readMultiple and doesn't in ::read. So perhaps the fix is to apply the same filtering to DatabaseStorage::readMultiple (and any other Storage class that doesn't already) instead of removing it.

I'll try that locally tomorrow to see if that also fixes the problem (I believe it will).

We just need to introduce some consistency and we'll be alright I think.

borisson_’s picture

Instead of rewriting this to a foreach, can we keep the array_filter, but add a specific filter function to it?

return array_filter($data_to_return, function ($it) {
  return $it !== FALSE;
});
grayle’s picture

Okay, so I took a look at the StorageInterface docblock and all implementations of it. It's not perfect.

StorageInterface definitions:

  /**
   * ...
   *
   * @return array|bool
   *   The configuration data stored for the configuration object name. If no
   *   configuration data exists for the given name, FALSE is returned.
   */
  public function read($name);

In short: if it exists in any form, return it. If not, return FALSE.

  /**
   * ...
   *
   * @return array
   *   A list of the configuration data stored for the configuration object name
   *   that could be loaded for the passed list of names.
   */
  public function readMultiple(array $names);

In short: if it exists in any form, make it a part of the returned list. If not, omit it from said list.

Let's see how each implementation of this interface implements these methods.

DatabaseStorage

DatabaseStorage::readMultiple

If it is found, it is unserialized and added to the return list. Empty string, empty array, anything.
If it is not found, it is omitted.
Pass

DatabaseStorage::read

If it is found, it is unserialized and returned. Empty string, empty array, anything.
If it is not found, FALSE is returned.
Pass

FileStorage

FileStorage::readMultiple

Uses the output from FileStorage::read in a loop, with a truthy check. Currently, because of the PHP bug mentioned below, this doesn't matter. Once that is fixed, this check will filter out valid yet empty configuration data.
Fail

FileStorage::read

If it is found, it is returned. Currently PHP doesn't treat empty YAML files (only comments for example) as valid. But the spec says they are, so once PHP fixes that this method will also return whatever it finds.
If the file can't be found, FALSE is returned.
Pass

InstallStorage

Extends FileStorage.

ExtensionInstallStorage

Extends InstallStorage.

TestInstallStorage

Extends InstallStorage.

NullStorage

NullStorage::readMultiple

Always returns an empty array, indicating no configuration is found.
Pass

NullStorage::read

Always returns an empty array, indicating empty configuration is found.
Fail

StorageReplaceDataWrapper

StorageReplaceDataWrapper::readMultiple

Wraps other Storages, just like CachedStorage. But because it doesn't cache missing configuration, it returns exactly whatever the wrapped storage gives it. And the check to see if it is cached is an isset(), not empty().
Can also wrap itself without issues.
Pass

StorageReplaceDataWrapper::read

Checks if it has it in its own cache with an isset() and either returns that or passes through to the wrapped storage.
Pass

CachedStorage

CachedStorage::readMultiple

Reads from wrapped storage, sets missing configuration (which isn't returned from valid Storages' readMultiple) to FALSE. Caches everything. Uses array_filter afterwards, removing falsy configurations from its return.
Can also not wrap itself due to this. On subsequent readMultiples it pollutes the cache with FALSEs where empty values should be, influencing the result from ::read.
Fail

CachedStorage::read

Reads from wrapped storage, unless it's in cache. Returns whatever it finds wherever it finds it, and caches it too.
Pass

Tomorrow (or soon anyway) I'll write two patches. One which brings everything in line with the current Interface definitions. And one which changes the interface definition to equate empty config with missing config, and then brings everything in line with that definition. See which one breaks less things. I'm hoping both will pass. Then there's further research: I checked all of Core's Storages. But there's Contrib too, and we need to see if this will impact them as well. There's a temporary (incorrect, array_filter-based) patch committed to config_filter so config_ignore can filter language Collections properly (which is full of empty config), which will be patched correctly if and when one of these fixes goes in. But there could be more implementations out there. And they may be wrong, or they may be right. So we'll need a, uh, Environmental Impact Report, if you will.

Then there's a choice to be made. Which solution will make it in? Or perhaps there's another solution out there I've missed. I'm just hoping a solution makes it in, clearly the current state is not ideal.

@borisson_: Sure thing.

grayle’s picture

Lost track of time, but here's the patch which fixes the implementations.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

larowlan’s picture

This needs tests and likely a re-roll

Came here from the random issue triage button on https://lendude.gitlab.io/bug-smash-initiative/

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

immaculatexavier’s picture

StatusFileSize
new1.49 KB
new1.47 KB

Rerolled patch #7 against 9.5.x. Kindly review

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

shubham chandra’s picture

StatusFileSize
new1.5 KB

Applied patch #16 in Drupal 10.1.x.

Status: Needs review » Needs work

The last submitted patch, 18: 3002457-18.patch, failed testing. View results

sandeepsingh199’s picture

StatusFileSize
new1.5 KB

Re-rolling patch #7 against 10.1.x. Kindly check & validate

sandeepsingh199’s picture

Status: Needs work » Needs review
smustgrave’s picture

Status: Needs review » Needs work
Issue tags: -Needs reroll +Needs Review Queue Initiative

This issue is being reviewed by the kind folks in Slack, #needs-review-queue-initiative. We are working to keep the size of Needs Review queue [2700+ issues] to around 400 (1 month or less), following Review a patch or merge request as a guide.

Tests and issue summary still need to happen before review.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.