Gracefully handle a non-existent cache service like cache.backend.memcache

Agree on that this is a problem, but also a problem that Drupal stores the cache in the database, then it should not. I rather would have preferred something like cache.backend.null at this phase - but that does not exist by default.

The config line that matters for the demonstration is $settings['cache']['default'] = 'cache.backend.memcache';. The same issue exists in Drupal 7. But to be fair, the memcache documentation explains the right order to do this.

For me, the right way should be that it works without I have to edit the files manually after a deploy. Also, when you have a workflow then you install you site over and over again - this is a real quality blocker.

I have tested this workaround, that seems to work well:

In settings.php:

if (!drupal_installation_attempted()) {
  $settings['cache']['default'] = 'cache.backend.memcache';
}

Thanks for the patch, @frankcarey, it's sorely needed. A question: can you clarify the purpose of your change to CacheFactory.php?

MiSc unfortunately your code makes too many assumptions. For instance: it doesn't work if you have an existing site to which you're trying to deploy a new cache backend.

@cilefen it's not an issue in D7 with the memcache module because the file that contains the cache class is specified right in settings.php and the bootstrap process ensures it's loaded, regardless of whether the module is enabled.

The patch in #1 no longer applies cleanly so here's a re-roll *without* the changes in ConfigFactory.php since I didn't really understand the logic there and it appears to be working for me without that.

I spoke too soon. This patch appears to resolve nothing for me. Continuing to look into it.

Ok, I think this is all that's needed.

I tried the patch and it wasn't working. The issue seemed to be the the site was calling cache.backend.chainedfast, which in turn reloaded the settings that were incorrect. I've attached a patch that addresses this.

This patch may warrant a test before I would approve it, but it works in context so far.

This will lead to outdated cache inside the database - if the other service is not available the cache will fallback on outdated cache entries.

@MiSc What direction do you think we should take this? I agree that outdated cache entries are no good, but I cannot see trying to wipe the cache every time we need to fall back on the database.

I envision this patch as more of a stop-gap measure. If your cache backend is incorrectly configured, this patch is designed to keep the site from breaking completely. To that end, I think adding an entry to the status report and maybe even an admin warning might be the best plan.

Tagging the issue, and see if we could get some more thoughts about a solution for this.

So there are a handful of backend cache options available to us:

• cache.backend.apcu
• cache.backend.chainedfast
• cache.backend.database
• cache.backend.memory
• cache.backend.php

None of these are ideal candidates for fallback because they are actually caching. I think the best plan may actually be to expose NullBackend as a new cache.backend.null service. It is designed to 'short-circuit' the caching system so no actual cache is stored. It is also, by definition, a 'stub' meaning I do not know if it is complete enough to operate in this capacity, though it really needs to be only good enough to deliver the message that the caching configuration is broken.

I hacked together this patch that adds the cache.backend.null service and also brings in a notice on the status report if things are broken. This is rolled against 8.2, so it'll eventually need to be re-rolled on 8.4 to get committed.

+1 for dhansens solution, also make sense to have 'null' for caching then the choosen service does not exist. And we do not want random behaviours with outdated caches.

Missing newline in end.

Test failures don't seem related to this patch, maybe try running tests again. I'm also testing this locally.

Don't know why tests are failing but #25 works for me.

+++ b/core/lib/Drupal/Core/Cache/CacheFactory.php
@@ -80,6 +80,12 @@ public function get($bin) {
+    // Fall back to no caching if specified service is missing.
+    if (!$this->container->has($service_name)) {
+      $service_name = 'cache.backend.null';
+    }

Maybe we should still log something so people can fix it.

Working now with 8.3.x. The patch #25 is almost completly in Core, except for that one line.

Attaching patch for that one line in 8.3.x, as the patch in #25 doesn't work anymore.

Should've tested better...

I do fail even before the ChainedFastBackendFactory, in CacheFactory when getting the service (I guess...)

Attaching a patch which handles that, using the same logic as before in the other file. I'm not sure if that's the way to go, if that's dirty and shouldn't ever be done to Drupal-Core, but it works for me and might help others.

#35 Not working for me

Error Before:
[23-Aug-2017 18:54:22 America/Denver] Symfony\Component\DependencyInjection\Exception\ServiceNotFoundException: You have requested a non-existent service "cache.backend.memcache". Did you mean one of these: "cache.backend.database", "cache.backend.apcu", "cache.backend.php", "cache.backend.memory"? in .../public_html/core/lib/Drupal/Component/DependencyInjection/Container.php on line 157 #0 .../public_html/core/lib/Drupal/Core/Cache/CacheFactory.php(83): Drupal\Component\DependencyInjection\Container->get('cache.backend.m...')

Error after:
[23-Aug-2017 19:22:08 America/Denver] Symfony\Component\DependencyInjection\Exception\ServiceNotFoundException: You have requested a non-existent service "cache.backend.null". Did you mean one of these: "cache.backend.apcu", "cache.backend.php", "cache.backend.memory"? in .../public_html/core/lib/Drupal/Component/DependencyInjection/Container.php on line 157 #0 .../public_html/core/lib/Drupal/Core/Cache/CacheFactory.php(89): Drupal\Component\DependencyInjection\Container->get('cache.backend.n...')

Updated patch in 35 to use database instead of null. Null backend cache is not defined by default in any Drupal services.

@nikunjkotecha as discussed in #16-#22, the patch was previously set up to add the null cache service. The logic was that if you're intentionally trying to point the cache away from the database cache, then having it default to the database cache is obviously not the correct behavior and would be outdated anyways.

I'm not sure when that got taken out or why, probably around #34 where @SerkanB basically seemed to restart the patch from scratch (or was it meant to be an interdiff on the last patch?), but I think it should be added back in and will try to make time to reroll the patch in the near future for 8.x-4.x.

1. Replaced \Drupal::hasService() with $this->container->has().
2. Corrected coding standards errors.
3. Replaced cache.backend.database with cache.backend.null

I started this patch before @dhansen added #41, hopefully it still helps, but sorry if I'm just adding to the noise!

@dhansen, I missed a lot of discussion here and tried to update patch in a hurry. Apologies for that. For me as of now the priority was to get install working without changing anything in core. Which meant - not adding a service which is not there in Core and still use this for adding memcache support.

All patches since #25 seem to be noise deviating from the consensus reached in the discussion up to #25. Attached you'll find a reroll of #25 against 8.4.x.

Turns out we should default to the database backend, but fallback to null backend if the configured service doesn't exist.

Let's fix the code standards violations while we're at it.

While working on this I noticed CKEditorLoadingTest could do with some refactoring. Created #2908842: Replace deprecated function calls in CKEditorLoadingTest for that, which also has a patch.

Can we write a test for this functionality?

@legolasbo - #49 generates the following for us on /admin/config, /admin/reports/status, and possibly elsewhere:

Notice: Undefined index: default in system_requirements() (line 985 of core/modules/system/system.install)....

and

Notice: Undefined index: default in system_requirements() (line 988 of core/modules/system/system.install)....

This should resolve the Undefined index and added a test to check the cache factory falls back to null backend if the service doesn't exist.

1. +++ b/core/core.services.yml
   @@ -202,6 +202,9 @@ services:
   +  cache.backend.null:
   

   I guess this is good to save on memory, as opposed to using the memory backend. However, if this is going to be defined in core.services.yml it should be maybe just be removed from development.services.yml?

2. +++ b/core/lib/Drupal/Core/Cache/ChainedFastBackendFactory.php
   @@ -47,7 +47,11 @@ public function __construct(Settings $settings = NULL, $consistent_service_name
   +      $consistent_service_name = \Drupal::hasService($defaultCache) ? $defaultCache : 'cache.backend.null';
   

   The ChainedFastBackendFactory is already container aware, so no need to use \Drupal:: calls? Also, shouldn't this code live outs of the if () block? If a consistent service name that's invalid is passed in, it would still break.

3. +++ b/core/tests/Drupal/Tests/Core/Cache/CacheFactoryTest.php
   @@ -70,6 +70,36 @@ public function testCacheFactoryWithCustomizedDefaultBackend() {
   +    $custom_default_backend_factory = $this->getMock('\Drupal\Core\Cache\CacheFactoryInterface');
   

   This naming is confusing. The custom one defined in the settings should never be used right? so this mock would be better called $null_backend_factory? In which case would it just be simpler to avoid mocks altogether and just make sure an instance of NullBackend is returned?

The ChainedFastBackendFactory is already container aware, so no need to use \Drupal:: calls? Also, shouldn't this code live outs of the if () block? If a consistent service name that's invalid is passed in, it would still break.

We do, because this call happens in a constructor ...

This naming is confusing. The custom one defined in the settings should never be used right? so this mock would be better called $null_backend_factory? In which case would it just be simpler to avoid mocks altogether and just make sure an instance of NullBackend is returned?

I know this is coming from the other test functions, but well, better be better than consistent :)

Here are a good amount of additional fixes, variable names, documentation improvements etc.

Looks nice.

1. +++ b/core/core.services.yml
   @@ -202,6 +202,9 @@ services:
        class: Drupal\Core\Cache\MemoryBackendFactory
      # A special cache bin that does not persist beyond the length of the request.
   +  cache.backend.null:
   +    class: Drupal\Core\Cache\NullBackendFactory
   +  # This bin provides no actual caching and serves as a backup for unconfigured or incorrectly configured cache
      cache.static:
   

   I think these comments needs to be swapped. Main reason for NW.

2. +++ b/core/lib/Drupal/Core/Cache/CacheFactory.php
   @@ -80,6 +80,14 @@ public function get($bin) {
   +    // Fall back to the NULL cache backend in the edge case that the
   +    // configured backend is provided by a module that is not installed yet
   

   Maybe 'Fall back to the no-op cache backend ...' may make more sense?

3. +++ b/core/tests/Drupal/Tests/Core/Cache/CacheFactoryTest.php
   @@ -2,6 +2,9 @@
   +use Drupal\Core\Cache\CacheBackendInterface;
   +use Drupal\Core\Cache\CacheFactoryInterface;
   +use Drupal\Core\Cache\NullBackendFactory;
    use Drupal\Core\DependencyInjection\ContainerBuilder;
    use Drupal\Core\Cache\CacheFactory;
    use Drupal\Core\Site\Settings;
   

   In scope to alphabetize this?

4. +++ b/core/tests/Drupal/Tests/Core/Cache/ChainedFastBackendFactoryTest.php
   @@ -0,0 +1,63 @@
   +namespace Drupal\Core\Cache {
   +  function drupal_installation_attempted() {
   +    return FALSE;
   +  }
   +}
   

   Worth a comment about why this is needed? Think we also need a blank line before/after the function?

And for the sake of discussion, can/should we use the static cache instead of the null cache? I don't think so, but the thought popped into my head when I was looking at this.

I believe these issues can be addressed without patching core. Falling back to null or database for a non-existent cache service could cause major performance issues during site installs and deploys so a more ideal solution would be to autoload required classes and services in cases where the module is not installed and config exists in settings.php.

Since drupal bootstraps the service container to the database by default, the below configuration overrides this to use memcache as a backend for the container in addition to overriding the default configuration for cache_config, cache_bootstrap, and cache_discovery bins to prevent issues caused by race conditions and stale container and static cache data on site installs / deploys / config updates.

if (class_exists(\Composer\Autoload\ClassLoader::class)) {
    $class_loader = new \Composer\Autoload\ClassLoader();
    $class_loader->addPsr4('Drupal\\memcache\\', 'modules/contrib/memcache/src');
    $class_loader->register();
    $settings['container_yamls'][] = DRUPAL_ROOT . '/modules/contrib/memcache/memcache.services.yml';

    // Bootstrap cache.container with memcache rather than database.
    $settings['bootstrap_container_definition'] = [
      'parameters' => [],
      'services' => [
        'database' => [
          'class' => 'Drupal\Core\Database\Connection',
          'factory' => 'Drupal\Core\Database\Database::getConnection',
          'arguments' => ['default'],
        ],
        'settings' => [
          'class' => 'Drupal\Core\Site\Settings',
          'factory' => 'Drupal\Core\Site\Settings::getInstance',
        ],
        'memcache.config' => [
          'class' => 'Drupal\memcache\DrupalMemcacheConfig',
          'arguments' => ['@settings'],
        ],
        'memcache.backend.cache.factory' => [
          'class' => 'Drupal\memcache\DrupalMemcacheFactory',
          'arguments' => ['@memcache.config'],
        ],
        'memcache.backend.cache.container' => [
          'class' => 'Drupal\memcache\DrupalMemcacheFactory',
          'factory' => ['@memcache.backend.cache.factory', 'get'],
          'arguments' => ['container'],
        ],
        'lock.container' => [
          'class' => 'Drupal\memcache\Lock\MemcacheLockBackend',
          'arguments' => ['container', '@memcache.backend.cache.container'],
        ],
        'cache_tags_provider.container' => [
          'class' => 'Drupal\Core\Cache\DatabaseCacheTagsChecksum',
          'arguments' => ['@database'],
        ],
        'cache.container' => [
          'class' => 'Drupal\memcache\MemcacheBackend',
          'arguments' => ['container', '@memcache.backend.cache.container', '@lock.container', '@memcache.config', '@cache_tags_provider.container'],
        ],
      ],
    ];

      // Override default configuration for static cache bins. 

        $settings['cache']['bins']['bootstrap'] = 'cache.backend.memcache';
        $settings['cache']['bins']['discovery'] = 'cache.backend.memcache';
        $settings['cache']['bins']['config'] = 'cache.backend.memcache';

      // Use memcache as the default bin.
      $settings['cache']['default'] = 'cache.backend.memcache';
  }

What I do for this type of issues is configure memcache in a closure in settings.local.php and only enable it if available, a bit like:

$configure_memcache = function (array $settings) {
  if (!class_exists(MemcacheBackendFactory::class)) {
    return $settings;
  }
  // Then configure all the settings
  return $settings;
}

// When developing, this like makes it easy to turn everything off back to a default by just commenting it.
$settings = $configure_memcache($settings);

I use this in settings.php for redis:

if (!drupal_installation_attempted() && class_exists(PhpRedis::class)) {
  $settings['container_yamls'][] = 'modules/redis/example.services.yml';
  .
  .
  .

Note that there were some changes to class names and class constructors in Memcache 8.x-2.0-alpha7, and the example in #57 will need some adjustment.

1) 'memcache.config' is now 'memcache.settings'.
2) 'Drupal\memcache\DrupalMemcacheConfig' is now 'Drupal\memcache\MemcacheSettings'.
3) 'Drupal\memcache\DrupalMemcacheFactory' is now 'Drupal\memcache\Driver\MemcacheDriverFactory'.
4) The 'cache.container' service takes fewer arguments now. '@lock.container' and '@memcache.config' can be removed.

Here is a full snippet for context. (This is based on Acquia's version, so there are a few differences from #57)

if (getenv('AH_SITE_ENVIRONMENT') &&
  isset($settings['memcache']['servers'])
) {

// Check for PHP Memcached libraries.
  $memcache_exists = class_exists('Memcache', FALSE);
  $memcached_exists = class_exists('Memcached', FALSE);
  $memcache_module_is_present = file_exists(DRUPAL_ROOT . '/modules/contrib/memcache/memcache.services.yml');
  if ($memcache_module_is_present && ($memcache_exists || $memcached_exists)) {
    // Use Memcached extension if available.
    if ($memcached_exists) {
      $settings['memcache']['extension'] = 'Memcached';
    }
    if (class_exists(ClassLoader::class)) {
      $class_loader = new ClassLoader();
      $class_loader->addPsr4('Drupal\\memcache\\', 'modules/contrib/memcache/src');
      $class_loader->register();
      $settings['container_yamls'][] = DRUPAL_ROOT . '/modules/contrib/memcache/memcache.services.yml';
      // Bootstrap cache.container with memcache rather than database.
      $settings['bootstrap_container_definition'] = [
        'parameters' => [],
        'services' => [
          'database' => [
            'class' => 'Drupal\Core\Database\Connection',
            'factory' => 'Drupal\Core\Database\Database::getConnection',
            'arguments' => ['default'],
          ],
          'settings' => [
            'class' => 'Drupal\Core\Site\Settings',
            'factory' => 'Drupal\Core\Site\Settings::getInstance',
          ],
          'memcache.settings' => [
            'class' => 'Drupal\memcache\MemcacheSettings',
            'arguments' => ['@settings'],
          ],
          'memcache.backend.cache.factory' => [
            'class' => 'Drupal\memcache\Driver\MemcacheDriverFactory',
            'arguments' => ['@memcache.settings'],
          ],
          'memcache.backend.cache.container' => [
            'class' => 'Drupal\memcache\Driver\MemcacheDriverFactory',
            'factory' => ['@memcache.backend.cache.factory', 'get'],
            'arguments' => ['container'],
          ],
          'lock.container' => [
            'class' => 'Drupal\memcache\Lock\MemcacheLockBackend',
            'arguments' => ['container', '@memcache.backend.cache.container'],
          ],
          'cache_tags_provider.container' => [
            'class' => 'Drupal\Core\Cache\DatabaseCacheTagsChecksum',
            'arguments' => ['@database'],
          ],
          'cache.container' => [
            'class' => 'Drupal\memcache\MemcacheBackend',
            'arguments' => [
              'container',
              '@memcache.backend.cache.container',
              '@cache_tags_provider.container',
            ],
          ],
        ],
      ];
      // Override default fastchained backend for static bins.
      // @see https://www.drupal.org/node/2754947
      $settings['cache']['bins']['bootstrap'] = 'cache.backend.memcache';
      $settings['cache']['bins']['discovery'] = 'cache.backend.memcache';
      $settings['cache']['bins']['config'] = 'cache.backend.memcache';
      // Use memcache as the default bin.
      $settings['cache']['default'] = 'cache.backend.memcache';
      // Enable stampede protection.
      $settings['memcache']['stampede_protection'] = TRUE;
      // Move locks to memcache (This path may need to be modified)
      $settings['container_yamls'][] = 'memcache.yml';
      // Enable compression for PHP 7.
      $settings['memcache']['options'][Memcached::OPT_COMPRESSION] = TRUE;
    }
  }
}

I've tested patch #55. We are using Acquia Cloud Site Factory hosting, and with this patch applied site didn't install, although already installed sites work ok. We changeds this condition:

if (getenv('AH_SITE_ENVIRONMENT') &&
  isset($settings['memcache']['servers'])
 ) {

to

if (getenv('AH_SITE_ENVIRONMENT') &&
  isset($settings['memcache']['servers']) &&
  !drupal_installation_attempted()
 ) {

Not sure if this is the right approach, and not sure if this only impact on ACSF hosting or because something we have in our code, but I guess is good to mention here.

What about situations like using memcache replacement for lock service? Following https://git.drupalcode.org/project/memcache/blob/8.x-2.x/README.txt I added :
services:
# Replaces the default lock backend with a memcache implementation.
lock:
class: Drupal\Core\Lock\LockBackendInterface
factory: memcache.lock.factory:get

and now "drush site:install --existing-config --yes" as well as UI installation fails with:

The service "lock" has a dependency on a non-existent service "memcache.lock.factory". in Symfony\Component\DependencyInjection\Compiler\CheckExceptionOnInvalidReferenceBehaviorPass->processValue() (line 31 of /var/www/vendor/symfony/dependency-injection/Compiler/CheckExceptionOnInvalidReferenceBehaviorPass.php).

I think that this issue should be broader then just have one use case cover. Or we need a new issue to deal with non existent services on installation, something similar to #2863986: Allow updating modules with new service dependencies.

The way I handle this - and I suggest it should be more commonly used - is to recognize a site really has 2 different states: one during "build", where services only rely on the default (database) implementations, and one during "run" where all optional modules have been installed and the container can be rebuilt accordingly.

I do this using settings.build.local.php and settings.run.local.php and the (composer-based) deployment commands switch the symlink settings.local.php from one to the other.

Re-rolling patch 55 for 8.7.x

Patch #67 failed to apply with composer-patches, changes to sites/development.services.yml is out of drupal/core package, any idea how to fix this?

For the moment, remove changes to sites/development.services.yml.

Patch #69 seems to work great. thanks.

Correct coding standards errors in patch #67.

Re-rolling patch 55 for 8.7.x

#2429321: Verify that the configured service exists before calling it in CacheFactory was filed on 19 Feb 2015, which is similar to this and created earlier than this, let's mark this one as duplicate?

#2429321: Verify that the configured service exists before calling it in CacheFactory