Cache backend overrides does not work

Problem/Motivation

There is a common best practice in the community to override cache backend with Redis without having the module enabled. It took time while the current best practice was developed because replacing the default cache backend and container backend is not that simple (f.e. while the site is installed it cannot be done).

The problem is the current approach is not a bulletproof solution either. If you have a service provider which calls a cached service you get an error like:

In ContainerBuilder.php line 1060:
  You have requested a non-existent service "cache.backend.redis".  

I believe this issue could be only fixed by changing things in the core because currently, it seems there is no actual way for contrib modules to replace the default cache backends, or does and don'ts are not documented clearly.

Steps to reproduce

1. Add a service provider to a module that tries to access a cached service, like a module handler.

final class MYMODULEServiceProvider implements ServiceProviderInterface {

  /**
   * {@inheritdoc}
   */
  public function register(ContainerBuilder $container): void {
    /** @var \Drupal\Core\Extension\ModuleHandlerInterface $module_handler */
    $module_handler = $container->get('module_handler');
    try {
      $this_module_relative_path = $module_handler->getModule('MYMODULE')->getPath();
    }
    catch (\Exception $e) {
      throw new LogicException('Unable to identify installation path of this module.');
    }
  }

}

2. Enable the module.
3. Install the Redis module with Composer but do not enable it.
4. Add the above-linked configuration to settings.php which replaces the default cache backend and container cache with Redis.
5 Run drush cr and you should see the error above.

Technical analysis

So what is happening here?

1. $settings['container_yamls'] registers Redis services among the site-level service YAMLs
2. $settings['bootstrap_container_definition'] ensures that Redis is used in the "bootstrap_container". This feature is undocumented.
3. Service definitions from the "bootstrap_container" are not being used when the site container is being built here.
4. When the container is being built then first app-level services gets registered (everything that is core and enabled core/contrib modules). Site level services from YAMLs (like sites/default/services.yml) and service providers only get registestered after because they are considered as overrides.

and BUMMM!!!!

1. Redis module is not enabled there it is not registered as an app-level service.
2. Module handler (or any other cached app-level service) depends on Redis cache backend but the cached backend is not registered because
3.

$settings['container_yamls'] registers Redis services among the site-level service yamls

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet