Problem/Motivation

Classes that need to rebuild the container must interact with phpstorage() directly. This unnecessarily couples them to the implementation of the compiled container.

Proposed resolution

- Ensure that container storage is completely independent from any other things.

Remaining tasks

- Port patch from #2497243: Replace Symfony container with a Drupal one, stored in cache

User interface changes

- None

API changes

- API addition:

\Drupal::service('kernel')->invalidateContainer();

Data model changes

- None

Original report

There is no API for triggering a container rebuild. I propose that we add a new event to signal when the current container is invalid and a listener that performs whatever necessary to invalidate the container, so that it isn't used on subsequent requests.

Container rebuilds are mostly needed during module installs, where ModuleInstaller calls

   $this->kernel->updateModules($module_filenames, $module_filenames);

Which leads to the kernel calling `save` on it's instance of PhpStorageInterface. The new container is created on the same request, and is used for the duration.

The kernel also has a `rebuildContainer` method, but from what I can tell, it's only ever called in the installer and in tests. It uses the same code path as before that saves and loads the new container in-request.

The only other change I know of that needs to trigger a container rebuild is changing the language, but I imagine there are or will be more.

Both ConfigurableLanguageManager and ConfigSubscriber do this (not sure why they both exists) by interacting directly with the php storage object for the service container:


PhpStorageFactory::get('service_container')->deleteAll();

In this case, the container is being deleted from disk to mark that it needs to be rebuilt on the next request. No attempt is made to rebuild it during the same request.

There are two different use cases here for needing to rebuild the container that are only defined by our current implementation. I'm not sure if both uses cases can be covered by a new event, but the latter case of invalidating the container to be rebuilt on the next request definitely can.

This would decouple any class that needs a rebuild from PhpStorageInterface, which is a very specific implementation detail of the current container, as well as provide a single place for the logic of how a container is invalidated.

This is especially important when considering issues like #2301163: Create a phpstorage backend that works with a local disk but coordinates across multiple webheads, where calling `deleteAll` on a php storage object doesn't invalidate the container, and a counter needs to be incremented instead.

Beta phase evaluation

Reference: https://www.drupal.org/core/beta-changes
Issue category Task, because nothing is really broken. (except for an edge case in updateModules that never was trigged in practice)
Issue priority Major because the invalidation service needs to be swappable to support multiple webservers. Currently PhpStorage is hardcoded all over core, but in reality it is an internal implementation detail of the Kernel where and how the container is stored.
Prioritized changes The main goal of this issue is scalability (preparation work), to reduce fragility (by isolating internals into the kernel) and to help a critical, which uses a different invalidation service. (database)
Disruption Potentially disruptive (in the future) for drush or projects like registry_rebuild that relied on PhpStorage to delete the service container directly. Currently the storage does not change, so in this patch: Not disruptive, however contrib should maybe be informed of the new API?

Comments

fabianx’s picture

fabianx’s picture

Assigned: Unassigned » fabianx
fabianx’s picture

Title: Add new event for handling container rebuilds » Add centralized container invalidation method
Issue summary: View changes
fabianx’s picture

Status: Active » Needs review
StatusFileSize
new6.78 KB

Here is the patch from that issue.

fabianx’s picture

StatusFileSize
new468 bytes
new6.75 KB

Uhm, #4 was fail as it should have used $this->storage() for now.

I checked that there are no more direct references to the PhpStorageFactory with the container directly.

dawehner’s picture

Just some quick feedback about the API.

+++ b/core/lib/Drupal/Core/DrupalKernel.php
@@ -357,7 +357,7 @@ public function getAppRoot() {
+  public function boot($rebuild = FALSE) {
     if ($this->booted) {
       return $this;

Mh I'm not sure whether we really need a rebuild parameter. This feels optional for me. Can't you also always use $kernel->invalidateContainer() and then boot()? Splitting it in two steps is not a bad idea, IMHO.

fabianx’s picture

#6: No, this is needed for being able to do core/rebuild.php.

Else this will quickly get really complicated, because without bootstrap container, how could you access the database or any other service to delete?

Or rather how can invalidateContainer() ensure that the container is invalidated, when settings might not yet be loaded.

Unless we want to do:

invalidateContainer() {
  if (!$this->booted) {
    $this->rebuildContainer = TRUE;
    return;
  }

  // Do the real invalidation.
}

That could work. Thoughts?

The last submitted patch, 4: add_centralized-2493665-4.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 5: add_centralized-2493665-5.patch, failed testing.

fabianx’s picture

Status: Needs work » Needs review
StatusFileSize
new7.01 KB
new8.84 KB

New approach using a flag if the container needs to be rebuild on next initialization or not.

I think that should be more correct in general anyway.

e.g. updateModules() if ever called before boot (no one in core) would have been broken, because if there would have been a cached container it would have been used.

dawehner’s picture

This looks better for me!

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

For now I think this is a sane step forward.

I'm not sure whether its a good idea to encourage more code to use the kernel directly instead of providing something in the middle which does the rebuilding, and is used by kernel.

alexpott’s picture

Status: Reviewed & tested by the community » Needs review
Issue tags: +Needs tests

Is the invalidate method testable using a unit test or a kernel test?

+++ b/core/lib/Drupal/Core/DrupalKernel.php
@@ -781,6 +788,11 @@ protected function initializeContainer($rebuild = FALSE) {
+    // The container was rebuild successfully.
+    if ($this->containerNeedsRebuild) {
+      $this->containerNeedsRebuild = FALSE;
+    }

Why bother with the if?

alexpott’s picture

Forgot to say that this is way better than reaching into PhpStorage from the config system, +1 to the approach.

jibran’s picture

Issue tags: +Needs beta evaluation

Can we please have a beta eval as well?

msonnabaum’s picture

Issue summary: View changes

Edited the recent issue summary addition so that we're not talking about things in terms of right and wrong.

msonnabaum’s picture

Any reason the event idea wasn't implemented? This is certainly an improvement, but I was hoping to avoid coupling having direct calls to the kernel like in the patch.

Also, it's not clear to me how this helps in the case of needing to increment a counter rather than deleting from php storage, since this just deletes the php storage directly in DrupalKernel.

fabianx’s picture

#17 I don't believe an event will work in the core/rebuild.php case or in the drush cr case, because it means you need to have the invalidation service loaded to invalidate the container.

That feels wrong, while isolating to the Kernel we can actually swap out the service.

Edit:

Talked a little more in IRC:

< Fabianx-screen> msonnabaum: The idea is to make that swappable.
< Fabianx-screen> msonnabaum: Once it is internal to the kernel (regardless if we have a bootstrap container or not) we can implement swappability without any overhead.
< Fabianx-screen> msonnabaum: And I think that is better than something listening to an event that is not even loaded yet at core/rebuild.php time.
< Fabianx-screen> e.g.
< Fabianx-screen> msonnabaum: So a module would miss most of the rebuild information.
fabianx’s picture

Issue summary: View changes
Issue tags: -Needs tests, -Needs beta evaluation
StatusFileSize
new7.01 KB
new3.08 KB
new11.88 KB

Thanks for all the reviews!

#13: Fixed.

Added tests and beta evaluation.

The last submitted patch, 19: add_centralized-2493665--tests.patch, failed testing.

fabianx’s picture

Assigned: fabianx » Unassigned
StatusFileSize
new3.58 KB

Adding correct interdiff for #19.

fabianx’s picture

StatusFileSize
new788 bytes
new11.86 KB

Removed the sleep(5), which was only necessary because my local opcache is set to 2s revalidate interval, so it saw the old settings.php still.

znerol’s picture

Status: Needs review » Reviewed & tested by the community

Test looks fine, thanks!

I also very much like the fact that this issue moved away from events. Container rebuilds are to avoid at all cost on production sites. Exposing a high-level API for rebuilding the container is not a good idea IMHO. I am fine with the new approach though.

dawehner’s picture

+1 for the RTBC

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs change record

Let's get a CR for this. So that if any custom/contrib code is calling PhpStorageFactory::get('service_container')->deleteAll(); they can discover it needs changing.

catch’s picture

Just nits.

  1. +++ b/core/includes/common.inc
    @@ -1295,8 +1295,10 @@ function drupal_flush_all_caches() {
    +  // Wipe the PHP Storage cache.
    

    Wipe the Twig Php Storage cache?

  2. +++ b/core/lib/Drupal/Core/DrupalKernel.php
    @@ -127,6 +127,13 @@ class DrupalKernel implements DrupalKernelInterface, TerminableInterface {
    +   * Whether the container needs to be rebuild the next time it is initialized.
    

    s/rebuild/rebuilt

  3. +++ b/core/lib/Drupal/Core/DrupalKernel.php
    @@ -695,11 +702,13 @@ public function updateModules(array $module_list, array $module_filenames = arra
    +    // containerNeedsRebuild flag so that a potentially cached container is not
    

    Does this need a self::?

  4. +++ b/core/lib/Drupal/Core/DrupalKernel.php
    @@ -781,6 +788,9 @@ protected function initializeContainer($rebuild = FALSE) {
    +    // The container was rebuild successfully.
    

    rebuilt.

  5. +++ b/core/lib/Drupal/Core/DrupalKernel.php
    @@ -998,15 +1008,33 @@ protected function persistServices(ContainerInterface $container, array $persist
    +    // Also wipe the PHP Storage caches, so that the container is rebuild
    

    rebuilt.

fabianx’s picture

Issue tags: -Needs change record

Created https://www.drupal.org/node/2528246 as change record draft.

Will work on the nits next.

fabianx’s picture

Status: Needs work » Reviewed & tested by the community
StatusFileSize
new2.43 KB
new11.88 KB

Fixed all the nits.

  • alexpott committed 5a528e5 on 8.0.x
    Issue #2493665 by Fabianx, dawehner, msonnabaum, catch: Add centralized...
fabianx’s picture

Status: Reviewed & tested by the community » Fixed
alexpott’s picture

Committed 5a528e5 and pushed to 8.0.x. Thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.