Comments

sun’s picture

catch’s picture

Not too worried about a performance regression here. There's several bits in module_enable() where it behaves differently if you install one module vs. ten, we should just fix that.

dawehner’s picture

Title: Ensure routes are rebuilt when enabling modules » Ensure routes are rebuilt when install modules

Note: we should do the same when we uninstall a module.

swentel’s picture

The update module is not enabled on install, see #2307939: Update notifications are not enabled.

berdir’s picture

Category: Task » Bug report
Priority: Normal » Major
Status: Active » Needs review
StatusFileSize
new1.15 KB

Reporting live from DrupalCon pre-sprints...

Due to a related issue, we found a considerably worse problem. And that is that we never dump the container after enabling a module. It only works because all the relevant usages (UI, drush) manually call drupal_flush_all_caches(). Without that, on the next request, we still get the old container/kernel, with the old module list and so on.

The attached patch changes that and also calls setRebuildNeeded(). This could cause some tests to fail.

Also related: #2313135: setting page_cache_without_database in settings.php prevents the container from being dumped

Status: Needs review » Needs work

The last submitted patch, 5: install-does-not-dump-2345611-5.patch, failed testing.

dawehner’s picture

  1. +++ b/core/lib/Drupal/Core/DrupalKernel.php
    @@ -636,6 +636,9 @@ public function updateModules(array $module_list, array $module_filenames = arra
    +      if ($this->containerNeedsDumping && !$this->dumpDrupalContainer($this->container, static::CONTAINER_BASE_CLASS)) {
    +        $this->container->get('logger.factory')->get('DrupalKernel')->notice('Container cannot be written to disk');
    +      }
    

    What about just calling getContainer() or extract this logic from getContainer()?

  2. +++ b/core/lib/Drupal/Core/Extension/ModuleHandler.php
    @@ -880,6 +880,8 @@ public function install(array $module_list, $enable_dependencies = TRUE) {
     
    +    \Drupal::service('router.builder')->setRebuildNeeded();
    +
    

    Ah great this is already in a state, so it will happen in the next request, if needed.

berdir’s picture

Status: Needs work » Needs review
StatusFileSize
new3.83 KB
new2.68 KB

1. Related: #2313135: setting page_cache_without_database in settings.php prevents the container from being dumped. Not sure why we are not doing it inside initializeContainer() right now?

2. Yep, just not sure if we should call it per module, before we call hook_install(), just like every other thing (caches, schema is there, default config is imported...). Should be a pretty cheap thing to do?

Fixed the views test for now. As discussed, the broken code there that is now actually triggered correctly (because the module exists check return TRUE) is dead, that no longer makes sense. So removed that and cleaned up related definitions.

znerol’s picture

Interdiff in #8 should probably go into #2342807: DisplayPathTest methods enable menu_ui module when it is already enabled.

+++ b/core/lib/Drupal/Core/DrupalKernel.php
@@ -636,6 +636,9 @@ public function updateModules(array $module_list, array $module_filenames = arra
     // and container.
     if ($this->booted) {
       $this->initializeContainer(TRUE);
+      if ($this->containerNeedsDumping && !$this->dumpDrupalContainer($this->container, static::CONTAINER_BASE_CLASS)) {
+        $this->container->get('logger.factory')->get('DrupalKernel')->notice('Container cannot be written to disk');
+      }
     }

This looks very similar to the problem encountered over in #2331909-20: Move DrupalKernel::initializeCookieGlobals() into page cache kernel decorator, comment in #23:

This patch accidentally removed every invocation of DrupalKernel::getContainer() and therefore the container was not dumped anymore.

Why don't we dump the container in initializeContainer()? Maybe @neclimdul remembers the details?

berdir’s picture

@znerol: #2313135: setting page_cache_without_database in settings.php prevents the container from being dumped is doing exactly that now, seems to be working fine. Review of that would be great.

fabianx’s picture

I am pretty sure this needs a re-roll now ...

Status: Needs review » Needs work

The last submitted patch, 8: module-install-2248297-8.patch, failed testing.

fabianx’s picture

Issue tags: +Needs tests
+++ b/core/lib/Drupal/Core/Extension/ModuleHandler.php
@@ -880,6 +880,8 @@ public function install(array $module_list, $enable_dependencies = TRUE) {
+    \Drupal::service('router.builder')->setRebuildNeeded();
+

These are the only two lines of code that need to be left in this patch! Once those two lines are added and we have some test coverage, this is RTBC.

berdir’s picture

Status: Needs work » Needs review
StatusFileSize
new3.12 KB

Doing a bit more, trying to figure out if and which full cache rebuilds we can remove.

Status: Needs review » Needs work

The last submitted patch, 15: module-install-2248297-15.patch, failed testing.

fabianx’s picture

Status: Needs work » Needs review

That would be awesome - if we could!

fabianx’s picture

Only 4 fails, go berdir!

berdir’s picture

Status: Needs review » Needs work

The last submitted patch, 19: module-install-2248297-18.patch, failed testing.

berdir’s picture

Status: Needs work » Needs review
StatusFileSize
new20.09 KB

Looks like the error is related to instantiating all those controller/form objects during route rebuilding. This patch includes #2300131: EntityResolverManager instantiates objects unnecessarily, which fixes that.

Status: Needs review » Needs work

The last submitted patch, 21: module-install-2248297-21.patch, failed testing.

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 21: module-install-2248297-21.patch, failed testing.

berdir’s picture

Status: Needs work » Needs review
Issue tags: -Needs tests
StatusFileSize
new26.87 KB
new4.19 KB

Ok, so I spent a lot of time in InstallUninstallTest and finally figured it out.

It goes like this:

- FormBuilder
-- ModulesListForm::submitForm()
--- ModuleHandler::install(array('search')) => This rebuilds the container, and updates the current module handler. So we now have two containers in the system.
- FormBuilder::sendResponse()
-- HttpKernel::terminate() (this the old HTTP kernel from the old container)
--- RouterBuilder::rebuild() (This is the old service, so it has the old stuff but also the module handler, which has been updated already)
---- ModuleHander::getModuleDirectories() (this already returns search now)
---- ControllerResolver::getControllerFromDefinition() (for SearchPageRoutes)
----- SearchPageRoutes::create() now wants the 'search.search_page_repository' service, but it has the old container, and that doesn't have the search.module services yet.
------ Boom.

This only shows up in the error log, because it happens in terminate, after returning the response. Until it fails because the rebuild failed and it can't find an update service.

InstallUninstallTest runtime on my system:
HEAD: ~12m30
This patch: ~10m30

So it's not a huge difference, but it is one.

berdir’s picture

Forgot to mention how I fixed this. By using DrupalKernel instead, which is a synthetic service and survives the container rebuild.

Fixed the unit tests.

The type hint and unit tests are cheating, Neither HttpKernelInterface nor DrupalKernelInterface implement terminate(), that is separated in TerminableInterface. The unit test solves that by mocking the real object, so I kept that. Not a problem we have to solve here..

berdir’s picture

Issue tags: +Performance

The last submitted patch, 25: module-install-2248297-25.patch, failed testing.

dawehner’s picture

  1. +++ b/core/lib/Drupal/Core/Extension/ModuleHandler.php
    @@ -999,7 +1000,9 @@ public function uninstall(array $module_list, $uninstall_dependents = TRUE) {
    +      \Drupal::service('router.builder')->setRebuildNeeded();
    

    Just in general: You could argue that this maybe belongs in the DrupalKernel but yeah out of scope of this issue.

  2. +++ b/core/lib/Drupal/Core/Form/FormBuilder.php
    @@ -61,11 +62,11 @@ class FormBuilder implements FormBuilderInterface, FormValidatorInterface, FormS
    +   * The kernel to handle forms returning response objects.
    ...
    +   * @var \Drupal\Core\DrupalKernelInterface
        */
    -  protected $httpKernel;
    +  protected $kernel;
    

    Can we explain somewhere why we use the drupal kernel and not the HTTP handler? Code here is always potentially fragile

  3. +++ b/core/tests/Drupal/Tests/Core/Form/FormTestBase.php
    @@ -117,7 +117,7 @@
        */
       protected $httpKernel;
    

    Meh

fabianx’s picture

#26 Patch looks great, but why does FormBuilder needs to use DrupalKernel instead of HttpKernel and under what circumstances would other modules, e.g. contrib, need to use the same?

Status: Needs review » Needs work

The last submitted patch, 26: module-install-2248297-26.patch, failed testing.

berdir’s picture

znerol’s picture

We definitely have a problem with mid-request container rebuilds. This patch only fixes occurrences where this happens from within forms. We should not take that as granted though, therefore I think we need a different solution here.

Ironically only new style OO code is affected by this problem. In contrast in procedural code (e.g., hook implementations) a fresh version of a service is always fetched directly from the container.

While ugly, I believe something like this would be safer because it will also work when the container rebuild was performed outside of a form.

diff --git a/core/lib/Drupal/Core/EventSubscriber/RouterRebuildSubscriber.php b/core/lib/Drupal/Core/EventSubscriber/RouterRebuildSubscriber.php
index 584b141..a93609d 100644
--- a/core/lib/Drupal/Core/EventSubscriber/RouterRebuildSubscriber.php
+++ b/core/lib/Drupal/Core/EventSubscriber/RouterRebuildSubscriber.php
@@ -43,7 +43,7 @@ public function __construct(RouteBuilderInterface $route_builder) {
    *   The event object.
    */
   public function onKernelTerminate(PostResponseEvent $event) {
-    $this->routeBuilder->rebuildIfNeeded();
+    \Drupal::service('router.builder')->rebuildIfNeeded();
   }
 
berdir’s picture

Status: Postponed » Needs review
StatusFileSize
new14.14 KB
new3.76 KB

I would be OK with making that change, but I think the change here makes sense anyway, it's now consistent with how we call terminate() in index.php.

Maybe we can discuss doing that change in a separate issue? Because the next question then is if we should do that elsewhere too, route rebuilding is probably just one of multiple cases where things could go crazy after a a container rebuild.

The actual reason that it is problematic I guess is that we update our live module handler to already know about the new module, but the other services that depend on the module list might not. There were discussions about moving install()/uninstall() to a separate service, then the old moduleHandler would at least be consistent. Which could have nasty side effects too.

The two other issues got committed, so here is a updated patch. Did some more cleanup and updated the RouterTest for something that now actually works. Also included a test for installing again. Wondering if that now also works in kernel tests, if yes, then we could possibly remove a lot of rebuild() calls from them.

Fixed reviews. @Fabianx: The why is explained in comment #26, #25 and now in the interdiff as well.

Note that the interdiff doesn't contain the stuff that was committed in #2300131: EntityResolverManager instantiates objects unnecessarily.

Status: Needs review » Needs work

The last submitted patch, 34: module-install-2248297-34.patch, failed testing.

berdir’s picture

Status: Needs work » Needs review
StatusFileSize
new13.28 KB
new1.1 KB

Deleted a bit too much, we still need the reinject part in one case.

tstoeckler’s picture

Probably in a follow-up, but there are lots of $this->rebuild() calls in KernelTestBase tests which can be removed with this.

alexpott’s picture

Status: Needs review » Reviewed & tested by the community

@Berdir rocks!

This change is fantastic.

  • catch committed 6ef8ba7 on 8.0.x
    Issue #2248297 by Berdir: Fixed Ensure routes are rebuilt when install...
catch’s picture

Status: Reviewed & tested by the community » Fixed

Yes this is one of my favourite patches for a very long time. Took us the past three years to get to the point where this was viable.

  1. +++ b/core/lib/Drupal/Core/Extension/ModuleHandler.php
    @@ -1022,8 +1025,6 @@ public function uninstall(array $module_list, $uninstall_dependents = TRUE) {
    -    drupal_flush_all_caches();
    

    Yep.

  2. +++ b/core/modules/config/src/Form/ConfigSync.php
    @@ -396,7 +396,6 @@ public static function finishBatch($success, $results, $operations) {
    -    drupal_flush_all_caches();
    

    Yep.

  3. +++ b/core/modules/system/src/Form/ModulesListForm.php
    @@ -511,7 +511,6 @@ public function submitForm(array &$form, FormStateInterface $form_state) {
    -      drupal_flush_all_caches();
    

    Yep.

Yep yep yep yep.

Committed/pushed to 8.0.x, thanks!

fabianx’s picture

@Berdir: On that reInjectMe: I assume this is done manually right now?

Can't we have an old kernel subscriber that informs all the services with the new injected services.

I now that injection works via __construct, but if we just theoretically made all objects be like:

interface InjectableInterface {
  // Nothing here as injectServices is generic.
}

abstract class InjectableObject implements InjectableInterface {
  public function __construct() {
    call_user_func_array(array($this, 'injectServices'), func_get_args());
  }
}

MyObject extends InjectableObject {

  public function injectServices($container, ) {
    $this->container = $container;
  }
}

and have the container keep track of all objects implementing InjectableInterface would that it not make possible to generically re-inject all services from the container?

Without using synthetic services at all ...

znerol’s picture

I'd rather prefer to get rid of mid-request container rebuilds in the long run. This should be the responsibility of a (headless?) installer.

fabianx’s picture

Works for me! - If that is not a requirement, it was just an idea how to solve it more generically.

berdir’s picture

That seems like a rather unrealistic dream :)

Note that I did not remove reinjectMe() here. I removed one of two use cases, as we no longer need to do additional, manual flushing after installing a module.

znerol’s picture

Oh, and there is still #2285621: Remove persist-tag and replace it with a container.state_recorder service where it is pointed out why we should not delegate state recording/restoring to the services themselves.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.