Follow-up of #2201785: Remove drupal_flush_all_caches() from installer
Problem/Motivation
ModuleHandler::Install() currently does not rebuild the routes, so that needs to happen manually. That's unlike everything else.
See discussion in parent issue.
Proposed resolution
Fix that somehow, while preventing a performance regression.
| Comment | File | Size | Author |
|---|---|---|---|
| #36 | module-install-2248297-36-interdiff.txt | 1.1 KB | berdir |
| #36 | module-install-2248297-36.patch | 13.28 KB | berdir |
| #34 | module-install-2248297-34-interdiff.txt | 3.76 KB | berdir |
| #34 | module-install-2248297-34.patch | 14.14 KB | berdir |
| #26 | module-install-2248297-26-interdiff.txt | 1.01 KB | berdir |
Comments
Comment #1
sunComment #2
catchNot too worried about a performance regression here. There's several bits in module_enable() where it behaves differently if you install one module vs. ten, we should just fix that.
Comment #3
dawehnerNote: we should do the same when we uninstall a module.
Comment #4
swentel commentedThe update module is not enabled on install, see #2307939: Update notifications are not enabled.
Comment #5
berdirReporting live from DrupalCon pre-sprints...
Due to a related issue, we found a considerably worse problem. And that is that we never dump the container after enabling a module. It only works because all the relevant usages (UI, drush) manually call drupal_flush_all_caches(). Without that, on the next request, we still get the old container/kernel, with the old module list and so on.
The attached patch changes that and also calls setRebuildNeeded(). This could cause some tests to fail.
Also related: #2313135: setting page_cache_without_database in settings.php prevents the container from being dumped
Comment #7
dawehnerWhat about just calling getContainer() or extract this logic from getContainer()?
Ah great this is already in a state, so it will happen in the next request, if needed.
Comment #8
berdir1. Related: #2313135: setting page_cache_without_database in settings.php prevents the container from being dumped. Not sure why we are not doing it inside initializeContainer() right now?
2. Yep, just not sure if we should call it per module, before we call hook_install(), just like every other thing (caches, schema is there, default config is imported...). Should be a pretty cheap thing to do?
Fixed the views test for now. As discussed, the broken code there that is now actually triggered correctly (because the module exists check return TRUE) is dead, that no longer makes sense. So removed that and cleaned up related definitions.
Comment #9
znerol commentedInterdiff in #8 should probably go into #2342807: DisplayPathTest methods enable menu_ui module when it is already enabled.
This looks very similar to the problem encountered over in #2331909-20: Move DrupalKernel::initializeCookieGlobals() into page cache kernel decorator, comment in #23:
Why don't we dump the container in
initializeContainer()? Maybe @neclimdul remembers the details?Comment #10
berdir@znerol: #2313135: setting page_cache_without_database in settings.php prevents the container from being dumped is doing exactly that now, seems to be working fine. Review of that would be great.
Comment #12
fabianx commentedI am pretty sure this needs a re-roll now ...
Comment #14
fabianx commentedThese are the only two lines of code that need to be left in this patch! Once those two lines are added and we have some test coverage, this is RTBC.
Comment #15
berdirDoing a bit more, trying to figure out if and which full cache rebuilds we can remove.
Comment #17
fabianx commentedThat would be awesome - if we could!
Comment #18
fabianx commentedOnly 4 fails, go berdir!
Comment #19
berdirDoesn't look too bad, but we might need to wait on #2230091: Route rebuilding is not guaranteed to finish in time for the next request.
Comment #21
berdirLooks like the error is related to instantiating all those controller/form objects during route rebuilding. This patch includes #2300131: EntityResolverManager instantiates objects unnecessarily, which fixes that.
Comment #25
berdirOk, so I spent a lot of time in InstallUninstallTest and finally figured it out.
It goes like this:
- FormBuilder
-- ModulesListForm::submitForm()
--- ModuleHandler::install(array('search')) => This rebuilds the container, and updates the current module handler. So we now have two containers in the system.
- FormBuilder::sendResponse()
-- HttpKernel::terminate() (this the old HTTP kernel from the old container)
--- RouterBuilder::rebuild() (This is the old service, so it has the old stuff but also the module handler, which has been updated already)
---- ModuleHander::getModuleDirectories() (this already returns search now)
---- ControllerResolver::getControllerFromDefinition() (for SearchPageRoutes)
----- SearchPageRoutes::create() now wants the 'search.search_page_repository' service, but it has the old container, and that doesn't have the search.module services yet.
------ Boom.
This only shows up in the error log, because it happens in terminate, after returning the response. Until it fails because the rebuild failed and it can't find an update service.
InstallUninstallTest runtime on my system:
HEAD: ~12m30
This patch: ~10m30
So it's not a huge difference, but it is one.
Comment #26
berdirForgot to mention how I fixed this. By using DrupalKernel instead, which is a synthetic service and survives the container rebuild.
Fixed the unit tests.
The type hint and unit tests are cheating, Neither HttpKernelInterface nor DrupalKernelInterface implement terminate(), that is separated in TerminableInterface. The unit test solves that by mocking the real object, so I kept that. Not a problem we have to solve here..
Comment #27
berdirComment #29
dawehnerJust in general: You could argue that this maybe belongs in the DrupalKernel but yeah out of scope of this issue.
Can we explain somewhere why we use the drupal kernel and not the HTTP handler? Code here is always potentially fragile
Meh
Comment #30
fabianx commented#26 Patch looks great, but why does FormBuilder needs to use DrupalKernel instead of HttpKernel and under what circumstances would other modules, e.g. contrib, need to use the same?
Comment #32
berdirOk, so we need #2230091: Route rebuilding is not guaranteed to finish in time for the next request and #2300131: EntityResolverManager instantiates objects unnecessarily to get in now. Postponing on that.
Comment #33
znerol commentedWe definitely have a problem with mid-request container rebuilds. This patch only fixes occurrences where this happens from within forms. We should not take that as granted though, therefore I think we need a different solution here.
Ironically only new style OO code is affected by this problem. In contrast in procedural code (e.g., hook implementations) a fresh version of a service is always fetched directly from the container.
While ugly, I believe something like this would be safer because it will also work when the container rebuild was performed outside of a form.
Comment #34
berdirI would be OK with making that change, but I think the change here makes sense anyway, it's now consistent with how we call terminate() in index.php.
Maybe we can discuss doing that change in a separate issue? Because the next question then is if we should do that elsewhere too, route rebuilding is probably just one of multiple cases where things could go crazy after a a container rebuild.
The actual reason that it is problematic I guess is that we update our live module handler to already know about the new module, but the other services that depend on the module list might not. There were discussions about moving install()/uninstall() to a separate service, then the old moduleHandler would at least be consistent. Which could have nasty side effects too.
The two other issues got committed, so here is a updated patch. Did some more cleanup and updated the RouterTest for something that now actually works. Also included a test for installing again. Wondering if that now also works in kernel tests, if yes, then we could possibly remove a lot of rebuild() calls from them.
Fixed reviews. @Fabianx: The why is explained in comment #26, #25 and now in the interdiff as well.
Note that the interdiff doesn't contain the stuff that was committed in #2300131: EntityResolverManager instantiates objects unnecessarily.
Comment #36
berdirDeleted a bit too much, we still need the reinject part in one case.
Comment #37
tstoecklerProbably in a follow-up, but there are lots of $this->rebuild() calls in KernelTestBase tests which can be removed with this.
Comment #38
alexpott@Berdir rocks!
This change is fantastic.
Comment #40
catchYes this is one of my favourite patches for a very long time. Took us the past three years to get to the point where this was viable.
Yep.
Yep.
Yep.
Yep yep yep yep.
Committed/pushed to 8.0.x, thanks!
Comment #41
fabianx commented@Berdir: On that reInjectMe: I assume this is done manually right now?
Can't we have an old kernel subscriber that informs all the services with the new injected services.
I now that injection works via __construct, but if we just theoretically made all objects be like:
and have the container keep track of all objects implementing InjectableInterface would that it not make possible to generically re-inject all services from the container?
Without using synthetic services at all ...
Comment #42
znerol commentedI'd rather prefer to get rid of mid-request container rebuilds in the long run. This should be the responsibility of a (headless?) installer.
Comment #43
fabianx commentedWorks for me! - If that is not a requirement, it was just an idea how to solve it more generically.
Comment #44
berdirThat seems like a rather unrealistic dream :)
Note that I did not remove reinjectMe() here. I removed one of two use cases, as we no longer need to do additional, manual flushing after installing a module.
Comment #45
znerol commentedOh, and there is still #2285621: Remove persist-tag and replace it with a container.state_recorder service where it is pointed out why we should not delegate state recording/restoring to the services themselves.