Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.I don't have any code for this yet, but I want to start the discussion.
Lazy loading modules was discussed and rejected during Drupal 7 during the work on the hook registry, but since we backed out the hook registry, added system_info(), module_implements() caching I think it's worth going over it again early in the cycle, even if it's only to reject it again. Have been meaning to post this issue for ages, but I just saw a Drupal 6 site with 250 modules spending 500ms in module_implements() and a big chunk of time in drupal_load(), which reminded me.
There's a lot of pieces involved here, will try to describe how I think it could work:
We currently have a number of places where we call functions, where we already have some metadata about which module the function belongs to:
* For menu router callbacks there's the 'file' key. Current problems with this are that we don't allow a file key for title, access, and other callbacks, and modules swapping out these via hook_menu_alter() can currently assume their .module will be loaded. This is probably solvable, although it may mean more complexity when you implement hook_menu_alter() to replace callbacks.
* For theme hooks, we already store 'file' or similar, that ought to be extendable as well, hook_theme_registry_alter() - see above.
* For hooks, we cache module_implements() already, woo! Trouble with modules implementing hooks on behalf of other modules.
* For modules calling each other's function directly, for example node_load(), we have dependencies[] in .info. So if we always load dependencies of a module when loading the module, that's covered. Some modules are going to be loaded all the time, but that's what we have now anyway.
I may have missed some, but this is more or less the idea - know the entry points to calling functions belonging to a module, and ensure the module is loaded before we call them. Direct function calls rely on dependencies (no module_invoke() or drupal_load() hopefully).
The absolute best example of where this would help is thinks like image derivative generation - where we currently load every single module, and also execute silly hooks like system_init() that add CSS to the 'page' we're going to serve. It ought to be possible to make those a lot slimmer alongside some other changes to the bootstrap.
The worst example is going to be a complex node/user view or listing page with a tonne of blocks, we still might see good saving there though (for example we might well be able to skip _ui modules on nearly any user facing page).
Comments
Comment #1
catchDoh I completely forgot reasons we'd want to do this, in no particular order:
1. We'd save looping through and requiring up to 250-300 modules regardless of whether they're actually needed for that request. _drupal_bootstrap_full() can end up taking a good 10-20% of page execution time and memory even with APC.
2. We'd be loading an interpreting a lot less codes on sites that don't run an opcode cache. Possibly /much/ less than we would've got from patches like #345118: Performance: Split .module files by moving hooks although it would need to be figured out (but this time we have xhprof!)
3. Even with an opcode cache, there are (reasonably small but very measurable) CPU and memory costs from loading constants, functions and classes into memory just from requiring a file - see #664940: investigate general php memory consumption fixes for background.
4. The more modular Drupal becomes, the more modules sites have to install. There are going to be some global caches that we're unlikely to stop growing linearly the more modules we add (for example, the module info cache), but not loading the actual modules themselves on every request will help at least a bit with this.
5. This doesn't stop modules organizing code with includes for specific hooks, callbacks, or classes, in fact it's pretty much orthogonal to that side of things.
6. #679112: Time for system.module and most of includes to commit seppuku already identified some areas of core that would likely take advantage of this - for example actions.inc
Comment #2
Crell CreditAttribution: Crell commentedSubscribing. And of course it must be mentioned that anything that gets moved to a class gives us lazy-loading almost for free. I'd argue that any piece of free-standing logic that we can encapsulate into a class and lazy-load is a good thing, for a variety of reasons but easy lazy-load being a major one.
Comment #3
MustangGB CreditAttribution: MustangGB commentedSubscribe
Comment #4
Anonymous (not verified) CreditAttribution: Anonymous commentedsubscribe
Comment #5
XanoI've been thinking about a unified callback system (my approach used a class). Callbacks have a function, file and module name, which is all they need, especially if we do something like #1074140: Force include files to be in /path-to-module/includes.
Comment #6
tugis CreditAttribution: tugis commentedSubscribing
Comment #7
mauritsl CreditAttribution: mauritsl commentedLoading many modules have indeed a big impact on performance. APC helps, but cannot solve it completely. Sure this will help a lot, more than 10%, even 20% on larger sites I guess. Callbacks that do not render complete pages profit even more (not only imagecache, also think about xmlrpc/rest and various ajax callbacks).
The module_invoke(_all) functions can take advantage of the module_implements() caching to know which modules to load. We have to load all modules the first time to know which modules are implementing this hook (or create a function registry instead). The second time we know what to load.
Not something that cannot be changed.
Also think about dependencies. Should we load dependencies when we load the module? Do we want modules to call drupal_load() right before they start using functions from other modules? This is particulary important for modules implementing hook_boot() or hook_init().
About the 'file' key. When not given, can we asume it's in the .module file from the module providing that callback?
Comment #8
Crell CreditAttribution: Crell commentedWith the amount of code that has or will be moving to OO, how much do we still need to worry about here? Especially with the new ExtensionHandler service?
Comment #9
catchLazy loading individual modules seems a bit pointless at this point, but it'd be good to do the following if we could:
- have the extension handler responsible for loading modules - and only do this when a hook is invoked. To do this we'd have to use the extension handler for all hook invocations, something that's not currently the case.
- as a follow-on to that, if hooks aren't implemented by modules they don't need to be loaded at all, so potentially skip loading any modules at all if there's a request that fires just a couple of un-implemented hooks.
Comment #10
pounardLazy loading modules is a bad idea, what if I call a public method from a module B that belongs to module A on which no hooks have run yet?
EDIT: If everything ends being OOP, that'd be great as suggested upper, and this would never be a problem anymore. For hooks there's hook groups to solve that. But as of now, most modules still, and lot probably will continue to, provide a procedural API.
Comment #11
catchWe can load all modules on demand, but yeah lazy-loading individual modules is a rats nest.
Marking this won't fix, but opened #1905334: Only load all modules when a hook gets invoked for #9.
Comment #12
pounardWe could have continued this issue, all what have been said here are valid concerns and may orient the other issue?
Comment #13
Sylvain Lecoy CreditAttribution: Sylvain Lecoy commentedI think the comparison for OSGi is still relevant: http://groups.drupal.org/node/137609#comment-480894.
I have realized that we could port some of the concept in PHP, in the mean time, explores unknown development aspects in PHP until then: http://en.wikipedia.org/wiki/OSGi.