Determine cause of high memory consumption

Not sure why we need this issue as #2289201: [Meta] Make drupal install and run within reasonable php memory limits so we can reset the memory requirements to lower levels is already a meta for exactly this task?

Hi-jacking issue.

Installation via drush needs 96 MB - peak memory usage currently.

Profiling just container rebuild: (rm -f php storage file)

- The container compilation itself takes 8 MB - and cannot be freed, not even by unsetting the container and loading dumped version.
- YAML file loader adds itself already 2 MB (probably static caching).
- The dumping takes another 1.6 MB that cannot be freed.

Container rebuild and Kernel takes 13 MB.

Having the classes in memory itself takes 8 MB for up to container + kernel (spl_autoload_call memory usage).

That means the majority of memory 'wasted' is just classes loaded into memory.

Most memory spent is in Module installation from the Installer.

----

We can free 500 kB by unsetting $this->serviceProviders and $this->servicesYamls at least

And we could save potentially 2 MB for not using static cache for installation for Yaml files ...
( Need override for test bot as it else gets slow again. )

Installer findings when installed via drush:

Summary:

- 22 MB / 96 MB is just loaded classes (spl_autoload_call)

- install_begin_request: 16-17 MB (this is comparable with a normal container + kernel booting), the difference comes from preHandle doing some includes (1-2 MB) and the theme system being used
- install_run_tasks - Has 78 MB peak memory usage / 66 memory usage

Of this 17 MB are used for installing the base system, 3 - 6 MB (mu - pmu) for the install form and 52 - 53 MB for the batch process itself.

This goes directly to _install_module_batch, which just calls Drupal\Core\Extension\ModuleInstaller::install.

The memory usage here is:

- 31 - 60 MB to install the DefaultConfig
- 16 MB to updateKernel
- 9.5 MB to invokeHooks
- 5 MB to invokeAll
- PluginManager and onEntityTypeCreate provide 3 MB and 1.6 MB to this

_install_module_batch: User 42.75 MB Diff: 0.00 MB
_install_module_batch: Field 43.50 MB Diff: 0.75 MB
_install_module_batch: Entity Reference 43.50 MB Diff: 0.00 MB
_install_module_batch: File 44.00 MB Diff: 0.50 MB
_install_module_batch: Filter 45.25 MB Diff: 1.25 MB
_install_module_batch: Text 45.25 MB Diff: 0.00 MB
_install_module_batch: Options 45.25 MB Diff: 0.00 MB
_install_module_batch: Block 45.25 MB Diff: 0.00 MB
_install_module_batch: Custom Block 46.50 MB Diff: 1.25 MB
_install_module_batch: Node 48.75 MB Diff: 2.25 MB
_install_module_batch: Breakpoint 48.75 MB Diff: 0.00 MB
_install_module_batch: Text Editor 48.75 MB Diff: 0.00 MB
_install_module_batch: CKEditor 48.75 MB Diff: 0.00 MB
_install_module_batch: Color 48.75 MB Diff: 0.00 MB
_install_module_batch: Comment 50.50 MB Diff: 1.75 MB
_install_module_batch: Configuration Manager 50.50 MB Diff: 0.00 MB
_install_module_batch: Contact 50.75 MB Diff: 0.25 MB
_install_module_batch: Contextual Links 50.75 MB Diff: 0.00 MB
_install_module_batch: Datetime 50.75 MB Diff: 0.00 MB
_install_module_batch: Database Logging 51.00 MB Diff: 0.25 MB
_install_module_batch: Field UI 51.25 MB Diff: 0.25 MB
_install_module_batch: History 51.25 MB Diff: 0.00 MB
_install_module_batch: Taxonomy 52.50 MB Diff: 1.25 MB
_install_module_batch: Help 52.50 MB Diff: 0.00 MB
_install_module_batch: Image 52.75 MB Diff: 0.25 MB
_install_module_batch: Link 52.75 MB Diff: 0.00 MB
_install_module_batch: Custom Menu Links 54.25 MB Diff: 1.50 MB
_install_module_batch: Menu UI 60.25 MB Diff: 6.00 MB
_install_module_batch: Path 60.25 MB Diff: 0.00 MB
_install_module_batch: Quick Edit 60.25 MB Diff: 0.00 MB
_install_module_batch: RDF 60.25 MB Diff: 0.00 MB
_install_module_batch: Search 60.25 MB Diff: 0.00 MB
_install_module_batch: Shortcut 60.25 MB Diff: 0.00 MB
_install_module_batch: Toolbar 60.25 MB Diff: 0.00 MB
_install_module_batch: Views 84.50 MB Diff: 24.25 MB
_install_module_batch: Views UI 84.25 MB Diff: -0.25 MB
_install_module_batch: Tour 84.25 MB Diff: 0.00 MB
_install_module_batch: Standard 93.00 MB Diff: 8.75 MB

createConfiguration: 27.50 MB Diff: 0.00 MB Sum: 0.00 MB
updateModules: 30.25 MB Diff: 2.75 MB Sum: 2.75 MB
createConfiguration: 32.25 MB Diff: 1.25 MB Sum: 1.25 MB
updateModules: 35.75 MB Diff: 0.50 MB Sum: 3.25 MB
createConfiguration: 40.25 MB Diff: 1.75 MB Sum: 3.00 MB
_install_module_batch: START user
_install_module_batch: User 42.75 MB Diff: 0.00 MB Sum: 0.00 MB
_install_module_batch: START field
updateModules: 43.75 MB Diff: 1.00 MB Sum: 4.25 MB
createConfiguration: 43.75 MB Diff: 0.00 MB Sum: 3.00 MB
_install_module_batch: Field 43.75 MB Diff: 1.00 MB Sum: 1.00 MB
_install_module_batch: START entity_reference
updateModules: 43.75 MB Diff: 0.00 MB Sum: 4.25 MB
_install_module_batch: Entity Reference 43.75 MB Diff: 0.00 MB Sum: 1.00 MB
_install_module_batch: START file
updateModules: 44.00 MB Diff: 0.25 MB Sum: 4.50 MB
createConfiguration: 44.25 MB Diff: 0.25 MB Sum: 3.25 MB
_install_module_batch: File 44.25 MB Diff: 0.50 MB Sum: 1.50 MB
_install_module_batch: START filter
updateModules: 45.00 MB Diff: 0.75 MB Sum: 5.25 MB
createConfiguration: 45.25 MB Diff: 0.25 MB Sum: 3.50 MB
_install_module_batch: Filter 45.25 MB Diff: 1.00 MB Sum: 2.50 MB
_install_module_batch: START text
updateModules: 45.50 MB Diff: 0.25 MB Sum: 5.50 MB
createConfiguration: 45.50 MB Diff: 0.00 MB Sum: 3.50 MB
_install_module_batch: Text 45.50 MB Diff: 0.25 MB Sum: 2.75 MB
_install_module_batch: START options
updateModules: 45.50 MB Diff: 0.00 MB Sum: 5.50 MB
_install_module_batch: Options 45.50 MB Diff: 0.00 MB Sum: 2.75 MB
_install_module_batch: START block
updateModules: 45.75 MB Diff: 0.25 MB Sum: 5.75 MB
_install_module_batch: Block 45.75 MB Diff: 0.25 MB Sum: 3.00 MB
_install_module_batch: START block_content
updateModules: 46.00 MB Diff: 0.25 MB Sum: 6.00 MB
createConfiguration: 46.50 MB Diff: 0.50 MB Sum: 4.00 MB
_install_module_batch: Custom Block 46.50 MB Diff: 0.75 MB Sum: 3.75 MB
_install_module_batch: START node
updateModules: 47.50 MB Diff: 0.75 MB Sum: 6.75 MB
createConfiguration: 49.25 MB Diff: 1.50 MB Sum: 5.50 MB
_install_module_batch: Node 49.25 MB Diff: 2.75 MB Sum: 6.50 MB
_install_module_batch: START breakpoint
updateModules: 49.25 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Breakpoint 49.25 MB Diff: 0.00 MB Sum: 6.50 MB
_install_module_batch: START editor
updateModules: 49.25 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Text Editor 49.25 MB Diff: 0.00 MB Sum: 6.50 MB
_install_module_batch: START ckeditor
updateModules: 49.25 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: CKEditor 49.25 MB Diff: 0.00 MB Sum: 6.50 MB
_install_module_batch: START color
updateModules: 49.25 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Color 49.25 MB Diff: 0.00 MB Sum: 6.50 MB
_install_module_batch: START comment
updateModules: 49.25 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 50.50 MB Diff: 1.00 MB Sum: 6.50 MB
_install_module_batch: Comment 50.50 MB Diff: 1.25 MB Sum: 7.75 MB
_install_module_batch: START config
updateModules: 50.50 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Configuration Manager 50.50 MB Diff: 0.00 MB Sum: 7.75 MB
_install_module_batch: START contact
updateModules: 50.50 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 51.00 MB Diff: 0.50 MB Sum: 7.00 MB
_install_module_batch: Contact 51.00 MB Diff: 0.50 MB Sum: 8.25 MB
_install_module_batch: START contextual
updateModules: 51.00 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Contextual Links 51.00 MB Diff: 0.00 MB Sum: 8.25 MB
_install_module_batch: START datetime
updateModules: 51.00 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Datetime 51.00 MB Diff: 0.00 MB Sum: 8.25 MB
_install_module_batch: START dblog
updateModules: 51.00 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 51.00 MB Diff: 0.00 MB Sum: 7.00 MB
_install_module_batch: Database Logging 51.00 MB Diff: 0.00 MB Sum: 8.25 MB
_install_module_batch: START field_ui
updateModules: 51.25 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 51.50 MB Diff: 0.25 MB Sum: 7.25 MB
_install_module_batch: Field UI 51.50 MB Diff: 0.50 MB Sum: 8.75 MB
_install_module_batch: START history
updateModules: 51.50 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: History 51.50 MB Diff: 0.00 MB Sum: 8.75 MB
_install_module_batch: START taxonomy
updateModules: 51.50 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 52.50 MB Diff: 0.50 MB Sum: 7.75 MB
_install_module_batch: Taxonomy 52.50 MB Diff: 1.00 MB Sum: 9.75 MB
_install_module_batch: START help
updateModules: 52.50 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Help 52.50 MB Diff: 0.00 MB Sum: 9.75 MB
_install_module_batch: START image
updateModules: 52.50 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 53.00 MB Diff: 0.25 MB Sum: 8.00 MB
_install_module_batch: Image 53.00 MB Diff: 0.50 MB Sum: 10.25 MB
_install_module_batch: START link
updateModules: 53.00 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Link 53.00 MB Diff: 0.00 MB Sum: 10.25 MB
_install_module_batch: START menu_link_content
updateModules: 53.00 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Custom Menu Links 54.00 MB Diff: 1.00 MB Sum: 11.25 MB
_install_module_batch: START menu_ui
updateModules: 54.25 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 54.25 MB Diff: 0.00 MB Sum: 8.00 MB
_install_module_batch: Menu UI 60.50 MB Diff: 6.50 MB Sum: 17.75 MB
_install_module_batch: START path
updateModules: 60.50 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Path 60.50 MB Diff: 0.00 MB Sum: 17.75 MB
_install_module_batch: START quickedit
updateModules: 60.50 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Quick Edit 60.50 MB Diff: 0.00 MB Sum: 17.75 MB
_install_module_batch: START rdf
updateModules: 60.50 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 60.50 MB Diff: 0.00 MB Sum: 8.00 MB
_install_module_batch: RDF 60.50 MB Diff: 0.00 MB Sum: 17.75 MB
_install_module_batch: START search
updateModules: 60.50 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 60.50 MB Diff: 0.00 MB Sum: 8.00 MB
_install_module_batch: Search 60.50 MB Diff: 0.00 MB Sum: 17.75 MB
_install_module_batch: START shortcut
updateModules: 60.50 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 60.50 MB Diff: 0.00 MB Sum: 8.00 MB
_install_module_batch: Shortcut 60.50 MB Diff: 0.00 MB Sum: 17.75 MB
_install_module_batch: START toolbar
updateModules: 60.50 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Toolbar 60.50 MB Diff: 0.00 MB Sum: 17.75 MB
_install_module_batch: START views
updateModules: 60.50 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 84.75 MB Diff: 24.25 MB Sum: 32.25 MB
_install_module_batch: Views 84.75 MB Diff: 24.25 MB Sum: 42.00 MB
_install_module_batch: START views_ui
updateModules: 84.75 MB Diff: 0.00 MB Sum: 6.75 MB
_install_module_batch: Views UI 84.50 MB Diff: -0.25 MB Sum: 41.75 MB
_install_module_batch: START tour
updateModules: 84.50 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 84.50 MB Diff: 0.00 MB Sum: 32.25 MB
_install_module_batch: Tour 84.50 MB Diff: 0.00 MB Sum: 41.75 MB
_install_module_batch: START standard
updateModules: 84.50 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 88.75 MB Diff: 4.25 MB Sum: 36.50 MB
_install_module_batch: Standard 93.00 MB Diff: 8.50 MB Sum: 50.25 MB
updateModules: 93.25 MB Diff: 0.00 MB Sum: 6.75 MB
createConfiguration: 93.25 MB Diff: 0.00 MB Sum: 36.50 MB

This means views has 24.25 MB.

Of that 0.25 MB are calculateDependencies(), commenting out calculateDependencies() saves at most 5s.

But views $this->addCacheMetadata() is responsible for 15.75 MB, which brings Drupal over 32 MB.

Therefore ensuring addCacheMetadata is stored on disk and not re-calculated in config import should make Drupal being able to install with 32 MB again.

And that is the most likely reason.

Almost, just standard profile that does not want to install with 32 MB.

Also Config::save is very costly due to:

     if ($this->typedConfigManager->hasConfigSchema($this->name)) {
       // Ensure that the schema wrapper has the latest data.
       $this->schemaWrapper = NULL;
       foreach ($this->data as $key => $value) {
         $this->data[$key] = $this->castValue($key, $value);
       }
     }

the casting here.

Looks like there has nothing been fixed.

- A lot of static memory (3.9 MB) is used by the DefaultPluginManager when standard is installed.

diff --git a/core/profiles/standard/standard.install b/core/profiles/standard/standard.install
index f9466ff..704e72b 100644
--- a/core/profiles/standard/standard.install
+++ b/core/profiles/standard/standard.install
@@ -33,7 +33,8 @@ function standard_install() {
   user_role_grant_permissions(DRUPAL_AUTHENTICATED_RID, array('access comments', 'post comments', 'skip comment approval'));
 
   // Enable all permissions for the administrator role.
-  user_role_grant_permissions('administrator', array_keys(\Drupal::service('user.permissions')->getPermissions()));
+  // user_role_grant_permissions('administrator', array_keys(\Drupal::service('user.permissions')->getPermissions()));
+
   // Set this as the administrator role.
   $user_settings->set('admin_role', 'administrator')->save();

should be done as its done via user_modules_installed().

Removing $entity->uri() from ::permissions of Node and Filter saves another 5-10 MB and around 2.5 seconds in router rebuild time.

Down to 38 MB peak memory usage with that, getting close.

--

Container compilation takes around 6 MB in Classes loaded, but difficult to fix this as container rebuild is needed during module installation.

- directly in default plugin manager or in the classes it uses? those do a lot of docblock loading and parsing, so not surprising that we lose memory there?

- the user role snippet is there for a reason. user_modules_installed() is in many ways flawed, it doesn't work with dynamic permissions and more important in this case, standard provides the admin role, so was never run before. that can't be removed...

#13

- measured within the DefaultPluginManager::this->definitions variable.

- user_modules_installed() does exactly the same by now as standard.profile.

it just gets _all_ permissions and grants those to the admin role.

Hence unnecessary to do it twice.

@berdir
Could we mark roles as being admin roles and instead of assigning all permissions to it automatically, adapt
\Drupal\user\RoleStorage::isPermissionInRoles to check for a boolean flag on the admin role entity?

@Fabianx: Ah, didn't know that.

@dawehner: I've thought of that today as well, thought we could check the admin role setting, but a flag on the role itself could indeed be an advantage because then we have nothing additional that we need to load.

Could even help the permissions page because then we could opt out of displaying checkboxes for that role completeley. Sounds like being able to remove certain permissions from the admin role is no longer an option anyway, if user_modules_installed() now works they way it does.

#16 #15: That is an excellent idea! I opened #2435075: Implement admin role as a flag on the role storage, simplify permissions page, remove user_modules_installed to implement it.

I have used a different profiling tool (https://github.com/arnaud-lb/php-memory-profiler) to get more accurate results for the memory footprint.

In standard profile we use 8-17 MB (!!!) of loaded classes with opcache. So most of our memory foot print is the sheer amount of classes we load.

This is highly due to AnnotationsParser loading classes (all comes down to Discovery of plugin managers) and also due to frequent calls to:

class_exists() to check interfaces, is_class() and $reflection.

I think we have been using those functions a little bit too carelessly - without taking into account that they need to load the class in question.

Also DrupalKernelListeners needing to call the Listeners (and such load the classes).

And the route CompilerPass and the other compiler pass that has a tag and checks the interface.

--

Performance wise I will open an issue to disable all caches during installation - this brings down to the time to install standard.profile from 35s to 9s.

Caches are cleared too much to be useful during installation.

The major caches that would profit are those from pluginManagers, but as those are not incremental, they are cleared by the installer anyway.

- Config import sets caches that are never used again.
- Also saves a lot of tag invalidation.

Disabling caches by hack with returning FALSE from all DB functions to set brings down the memory to 33 MB peak for standard profile.

--

For Annotations we should use the same FileCache as for YML parsing, also do the follow-up to store parsed YML in the FileCache.

Both (YML and Annotations) are our no #1 memory problematic things besides the class loading.

So even storing in PhpStorage and loading the parsed array() / objects() in serialized form, will help a _lot_ already in reducing the amount of classes loaded that are not used.

--

We can also do something like:

diff --git a/core/lib/Drupal/Core/State/State.php b/core/lib/Drupal/Core/State/State.php
index 5da5241..117c249 100644
--- a/core/lib/Drupal/Core/State/State.php
+++ b/core/lib/Drupal/Core/State/State.php
@@ -85,6 +85,9 @@ public function getMultiple(array $keys) {
    * {@inheritdoc}
    */
   public function set($key, $value) {
+    if (isset($this->cache[$key]) && $this->cache[$key] === $value) {
+      return;
+    }
     $this->cache[$key] = $value;
     $this->keyValueStore->set($key, $value);
   }
@@ -94,6 +97,10 @@ public function set($key, $value) {
    */
   public function setMultiple(array $data) {
     foreach ($data as $key => $value) {
+      if (isset($this->cache[$key]) && $this->cache[$key] === $value) {
+        unset($data[$key]);
+        continue;
+      }
       $this->cache[$key] = $value;
     }
     $this->keyValueStore->setMultiple($data);

To prevent race conditions we could check the current value in the store before setting, but might be early. Saves 600 ms for standard.profile due to setRebuildNeeded() and other things no longer needing to re-write to the database.

Just some notes:

- The ContainerBuilder 'leaks' at least 1 MB due to keeping $this->compiler in memory - even though incremental compilation is not supported - as far as I can see.

So for cold cache we could save 1 MB by extending ContainerBuilder with a removeCompiler() method, but would need a symfony patch. (The dependency tree is mainly a big memory consumer).

Probably better is to re-load the dumped container (and unset the container builder) and just vary the class at runtime, so we can load several classes for several container rebuilds mid-request.
( and use a hash for doing so )

And some more notes:

- There currently is an upstream PHP "opcode" cache memory leak bug, which leaks at leaks 0.5 MB for YAML parsing.

https://bugs.php.net/bug.php?id=67111

In Inline.php:parse* is the continue 2, which leaks the memory.

Just to give an idea of what a drush installation needs in terms of memory when all autoloaded classes are excluded (I hacked it by loading all 1440 classes up front that are used during a standard install.):

- The classes loaded up front take 40 MB ( but without opcache, with opcache it is 14.53 MB )
- As we can see drush takes a fair amount of memory as well, also some bits are going via xhprof collecting data.
- The rest 21 MB (in memory_get_usage() more like 32 MB) is plugin definitions, container builder memory, entity saving, etc.

See the attached diagram.

( This has both the cache tags and database hacked to be no-op versions. It also has various optimizations already applied in this thread. This means that actual performance might be worse. It's possible e.g. we leak memory (PHP?) during preparation for cache sets or its cache_get memory stored statically somewhere. This however needs further investigation. )

Here is some more:

Disclaimer: This is a brain dump for future writing up.

Symfony thinks in their full stack very much of pre-generation of both container and twig templates and other things.

Drupal does not do so currently.

Perhaps we should adopt this idea and also pre-generate parts of our app.

The advantage of pre-generated caches like classmap, container, twig compiled templates, YAML files, annotations, is that it is file based. As such it is a great first check.

e.g. consider for twig compiled templates a chain caching of:

- check $ROOT/sites/default/files/php/twig first (currently stored template)
- check $ROOT/cache/php/twig second
-- Just if that mtime does not match, re-generate the compiled file. (the mtime is not checked for the dynamic files dir)
- always store whats in $ROOT/cache/twig to $ROOT/sites/default/files/php/twig after first loading, so we avoid the double check.

This we could do with YAML files, with annotated classes (possibly) and other things, hence vastly simplifying our system.

On the argument of, but mtime() is expensive the answer is: Yes, but re-generating the cache is more expensive.

The same could be reasonably done for the container.

E.g. currently its impossible to have Drupal use a read-only container unless you never enable any module in prod.

However it is possible to at least save the compilation() step in a controlled workflow.

Again the default container would be in:

- $ROOT/sites/default/files/php/service_container/service_container_prod/

(as it is now)

but the cached container could be found in.

- $ROOT/cache/php/service_container/service_container_prod_

The service_container_prod could then be loading the real container if the hash is no longer matching, what is inside in e.g. modules information.

That does not yet solve the chicken-egg problem of not writing, but it could save the compilation step - if the container was e.g. compiled on staging and deployed with the right hash.

Would need a $container->getHash() functions thats meaningful for Drupal.

Would just copy the pre-compiled container to service_container_prod on php storage, so skip re-compilation.

--

One important part here is that phpstorage usually just allows loading the file, however there is also a getFullPath() function, so a file_copy() or file_get_contents() is theoretically possible.

--

This pre-caching needs especially benchmarks for e.g. a FileCache of YAML files.

---------------------------

Another important aspect is that e.g. a view should know its block plugins and not have to load it all for Discovery.

We currently have very very long dependency chains, where you start with a config import and end with views_views_data(). We need to de-normalize our data more and ensure run time impact is less.

While caches can do a lot to hide this complexity, not putting defined caches into the system and allowing pre-generation will lead to cache clears or module installation being very very costly.

This is (as already stated) especially true for PluginManagers as the caches are:

- getAll() / cacheAll() / clearAll()

The incremental part of caching and e.g. more persistent storage is missing.

The installer is just one symptom of that and we talk only about 41 core modules, not like > 100 contrib modules installed.

Some more thoughts on the Installer and Cache:

- Cache (DatabaseBackend) should possibly really be using mysqli and async, same for cache tags.

Not going via PDO should make these operations quite a bit faster.

Adding a specific mysqli cache backend sounds good in general. We can keep dbtng for sites running on postgres/sqlite that want to use it.

Just unassigning issues that haven't been developed for a bit in the D8 queue.

Is this still relevant? It's been almost 10 years since the last update and a lot has changed since then. It's always great to profile and reduce memory consumption, but it's unclear exactly where this issue is headed. Marking PMNMI for anyone involved to provide direction / next steps.

There are a few issues covering most of what's in here either recently fixed or in progress. The underlying problems haven't changed much, but I think the new issues cover enough of the same ground.

In terms of this installer, #3416522: Add the ability to install multiple modules and only do a single container rebuild to ModuleInstaller made a big difference.

One possible next step for reducing memory usage in the installer/module installs would be #3486503: Add a core FileCache implementation that is database-backed, that would heavily reduce the cost of plugin discovery because it would be built incrementally as more modules are installed. We also have #3492233: [meta] Reduce memory/cpu/io cost of attribute discovery open more generally.

Smaller views data issue #3519269: ViewsViewsData loads all actions multiple times unnecessarily.

Bigger views data issues #3380145: ViewsData should not cache by language and #3454277: Allow fields to be opted out of views data

Also #3503843: RoutePreloader loads a lot of routes, optimize it, that is a considerable chunk of memory savings there. Need to get back to it and summarize the current state. I think it's fine to not keep this issue as a meta.

Maybe install system is a better home for this

#3535199: Saving untrusted config results in memory leak fixes the config casting part of this :)

#3365551: Add the notion of a 'configured layout builder block' to solve a number of content-editor and performance pain points and anything else that gets rid of derivatives would help with #14.