Problem/Motivation

When we install modules, we create a new container.
But because the previous container isn't reset, it has references to it that cant be GCd

Steps to reproduce

Install demo_umami with drush
Watch memory usage climb

Proposed resolution

Reset the old container during module installation.

Remaining tasks

User interface changes

N/a

Introduced terminology

N/a

API changes

N/a

Data model changes

N/a

Release notes snippet

N/a

Issue fork drupal-3492453

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

larowlan created an issue. See original summary.

larowlan’s picture

Status: Active » Needs review

.

godotislate’s picture

Status: Needs review » Active
Issue tags: +Performance
Related issues: +#3492233: [meta] Reduce memory/cpu/io cost of attribute discovery
catch’s picture

Status: Active » Needs work

Had a feeling this might be one of the problems but didn't know where to start on fixing it. Turns out it's https://git.drupalcode.org/project/drupal/-/merge_requests/10486/diffs?f...

What a one-liner!

Did some quick testing.

drush si -vvv (standard profile)

10.4 HEAD:

 [notice] Performed install task: install_finished [6.6 sec, 33.22 MB]

HEAD:

 [notice] Performed install task: install_finished [11.77 sec, 252.87 MB]

MR:

 [notice] Performed install task: install_finished [10.4 sec, 44.81 MB]

MR from #3416522: Add the ability to install multiple modules and only do a single container rebuild to ModuleInstaller

 [notice] Performed install task: install_finished [6.7 sec, 70.91 MB]

Combining the two issues:

 [notice] Performed install task: install_finished [6.13 sec, 40.29 MB]

Just for kicks, adding #2422681: Remove the automatic cron run from the installer on top (i.e. the three issues combined).

 [notice] Performed install task: install_finished [4.56 sec, 34.31 MB]

So this fixes the vast majority of the memory leaks during install, but if we also do multi module install and remove the cron run, we can reduce memory usage by a further 25% - roughly 5mb for multi install and 5mb dropping cron, and install time goes down by nearly 2/3rds overall too. It only gets us back to 10.4 speeds but we know the OOP hooks bc layer is very expensive and there are other issues to hopefully continue optimising that.

Installing Umami via drush complains a lot for me with [warning] The "field_block:node:page:field_body" block plugin was not found [7.12 sec, 69.44 MB] but that also happens in HEAD.

ghost of drupal past’s picture

This is absolutely amazing work.

For follow up, definitely not to hold this one up: Can we remove the container from services? I guess that's a big meta. There's #3272093: Cache bin names should be set from service tags, not the service name and #3483996: Remove lazy declaration and proxy class for cron and use service closure instead and then there's the KeyValueFactory and LazyContextRepository and there's an unused container.trait service and ... Anyways it'd be nice to untangle the resulting circular dependencies.

longwave’s picture

longwave’s picture

Status: Needs work » Needs review

alexpott changed the visibility of the branch 3492453-hacky-version to hidden.

alexpott changed the visibility of the branch 3492453-memory-leak-in to hidden.

alexpott’s picture

Status: Needs review » Reviewed & tested by the community

This looks like a pragmatic fix for 11.1 and we can fix this better in follow-ups and #3416522: Add the ability to install multiple modules and only do a single container rebuild to ModuleInstaller

alexpott’s picture

Issue summary: View changes
alexpott’s picture

Issue summary: View changes

  • catch committed 2a62fe22 on 11.1.x
    Issue #3492453 by larowlan, catch, longwave, alexpott, godotislate:...

  • catch committed 57061f4d on 11.x
    Issue #3492453 by larowlan, catch, longwave, alexpott, godotislate:...
catch’s picture

Version: 11.x-dev » 11.1.x-dev
Status: Reviewed & tested by the community » Fixed

Committed/pushed to 11.x and cherry-picked to 11.1.x, thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.