Problem/Motivation

Unfortunately \Drupal\plugin\PluginType\PluginTypeManager::getPluginTypes() does not use FileCache correctly and this results in a massive amount of extra memory usage for no reason during a site install.

Steps to reproduce

Install a site with many modules and plugin among them. Run site install from config using drush -vvv.

Before this change:
[success] Installation complete. User name: dru_admin User password: ********* [68.05 sec, 1.05 GB]

After this:
[success] Installation complete. User name: dru_admin User password: ******** [71 sec, 224.18 MB]

Proposed resolution

Do not vary the file cache by provider it is misunderstanding how file cache works.

Remaining tasks

User interface changes

None

API changes

None

Data model changes

None

Issue fork plugin-3416294

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

alexpott created an issue. See original summary.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented
Issue summary: View changes
alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented
Status: Active » Needs review
andypost’s picture

andypost
he/him
Primary language Russian
commented

Sounds like needs backport to 10.2

Berdir opened merge request !8

andypost’s picture

andypost
he/him
Primary language Russian
commented
Status: Needs review » Reviewed & tested by the community
berdir’s picture

berdir
Primary language German
Location Switzerland
commented

Change makes sense, no idea what's going on with those numbers, I managed to get it to report 1.6GB, although I really wonder if those numbers are accurate or if it's confused somehow. top seems to match that.

Before:

 [success] Installation complete.  User name: admin  User password: **** [61.66 sec, 1.61 GB]

After:

 [success] Installation complete.  User name: admin  User password: **** [78.85 sec, 226.16 MB]
alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented
Status: Reviewed & tested by the community » Needs work

I guess another question here is why does the plugin module cause 10mb of memory usage...

Before I applied this change:

 [info] plugin module installed. [137.17 sec, 108.92 MB]
 [info] productdb_client module installed. [138.22 sec, 109.52 MB]
 [info] productdb_paragraphs module installed. [139.23 sec, 119.06 MB]
 [info] publication_date module installed. [140.3 sec, 130.55 MB]
 [info] redirect module installed. [141.41 sec, 141.22 MB]
 [info] schema_metatag module installed. [142.44 sec, 149.17 MB]
 [info] schema_article module installed. [143.48 sec, 159.83 MB]

After

 [info] plugin module installed. [13.74 sec, 98.41 MB]
 [info] productdb_client module installed. [13.85 sec, 98.94 MB]
 [info] productdb_paragraphs module installed. [13.95 sec, 109.19 MB]
 [info] publication_date module installed. [14.08 sec, 109.74 MB]
 [info] redirect module installed. [14.22 sec, 105.78 MB]
 [info] schema_metatag module installed. [14.32 sec, 108.63 MB]
 [info] schema_article module installed. [14.43 sec, 110.34 MB]

Oh I see.... this module results in the container being stored in static cache many many times... that's why the memory ballooned. But it should not even go in once!

More work incoming.

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented
Status: Needs work » Needs review

The current patch now results in less memory usage all the time:

Before patch

 [info] plugin module installed. [14.28 sec, 98.56 MB]
 [info] prefetcher module installed. [14.45 sec, 96.48 MB]
 [info] preserve_page_cache module installed. [14.55 sec, 107.57 MB]
 [info] productdb_client module installed. [14.64 sec, 119.94 MB]
 [info] productdb_paragraphs module installed. [14.74 sec, 129.66 MB]
 [info] publication_date module installed. [14.85 sec, 141.18 MB]
 [info] redirect module installed. [14.98 sec, 152.28 MB]
 [info] schema_metatag module installed. [15.08 sec, 160.24 MB]
 [info] schema_article module installed. [15.18 sec, 173.11 MB]

Before the commit to fix #8

 [info] plugin module installed. [14.57 sec, 98.55 MB]
 [info] prefetcher module installed. [14.73 sec, 100.8 MB]
 [info] preserve_page_cache module installed. [14.83 sec, 108.28 MB]
 [info] productdb_client module installed. [14.96 sec, 110.28 MB]
 [info] productdb_paragraphs module installed. [15.07 sec, 110.36 MB]
 [info] publication_date module installed. [15.22 sec, 110.81 MB]
 [info] redirect module installed. [15.37 sec, 106.63 MB]
 [info] schema_metatag module installed. [15.49 sec, 109.59 MB]
 [info] schema_article module installed. [15.62 sec, 110 MB]

After

 [info] plugin module installed. [13.8 sec, 97.46 MB]
 [info] prefetcher module installed. [13.95 sec, 96.81 MB]
 [info] preserve_page_cache module installed. [14.07 sec, 97.65 MB]
 [info] productdb_client module installed. [14.19 sec, 99.53 MB]
 [info] productdb_paragraphs module installed. [14.32 sec, 99.62 MB]
 [info] publication_date module installed. [14.46 sec, 99.23 MB]
 [info] redirect module installed. [14.6 sec, 100.63 MB]
 [info] schema_metatag module installed. [14.73 sec, 98.8 MB]
 [info] schema_article module installed. [14.86 sec, 100.7 MB]
alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented

I think we should add tests of installing multiple modules with plugin types so we can test that the caches work as expected.

berdir’s picture

berdir
Primary language German
Location Switzerland
commented
Status: Needs review » Needs work

Rebased and reviewed.

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Issue tags: +Performance

WOW, those numbers! 🤯

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented
Status: Needs work » Needs review