Problem/Motivation

  • Composer 2 memory usage is much improved compared to Composer 1. However, comments in #3160933: [META] Things to figure out in order to use Composer 2 to perform automatic updates and in https://github.com/composer/composer/pull/8850 still indicate that a composer update drupal/core-recommended --with-all-dependencies --profile hits ~400-600 MB of peak memory usage, even when there aren't very many contrib modules in the project.
  • Drupal's current memory requirement is 64MB. We could potentially add a higher requirement for enabling auto-updates, but ideally, we could keep it below what is available on most shared hosting providers. Otherwise, many of the site owners most in need of automatic updates would be excluded from it.
  • As just one example, HostGator defaults their shared hosting to 128MB, and does not allow customers to raise it above 256MB.
  • #3159975: [policy, no patch] composer.phar vs. API (automatic updates) proposes to use Composer's PHP API within the PHP request that initiates the update rather than executing composer update in a separate process. Do we know if doing this affects peak memory needs? On the one hand, the rest of the PHP request also needs some memory, which then adds to the total. On the other hand, does the PHP API provide us with more granular control of the operations that are performed, and can we use that to reduce peak memory needs?

Steps to reproduce

Proposed resolution

Remaining tasks

  • Research what PHP memory limit is available on most shared hosting companies, especially the most commonly used ones. E.g., if 512MB or higher is available for say 80% of shared hosting sites, then that might be a reasonable minimum to require to enable auto updates.
  • Discuss and explore ideas for reducing peak memory needs. Especially if there's any way to get it below 256MB.

User interface changes

API changes

Data model changes

Release notes snippet

CommentFileSizeAuthor
#28 composer2memtest-v3.txt2.36 KBressa
#27 composer2memtest-v2.txt2.36 KBressa
#27 final-result-Composer-version-2.0-dev-(2.0-dev+071350286fb196722728511aecd00e1defa826c7)-2020-07-17-14:16:58.txt734 bytesressa
#27 final-result-Composer-version-2.0-dev-(2.0-dev+2d3905157d6e0d699446391446ab50f2eef772f2)-2020-07-30-19:06:51.txt764 bytesressa
#27 final-result-Composer-version-2.0.0-alpha2-2020-06-24-21:36:18.txt764 bytesressa
#27 terminal-results-Composer-version-2.0-dev-(2.0-dev+071350286fb196722728511aecd00e1defa826c7)-2020-07-17-14:16:58.txt148.49 KBressa
#27 terminal-results-Composer-version-2.0-dev-(2.0-dev+2d3905157d6e0d699446391446ab50f2eef772f2)-2020-07-30-19:06:51.txt148.56 KBressa
#27 terminal-results-Composer-version-2.0.0-alpha2-2020-06-24-21:36:18.txt148.34 KBressa
#24 terminal-results.txt148.35 KBressa
#24 composer2memtest.txt1006 bytesressa

Comments

effulgentsia created an issue. See original summary.

aaronmchale’s picture

aaronmchale
Primary language English
Location Edinburgh, Scotland
commented

Couple of initial thoughts:

  1. To help with context it's probably also worth knowing when Drupal's memory usage requirements were last evaluated, for instance of 64MB has been the recommended amount since Drupal 6, then maybe we could afford to raise it, given how much the market has evolved since then and how much lower the cost of RAM is.
  2. Additionally it's probably also worth noting what the default memory allocation is for PHP and what its memory requirements will be by the time of Drupal 10.

Each of these could be factors in deciding how to proceed.

heddn’s picture

heddn
Primary language English
Location Illinois
commented

The default droplet for Digital Ocean is now 1GB. And that's been the default for a few years. I haven't tracked Linode, so I'm not sure how recently they changed, but their entry level plan is also 1GB. So the age-old truism that we have to fit Drupal on a 512MB box seems to not be the case any more.

Secondly, early performance numbers seem to show that 500-600MB of memory is needed by composer alone to run an update of Drupal core. Let's add 128M for Drupal core. So a good place to test/play around with is a 700MB memory limit for running automatic updates via composer. That's not a small amount, but it also isn't out of the reach of smaller hosting either. Especially if we can bump the memory at runtime for the update process only and not require all php threads have that high of dedicated memory.

effulgentsia’s picture

effulgentsia commented

it's probably also worth noting what the default memory allocation is for PHP and what its memory requirements will be by the time of Drupal 10

The default has been 128M since PHP 5.2.1, and still is as of 8.0.0-alpha3.

The default droplet for Digital Ocean is now 1GB.

1GB is reasonable for VPS setups. However, old-style shared hosting is still a thing, and I suspect that many of the large shared hosting providers limit well below that. Even for newer-style managed cloud container providers, the cost increases for memory. E.g., FortRabbit's pricing goes from $5/mo for 128M to $30/mo for 512M, and then way up to get above 512M.

heddn’s picture

heddn
Primary language English
Location Illinois
commented

I wondered about shared hosting as I've never entered that world much. Thanks for posting those numbers. Would it be worth it to some folks if they wanted automated updates to pay the extra a month needed for memory? I could see it being worth it for some people. I'm not sure we'll be able to get memory down much below 500-700 and also use composer.

webchick’s picture

webchick
she/they
Primary language English
Location Vancouver 🇨🇦
commented

Here's the list that's first in Google results when you search "most popular shared hosts" https://www.cnet.com/news/best-web-hosting-providers-in-2020/ ("best" is not the same as "most popular" but let's start there).

Ok I'm tired of this now, someone else can take it from here. ;)

heddn’s picture

heddn
Primary language English
Location Illinois
commented

So, to me, it looks like a stretch that we can get the memory needs down into the shared hosting limits. Yeah, there might be an exception, but by and large, the resounding message is 64-128. Maybe 512 as a stretch. But no where near the 700 that would sorta be needed. Where does that leave us? Auto updates for all but the extreme edge case of shared hosting (if that)?

aaronmchale’s picture

aaronmchale
Primary language English
Location Edinburgh, Scotland
commented

Re #7: Is it still the case that the memory intensive part is the building or reading of the composer.lock file? If I remember correctly I believe that was the issue with Composer 1.

effulgentsia’s picture

effulgentsia commented

Would it be worth it to some folks if they wanted automated updates to pay the extra a month needed for memory?

I imagine that it would be worth it for some and not for others, and that that ratio would depend to some extent on both the ease of upgrading their hosting plan and the cost. For Drupal as a project, we'd prefer for as many sites as possible to stay secure, and we know that without auto-updates there's way too many sites not applying security updates in a timely way.

A possible saving grace for #6 is that increasingly, shared hosting companies are providing the option to upgrade to a managed plan with dedicated resources. That's much better than asking people to switch to some other VPS provider, because:

  • Switching hosting companies is a pain, and requires migration.
  • Site owners currently on shared hosting probably don't want to deal with the hassle of a VPS that requires them to have to regularly take actions to keep their OS and PHP versions up to date and everything else on the system secure.

However, taking a couple of the examples from #6:

  • DreamHost lets you upgrade from shared hosting to managed VPS, which retains all the benefits of shared hosting, but gives you dedicated RAM. It's $15/mo (if you pay month-to-month) for 1GB of total RAM (PHP can use most of that, but not all of that), as compared to $5/mo (if you pay month-to-month) for their cheapest shared hosting plan.
  • Hostinger provides a similar concept, but they call that their Cloud Hosting plans. Their cheapest one is for 3GB RAM, and depending on which price you look at is either $29/mo (undiscounted), $7.45/mo (with a discount that doesn't specify eligibility rules), or $15.90/mo (the renewal price). Whereas their most popular shared hosting plan is about a quarter to a third of each of those corresponding prices.

A next step here might be to see if the other providers in #6 as well as other major providers offer something similar, where by simply spending $10/mo or so more money, you could get the needed RAM, without having to switch providers or take on any new sysadmin responsibilities.

webchick’s picture

webchick
she/they
Primary language English
Location Vancouver 🇨🇦
commented

Yeah, agreed that if they need to switch to an un-managed VPS in order to auto-update their Drupal, they're plugging their Drupal security problems but also giving themselves N more attack vectors to worry about. :\ Not a net win...

Neat to know about Managed VPS as a thing, though. I might switch to that myself. :)

fkelly12054@gmail.com’s picture

fkelly12054@gmail.com commented

@webchick ... on inmotionhosting, shared plan my php.ini shows:
max_execution_time = 30 ; Maximum execution time of each script, in seconds
max_input_time = 60 ; Maximum amount of time each script may spend parsing request data
memory_limit = 512M ; Maximum amount of memory a script may consume (32MB)

however under the core section when I run phpinfo.php it shows a memory_limit field of 768M. This is for php 7.3.20.

Not that either will be sufficient for composer. But

In most of the updating recommendations I see on Drupal.org they are saying "don't run composer against your production server".

What I, and I think most shared server users, am trying to do is set up a local version using Wampserver, (or equivalent) that mirrors the production server as exactly as possible and then run composer updates there. Getting this set up properly can be a p.i.t.a. ... for instance on a typical local server set up you will have multiple versions of PHP scattered around and need to find the right one to change the memory limits setting on. But once you do it is easy and fast to run composer update to do both core and contrib updates. This should give a set of files that can then be synched back over to the production server copy of Drupal. Run update.php and you should be ready to go.

effulgentsia’s picture

effulgentsia commented

Thanks for the info on InMotion, @fkelly!

In most of the updating recommendations I see on Drupal.org they are saying "don't run composer against your production server".

That is the current recommendation. This issue is part of the Automatic Updates Initiative. #2940731: Automatic Updates Initiative overview and roadmap contains an overview of the initiative, and what's being currently proposed and explored for that is for Drupal to use Composer 2 to perform an automatic update directly on the production server. That's still very much work in progress, and this issue is for getting some more clarity on what that would mean in terms of being able to work on shared hosting.

effulgentsia’s picture

effulgentsia commented

I just found #2958021-47: Proposal: Composer Support in Core initiative, which is a comment from 2 years ago that looked into where Composer consumes the bulk of memory:

Right now, to install one module, composer is going to download about 110MB of metadata. When composer loads that metadata, json_decode uses up about 384MB of memory to load *just the packages that come with drupal, out of the box*.

That was for Composer 1. Not sure if that's changed for Composer 2.

webchick’s picture

webchick
she/they
Primary language English
Location Vancouver 🇨🇦
commented

There was talk at some point of running Composer-as-a-Service (and charging some fee for it), so we could offset the memory requirement to some super-beefed up server that the DA runs for people who aren't able to run Composer in production due to RAM limitations. Not sure if that's a viable path...

effulgentsia’s picture

effulgentsia commented

Has the peak memory situation improved with Drupal 9's drupal/recommended-project? Here's what I'm seeing on my MacBook, using https://getcomposer.org/download/2.0.0-alpha2/composer.phar.

First, create a new project for Drupal 9.0.1:

php composer2-alpha2.phar create-project drupal/recommended-project recommended-project 9.0.1
cd recommended-project

Next, profile a dry run of updating it to 9.0.2:

php ../composer2-alpha2.phar update -v --dry-run --profile

Which outputs:

[4.1MiB/0.00s] > command: Drupal\Composer\Plugin\Scaffold\Plugin->onCommand
[4.4MiB/0.02s] Loading composer repositories with package information
[4.5MiB/0.02s] Updating dependencies
[113.4MiB/11.46s] Dependency resolution completed in 0.115 seconds
[58.6MiB/11.50s] Analyzed 11372 packages to resolve dependencies
[58.6MiB/11.50s] Analyzed 70557 rules to resolve dependencies
[58.6MiB/11.50s] Lock file operations: 0 installs, 4 updates, 0 removals
[58.6MiB/11.50s] Updates: drupal/core-composer-scaffold:9.0.2, drupal/core-project-message:9.0.2, drupal/core:9.0.2, drupal/core-recommended:9.0.2
[58.6MiB/11.50s]   - Upgrading drupal/core (9.0.1 => 9.0.2)
[58.6MiB/11.50s]   - Upgrading drupal/core-composer-scaffold (9.0.1 => 9.0.2)
[58.6MiB/11.50s]   - Upgrading drupal/core-project-message (9.0.1 => 9.0.2)
[58.6MiB/11.50s]   - Upgrading drupal/core-recommended (9.0.1 => 9.0.2)
[59.0MiB/11.51s] Installing dependencies from lock file (including require-dev)
[59.3MiB/11.52s] Package operations: 0 installs, 4 updates, 0 removals
[59.3MiB/11.52s] Updates: drupal/core-composer-scaffold:9.0.2, drupal/core-project-message:9.0.2, drupal/core:9.0.2, drupal/core-recommended:9.0.2
[59.3MiB/11.52s]   - Upgrading drupal/core-composer-scaffold (9.0.1 => 9.0.2)
[59.3MiB/11.52s]   - Upgrading drupal/core-project-message (9.0.1 => 9.0.2)
[59.3MiB/11.52s]   - Upgrading drupal/core (9.0.1 => 9.0.2)
[59.3MiB/11.52s]   - Upgrading drupal/core-recommended (9.0.1 => 9.0.2)
[22.9MiB/11.54s] 28 packages you are using are looking for funding.
[22.9MiB/11.54s] Use the `composer fund` command to find out more!
[22.9MiB/11.54s] Memory usage: 22.9MiB (peak: 113.82MiB), time: 11.54s

According to that last line, this only required 114MiB of memory. Does that seem accurate, or is there something strange with my setup that's causing it to consume ~4x less RAM than others have reported?

heddn’s picture

heddn
Primary language English
Location Illinois
commented

I think it uses more memory when you limit it to _only_ drupal core. I did the same setup tasks as in #15 just now. Then added:

composer require drupal/pathauto drupal/group drupal/commerce drupal/admin_toolbar

Because the number of contrib dependencies also has an effect.

And with composer update drupal/core-recommended --with-all-dependencies -v --dry-run --profile, I got a much higher memory usage:

Memory usage: 65.34MiB (peak: 471.26MiB), time: 20.42s

ressa’s picture

ressa
he/him
commented

Using the recommended test method by The Composer developers, I get these results in Ubuntu 18.04, PHP 7.3.20, Composer 2.0.0-alpha2:

$ composer self-update --preview
$ composer create-project drupal/recommended-project:9.0.1
$ cd recommended-project
$ composer update --dry-run
$ COMPOSER_DISABLE_NETWORK=1 composer update -v --dry-run --profile
[26.8MiB/1.40s] Memory usage: 26.82MiB (peak: 117.67MiB), time: 1.4s
ressa’s picture

ressa
he/him
commented

Yes @heddn, you're right -- adding a few modules makes all the difference. Same procedure as above, only adding a few modules quadruples the memory usage:

$ composer require drupal/pathauto drupal/group drupal/admin_toolbar
[41.5MiB/3.48s] Memory usage: 41.46MiB (peak: 434.33MiB), time: 3.48s
effulgentsia’s picture

effulgentsia commented

It seems to depend on which contrib modules...

If I require most of the dependencies of the ones from #16:

php ../composer2-alpha2.phar require commerceguys/intl drupal/address drupal/admin_toolbar drupal/ctools drupal/entity drupal/entity_reference_revisions drupal/profile drupal/state_machine drupal/token drupal/variationcache

Then after a 

php ../composer2-alpha2.phar update drupal/core-recommended --with-all-dependencies -v --dry-run --profile

I still only see 172MiB of peak memory. There's also this line in the output that's interesting:

Analyzed 121318 rules to resolve dependencies

Which is 72% more than the 70557 from #15, and maybe is part of the reason for needing 50% more memory?

However, if I require any one of:

  • drupal/pathauto
  • drupal/group
  • drupal/commerce
  • drupal/inline_entity_form

Then, a 

php ../composer2-alpha2.phar update drupal/core-recommended --with-all-dependencies -v --dry-run --profile

analyzes over 600,000 rules and requires over 400MiB of RAM.

I wonder what it is about these modules that causes such a massive increase of rules that Composer needs to analyze.

ressa’s picture

ressa
he/him
commented

I can confirm the findings (Composer 2.0.0-alpha2 "preview"):

$ COMPOSER_DISABLE_NETWORK=1 composer update drupal/core-recommended --with-all-dependencies -v --dry-run --profile

Packages
commerceguys/intl drupal/address drupal/admin_toolbar drupal/ctools drupal/entity drupal/entity_reference_revisions drupal/profile drupal/state_machine drupal/token drupal/variationcache

Result:

[91.9MiB/1.79s] Analyzed 121320 rules to resolve dependencies
[32.8MiB/1.78s] Memory usage: 32.78MiB (peak: 172.31MiB), time: 1.78s

Packages
drupal/pathauto drupal/group drupal/admin_toolbar

Result:

[105.0MiB/3.58s] Analyzed 602159 rules to resolve dependencies
[44.9MiB/3.60s] Memory usage: 44.92MiB (peak: 456.77MiB), time: 3.6s
ressa’s picture

ressa
he/him
commented

Running Composer 2.0.0-alpha2 Drupal 9.0.1 > 9.0.2 with
$ COMPOSER_DISABLE_NETWORK=1 composer update drupal/core-recommended --with-all-dependencies -v --dry-run --profile:

  • drupal/pathauto
    [91.5MiB/1.72s] Analyzed 114021 rules to resolve dependencies
[32.4MiB/1.74s] Memory usage: 32.42MiB (peak: 167.27MiB), time: 1.74s
  • drupal/admin_toolbar
    [91.3MiB/1.69s] Analyzed 112582 rules to resolve dependencies
[32.3MiB/1.71s] Memory usage: 32.31MiB (peak: 165.41MiB), time: 1.71s
  • drupal/group
    [105.0MiB/3.58s] Analyzed 602146 rules to resolve dependencies
[44.8MiB/3.60s] Memory usage: 44.78MiB (peak: 456.7MiB), time: 3.6s
  • drupal/webform 
    [104.8MiB/3.55s] Analyzed 602141 rules to resolve dependencies
[44.8MiB/3.57s] Memory usage: 44.75MiB (peak: 456.46MiB), time: 3.57s
effulgentsia’s picture

effulgentsia commented

I'm now seeing the same on drupal/pathauto as #21, but when I wrote #19, I'm pretty sure that was the first one I tried and got the much larger result. Hm, not sure what changed.

effulgentsia’s picture

effulgentsia commented

There's a PR on https://github.com/zaporylie/composer-drupal-optimizations/pull/17 to make that plugin work on Composer 2. That PR is incomplete and fails to install on Composer 2. I'm working on a local copy that I now have installing correctly, but it's not making any dent in the numbers. I don't know yet if it's because that plugin solves a different problem space than what we're encountering in #20, or if my work in progress port of it is buggy. I'll post an updated PR there once I have something worth sharing.

ressa’s picture

ressa
he/him
commented
StatusFileSize
new composer2memtest.txt1006 bytes
new terminal-results.txt148.35 KB

Your work on zaporylie/composer-drupal-optimizations sounds interesting @effulgentsia, I will be ready to try it out, when there is something to test. I also wasn't sure if my results were consistent, possibly because I didn't run composer update to update composer.lock?

I wrote a little scripts which runs through the 30 most popular Drupal 9 modules. The modules fell into two categories, fast ones and slow ones, like we have already seen. I ran the process three times, with same results. I wonder what sets them apart?

Fast: 165 MB (17703 packages, 114015 rules)

  • admin_toolbar
  • backup_migrate
  • captcha
  • colorbox
  • ctools
  • devel
  • entity
  • entity_reference_revisions
  • google_analytics
  • imce
  • mailsystem
  • paragraphs
  • pathauto
  • redirect
  • search_api
  • smtp
  • token
  • views_bulk_operations
  • xmlsitemap

Slow: 456 MB (17704 packages, 602129 rules)

  • better_exposed_filters
  • block_class
  • bootstrap
  • features
  • honeypot
  • inline_entity_form
  • menu_block
  • metatag
  • panels
  • recaptcha
  • views_slideshow

Test update the 30 modules with Composer 2 with these commands. First, get Composer 2:
$ composer self-update --preview

Get the update script, rename (.sh files are not allowed to upload) and give it execute permissions:

$ wget https://www.drupal.org/files/issues/2020-07-30/composer2memtest.txt
$ mv composer2memtest.txt composer2memtest.sh
$ chmod a+rx composer2memtest.sh

Get Drupal 9, enter folder and run the update script:

$ composer create-project drupal/recommended-project:9.0.1 drupal901
$ cd drupal901
$ bash ../composer2memtest.sh

To build the list, copy and paste the resulting text from the terminal in a file called terminal-results.txt, and run these commands:

$ grep "\* drupal" terminal-results.txt > modules.txt
$ grep "Memory" terminal-results.txt | cut -d':' -f 3 | cut -d. -f 1 > memory.txt
$ paste memory.txt modules.txt | sort > final-result.txt
effulgentsia’s picture

effulgentsia commented

Thanks for #24!

To add to that, I found that composer require drupal/entity:^1.1 leads to a fast update (per #24), but composer require drupal/entity:^1.0 leads to a slow one.

What changed between 1.0 and 1.1 was that 1.0 had a require of drupal/core of "^8.7.7 || ^9" and 1.1 changed that to "^8.8 || ^9".

However, I tried replicating the exact same scenario on GitHub, but in that case composer require effulgentsia/composer-test:^1.1 and composer require effulgentsia/composer-test:^1.2 led to equally fast (low memory) updates.

So the change to drupal/entity's composer.json isn't the full story, but perhaps that in combination with something else that https://packages.drupal.org is doing for projects that change their composer.json across different tags?

naheemsays’s picture

naheemsays commented

@effulgentsia - could that be due to increased requirements checks? Maybe https://github.com/composer/composer/pull/8850 will help if it gets commited to Composer (they have a request for performance numbers in that pull request)

ressa’s picture

ressa
he/him
commented
StatusFileSize
new terminal-results-Composer-version-2.0.0-alpha2-2020-06-24-21:36:18.txt148.34 KB
new terminal-results-Composer-version-2.0-dev-(2.0-dev+2d3905157d6e0d699446391446ab50f2eef772f2)-2020-07-30-19:06:51.txt148.56 KB
new terminal-results-Composer-version-2.0-dev-(2.0-dev+071350286fb196722728511aecd00e1defa826c7)-2020-07-17-14:16:58.txt148.49 KB
new final-result-Composer-version-2.0.0-alpha2-2020-06-24-21:36:18.txt764 bytes
new final-result-Composer-version-2.0-dev-(2.0-dev+2d3905157d6e0d699446391446ab50f2eef772f2)-2020-07-30-19:06:51.txt764 bytes
new final-result-Composer-version-2.0-dev-(2.0-dev+071350286fb196722728511aecd00e1defa826c7)-2020-07-17-14:16:58.txt734 bytes
new composer2memtest-v2.txt2.36 KB

Great effulgentsia! Interesting with the differences in drupal/entity ...

You're right in your assumption @NaheemSays, the patch being investigated in Composer issue Filter dependent packages early #8850 fixes the issue, and all modules update using only 96 MB in 1.27 seconds, the results:

composer self-update --preview and composer self-update --snapshot

--preview: latest release of Composer version 2.0.0-alpha2 2020-06-24 21:36:18)
--snapshot: dev-release of Composer 2.0-dev (2.0-dev+2d3...2f2) 2020-07-30 19:06:51)
Same 19 fast and 11 slow modules as listed in #24:

  • 19 fast: 165 MB in 1.7 seconds
  • 11 slow: 456 MB in 3.6 seconds

19 fast modules (typical numbers)

  • Analyzed 17,704 packages to resolve dependencies
  • Analyzed 112,582 rules to resolve dependencies

11 slow modules (typical numbers)

  • Analyzed 17,705 packages to resolve dependencies
  • Analyzed 602,129 rules to resolve dependencies

php composer-test.phar

Updating with test-release for performance improvement Composer version 2.0-dev (2.0-dev+0713...6c7) 2020-07-17 14:16:58 from Filter dependent packages early #8850, all 30 modules peak at only around 96 MB memory usage in 1.27 seconds.

Typical numbers:

Memory usage: 30.12MiB (peak: 96.84MiB), time: 1.27s
Analyzed 3,626 packages to resolve dependencies
Analyzed 70,519 rules to resolve dependencies

I have tweaked the script a bit, to automate some of the tedious tasks. You can run it with these commands:

$ wget https://www.drupal.org/files/issues/2020-07-31/composer2memtest-v3.txt -O composer2memtest.sh
$ chmod a+rx composer2memtest.sh
$ composer create-project drupal/recommended-project:9.0.1 drupal901
$ cd drupal901
$ composer self-update --snapshot && bash ../composer2memtest.sh &> terminal-results.txt
$ composer self-update --preview && bash ../composer2memtest.sh &> terminal-results.txt

It will result in a file called something like final-result-Composer-version-2.0.0-alpha2-2020-06-24-21:36:18.txt.

To run the code with the composer-test.phar performance test version of Composer 2 from "Filter dependent packages early #8850", download the composer-test.phar:
$ wget https://seld.be/composer-test.phar

... and switch Composer version around line 23 of composer2memtest.sh, to make it look like this:

# COMPOSER_INSTANCE="composer"
COMPOSER_INSTANCE="php composer-test.phar"

Attaching result files for completeness.

ressa’s picture

ressa
he/him
commented
StatusFileSize
new composer2memtest-v3.txt2.36 KB

Updating test script, I uploaded a wrong version.

ressa’s picture

ressa
he/him
commented
Mixologic’s picture

Mixologic
He/Him/His
Primary language English
Location Portland, OR
commented

The main discrepancy between the groups of projects, is the slow group has a wider dependency net: drupal/core: ^8 || ^9.

This causes composer to consider every d8 and d9 version of core that exists - as of today, there are 167 releases of drupal core > 8.

Core itself has a very large dependency chain, thus having that many more releases means that composer creates hundreds of thousands of more rules to consider.

Which could help explain why that PR would help trim down the number of core versions that composer would consider before creating all the rules.

effulgentsia’s picture

effulgentsia commented
Title: Can Composer 2 peak memory usage for "composer update" be reduced to something supported by most shared hosting providers? » [PP-1-Upstream] Can Composer 2 peak memory usage for "composer update" be reduced to something supported by most shared hosting providers?
Status: Active » Postponed

Postponing this on https://github.com/composer/composer/pull/8850, which would solve this for us. If, for whatever reason, they don't merge it, we might need to explore alternatives, but for now, I think it makes sense to wait and see if they merge it.

bojanz’s picture

bojanz commented
Title: [PP-1-Upstream] Can Composer 2 peak memory usage for "composer update" be reduced to something supported by most shared hosting providers? » Can Composer 2 peak memory usage for "composer update" be reduced to something supported by most shared hosting providers?
Status: Postponed » Active

The PR has been merged and is included in the latest Composer pre-release (2.0-rc1).

effulgentsia’s picture

effulgentsia commented
Status: Active » Fixed

Yep, and I think that makes this issue fixed now. We can re-open it if/when we discover otherwise.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

ressa’s picture

ressa
he/him
commented
Related issues: +#3356739: Memory usage increases linearly when (un)installing modules via config import

I was reminded about this Composer/Drupal performance issue via #3356739: Memory usage increases linearly when (un)installing modules via config import, and ran a few commands with 55 modules. Composer peaks at ~150MiB and Drush at ~65MiB so it looks good:

Composer download

COMPOSER_DISABLE_NETWORK=1 composer require drupal/admin_toolbar drupal/allow_iframed_site drupal/autologout drupal/backup_migrate drupal/better_exposed_filters drupal/block_class drupal/cacheexclude drupal/captcha drupal/collapsiblock drupal/colorbox drupal/crawler_rate_limit drupal/easy_breadcrumb drupal/entity drupal/entity_reference_revisions drupal/facet_range_list_item drupal/facets drupal/facets_pretty_paths drupal/field_group drupal/field_token_value drupal/geofield drupal/google_analytics drupal/honeypot drupal/imce drupal/leaflet drupal/leaflet_more_maps drupal/login_security drupal/login_switch drupal/mailsystem drupal/matomo drupal/menu_block drupal/menu_migration drupal/metatag drupal/migrate_devel drupal/migrate_plus drupal/migrate_source_csv drupal/migrate_tools drupal/paragraphs drupal/pathauto drupal/perimeter drupal/rabbit_hole drupal/recaptcha drupal/redirect drupal/redis drupal/search_api drupal/search_api_solr drupal/security_review drupal/simple_sitemap drupal/site_audit drupal/smtp drupal/token drupal/views_argument_token drupal/views_bulk_operations drupal/views_flipped_table drupal/views_slideshow drupal/xmlsitemap -v --dry-run --profile
[...]
[16.4MiB/3.39s] Memory usage: 16.37MiB (peak: 59.99MiB), time: 3.39s

Composer update from Drupal 11.3.6 to 11.3.7

COMPOSER_DISABLE_NETWORK=1 composer update drupal/core-recommended --with-all-dependencies -v --dry-run --profile
[...]
[16.7MiB/5.93s] Memory usage: 16.75MiB (peak: 135.94MiB), time: 5.93s

Composer update from Drupal 11.3.6 to 11.3.7 + modules ("everything")

COMPOSER_DISABLE_NETWORK=1 composer update --dry-run --profile
[...]
[17.1MiB/6.73s] Memory usage: 17.1MiB (peak: 151.43MiB), time: 6.73s

Drush install

drush install admin_toolbar allow_iframed_site autologout backup_migrate better_exposed_filters block_class cacheexclude captcha collapsiblock colorbox crawler_rate_limit easy_breadcrumb entity entity_reference_revisions facet_range_list_item facets facets_pretty_paths field_group field_token_value geofield google_analytics honeypot imce leaflet leaflet_more_maps login_security login_switch mailsystem matomo menu_block menu_migration metatag migrate_devel migrate_plus migrate_source_csv migrate_tools paragraphs pathauto perimeter rabbit_hole recaptcha redirect redis search_api search_api_solr security_review simple_sitemap site_audit smtp token views_argument_token views_bulk_operations views_flipped_table views_slideshow xmlsitemap -y -vvv
[...]
[success] Module language has been installed. (Permissions - Configure) [7.29 sec, 48.55 MB]

Drush uninstall

drush pm:uninstall admin_toolbar allow_iframed_site autologout backup_migrate better_exposed_filters block_class cacheexclude captcha collapsiblock colorbox crawler_rate_limit easy_breadcrumb entity entity_reference_revisions facet_range_list_item facets facets_pretty_paths field_group field_token_value geofield google_analytics honeypot imce leaflet leaflet_more_maps login_security login_switch mailsystem matomo menu_block menu_migration metatag migrate_devel migrate_plus migrate_source_csv migrate_tools paragraphs pathauto perimeter rabbit_hole recaptcha redirect redis search_api search_api_solr security_review simple_sitemap site_audit smtp token views_argument_token views_bulk_operations views_flipped_table views_slideshow xmlsitemap -y -vvv
[...]
[success] Successfully uninstalled: admin_toolbar, allow_iframed_site [...] xmlsitemap [23.02 sec, 64.79 MB]