Update dependencies composer/composer ^2 and composer/semver to ^3

Bumping to critical and adding various tags - we need to decide whether we're going to try to squeeze this into the beta or not - gets harder after that.

Do we want to go straight to only ^2.0 or might we want to allow ^1.9.1 | ^2.0?

#3122112: Update dependencies for Drupal 8.9 updates the lock file versions of composer/composer and composer/semver.

It doesn't look like Composer 2 will be stable in time for Drupal 9 release, as per https://github.com/composer/composer/issues/8726#issuecomment-618854271

I'm sure we'll do a alpha release soonish, then barring any onslaught of issues I'm hoping for a stable sometime late May or latest June

@xjm suggested in Slack that we could still aim for ^1.whatever || ^2

This bumps us to 2.0-dev release as there is no alpha let alone stable release yet, but let's see what happens.

Composer 2.0 does bring a new dependency, react/promise - does this have to go through the dependency evaluation process?

$ composer-lock-diff --no-links
+--------------------+-------+-------+
| Production Changes | From  | To    |
+--------------------+-------+-------+
| composer/semver    | 1.5.1 | 2.0.0 |
+--------------------+-------+-------+

+-------------------+--------+---------+
| Dev Changes       | From   | To      |
+-------------------+--------+---------+
| composer/composer | 1.10.5 | 994a5b4 |
| react/promise     | NEW    | v2.7.1  |
+-------------------+--------+---------+

@greg.1.anderson, I am not sure if the latter can be updated right now in isolation. On 9.0.x branch:

 ~/w/d/drupal   9.0.x *…  composer why composer/semver                                                                                                                       composer/composer  1.10.5     requires  composer/semver (^1.0)
drupal/core        9.0.x-dev  requires  composer/semver (^1.0)

It's clear we need to update `composer/composer` in order to use `composer/semver`. I tried it anyway, and as expected, I got an error.

Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for composer/semver ^2.0 -> satisfiable by composer/semver[2.0.0, 2.x-dev].
    - composer/composer 1.10.5 requires composer/semver ^1.0 -> satisfiable by composer/semver[1.x-dev].
    - composer/composer 1.10.5 requires composer/semver ^1.0 -> satisfiable by composer/semver[1.x-dev].
    - composer/composer 1.10.5 requires composer/semver ^1.0 -> satisfiable by composer/semver[1.x-dev].
    - Conclusion: don't install composer/semver 1.x-dev
    - Installation request for composer/composer (locked at 1.10.5, required as ^1.9.1) -> satisfiable by composer/composer[1.10.5].


Installation failed, reverting ./composer.json to its original content.

I tried switching the version of composer/composer and get a very similar patch as #9. As far as the tests are concerned, I think the fixes are in #3126566: Allow Drupal to work with Composer 2 anyway. It seems to me that they both might need to go in at the same time.

Just to test, I combined the patch from #3126566-72: Allow Drupal to work with Composer 2 and #9 so that we can test.

Now that the tests are passing in #3126566-83: Allow Drupal to work with Composer 2, I am merging that patch with the one here in #9.

@greg.1.anderson, certainly. I just wanted to verify that tests pass. +1 to postpone until composer/composer 2 is stable. Right now, the lock is pointing to `dev-master`.

Quick re-roll of the patch, mostly everything already commited from previous one

So here just new removal of TODO

It's not clear which plugin API should be used

-    "plugin-api-version": "1.1.0"
+    "plugin-api-version": "2.0.0"

Added related issue which could be fixed same time - #3128641: Enable scaffold upgrade test for Composer 2

For PHP 8.0 support it needs to be at least https://github.com/composer/semver/releases/tag/3.0.0

+++ b/composer.json
@@ -18,7 +18,7 @@
+        "composer/composer": "^1.9.1|^2.0",

A double pipe (||) will be treated as a logical OR

I haven't found the docs for the usage of single pipe (|) on getcomposer.org. Not sure if a single pipe (|) is treated as OR per the official docs here for Version Range.

Comments regarding single pipe usage https://github.com/composer/composer/issues/6755

We can not ever remove this or it'd break existing versions, we can only tell you || is preferred now :)

-- @Seldaek commented on 7 Apr, @see https://github.com/composer/composer/issues/6755#issuecomment-610363856

Thanks, @voleger. then that means double pipe (||) is preferred.

@greg.1.anderson while composer stable release date is not clear we could upgrade semver to ^1|| ^3 maybe spind up separate issue?

I read through https://www.drupal.org/about/core/policies/core-change-policies/drupal-8...

There's nothing about composer dependencies, but looking at #3110972: Update easyrdf library to 1.0.0 I guess we should split upgrade of semver out of composer because later is our dev-dependency

https://www.drupal.org/about/core/policies/core-change-policies/allowed-... states that it doable - jump to ^3 but it needs to check usage in contrib and decide about possible disruption (which I guess small)

Meantime https://github.com/composer/composer/releases/tag/2.0.0-RC2

The current plan is to release a 2.0 final before end of October.

From the release note of https://github.com/composer/composer/releases/tag/2.0.0-RC2

markdown -> from reading the issues there, semver isn't going to be an issue for d9 compatibilty - thats on a whole new branch, apparently.
composer project on drupal.org I believe is outdated and shoudl not be used. (5 sites report using that mondule)
violinist_projects is used on one site, violinist.io, which is a composer based tool for managing a site - They'll easily be able to incorporate this change.

beetbox is sort of an abandonware vm project aimed at being a lando like env. It never really took off, and has no releases
livezilla+azure_appinsights is in the require-dev section, thus sort of irrelevant to us conflicting with it.
There is also one other theme, that also has composer/composer as a require-dev: https://www.drupal.org/project/base_bootstrap_4

But base_bootstrap4/livezilla/azure_appinsights are all made by the same dev, so, I think thats not going to be hard to fix if need be.

*HOWEVER*

I did find one other project that has a dependency on composer/semver, and its not pretty:

https://packagist.org/packages/composer/installers

composer/semver: 1.0.* || 2.0.*@dev

So if we make core just ^3 it wont work.

Granted, I think a case can be made for an upstream fix to composer installers. *but* composer/installers upgrades are the path to accidentally upgrading peoples sites if they have `prefer-stable:dev` in their minimum-stability.

Sooo....

composer/installers master is already compatible with composer/semver 1 and 3:

https://github.com/composer/installers/commit/c462a693445b8f6913e95c2ea7...

So I think it just needs a new release? Also this is only in require-dev so does it even matter?

edit: as it also requires composer/composer 1.6.* || 2.0.*@dev I suspect they might wait for Composer 2.0 final before upgrading that dependency too, and then doing a release.

@longwave so the problem is that composer/installers is pretty much the only upstream package that we have as a root requirement in the templates, which makes it a root requirement of most people's sites.

Originally the templates had 'minimum-stability: dev' in them, and 'prefer-stable:true'

Well we found out that when we upgraded composer/installers people's sites were resolving to upgrade drupal/core from 8.8.7->8.9.x-dev because that way it was able to get a more modern version of composer/installers. My theory is that this is due to it being a root requirement, and drupal/core is a transitive dependency of drupal/core-recommended

https://www.drupal.org/project/drupal/issues/3135247

So, we have a plan to prevent that from happening to future sites, but we dont have a good way to 'get the word out to everybody else' that they should change their minimum-stability setting.

Thus, if/when composer/installers puts out a new release, if we upgrade that, we run the risk of shuttling people to a version of drupal/core that they didnt actually want.

So one thing Im noticing here is that our *internal* usage of both composer/semver and composer/composer isnt materially changing from composer 1 -> composer 2, and as such, isnt really a blocker for "composer 2 support"

The sole exception that I see here is in the Scaffold tests we have several invocations like the following:

$this->mustExec("composer

If Im not mistaken, that should cause the tests to rely on whatever version of composer the user has installed, and not rely on whatever version of composer core has a dependency on.

Given that, I think we should ensure that those exec's run out of core's composer directory $this->mustExec("/vendor/bin/composer

Then, we can wait until composer has a stable release, composer installers has a stable release, before introducing composer2 support into core, because afaict, we dont really need it at all right now, and its not blocking anything, except some testing weirdness..

But as with all things composer, I can only remember 85% of the picture at any given time, so there might be some other reason.

Pointed out in slack, however, was that I failed to notice that the semver requirement in composer/installers was a require-dev, so its a total non-issue.

Do we still need to wait for Composer 2 to be stable? It does not quite match with our timeline as Composer 2 is planned to be stable in 2 weeks, and our beta is in 3 weeks. IMHO getting people to test this with the alphas would be better even with a Composer 2 RC2 that is so far available.

Also, there's no problem if Composer stable is on time, ahead of the Drupal stable release, and we update our version constraint before our stable.

This seems reasonable to me - I will also talk to @xjm about it since it is a bit borderline. We'd need to be prepared to undo the change if composer gets delayed.

Also unpostponing because we are not necessarily waiting for composer here any more.

Oh, I wasn't aware of this issue and had filed #3159967: Allow Drupal 9 (including dev builds) to use Composer 2, part 2.

Now it needs upgrade to 2.0.1 https://github.com/composer/composer/releases/tag/2.0.1

Here's

9.1.x patch

So the disruption would only be for composer/semver, used by a handful of contributed projects. Given those projects are built around composer, they're going to have to update for composer 2 themselves anyway or not be very useful, so I think it's worth doing this and I don't think we have a choice for PHP 8 support.

Untagging for release manager review.

https://github.com/composer/composer/releases/tag/2.0.2

Needs reroll for it

Working on this.

Rerolled the patch for 9.2.x. Please Review.

Fixed patches to use 2.0.2 as ATM this is latest bugfix release

going back to #33 again.

$ find . -name composer.json -exec grep -H composer/composer {} \;
./violinist_projects/composer.json:    "composer/composer": "^1.7.0",
./composer/composer.json:    "composer/composer": "~1.9.3"
./beetbox/composer.json:    "composer/composer": "dev-master",
./livezilla/composer.json:        "composer/composer": "^1.3",
./azure_appinsights/composer.json:        "composer/composer": "^1.3",

So with these projects they're all going to get disrupted, the question is when. In the meantime it would be worth opening issues against those to warn them they'll need to update to composer 2.

If we only commit this to 9.2.x, then they have ~ 8 months to update, but we won't be able to properly support PHP 8 until 9.2.0. If we commit this to 9.1.x, they only have a few weeks but there's a small chance we'll be able to finish PHP 8 support.

Checked the stability and usage of each of those projects:

violinist_projects - dev only, no usage recorded.

composer - stable - 5 users.

beetbox - dev only, no usage recorded.

livezilla - alpha - 22 usage

azure_appinsights - alpha - 6 usage

Retagging for release manager review for 9.1.x backport discussion, but I think this can go into 9.2.x as soon as it's ready.

beetbox, livezilla and azure_appinsights have composer/composer in the require-dev section, so I think this is a non-issue - users will be unaffected, only developers of the modules themselves will run into the clash and they can solve this in their own time.

In violinist_projects I can't see any usage of Composer in the code itself. Raised #3178947: Drupal core is upgrading to Composer 2.

composer is a Drush extension which allows use of Composer commands from inside Drush. I am not really sure I see the point of this module when you can just use Composer from the CLI already, but raised #3178950: Upgrade to Composer 2 anyway.

test requeued and follow-ups are filed

There's also https://github.com/cweagans/composer-patches which could affect many sites that use that tool. It probably needs to be reviewed/tested for Composer 2 compatibility as this sounds like the author isn't confident:
https://github.com/cweagans/composer-patches/commit/7ee3bb5cd61ac2f9b26d...

Adding back the followup tag for that one. Also adding credits from the dupe I filed (when you close duplicates, please ensure relevant credits get transferred to the other issue).

I'm using cweagans/composer-patches ^1.7 last 2 weeks with composer 1 and 2 and no issues faced yet

https://github.com/cweagans/composer-patches 1.7 has been out for a while, and that, again, mixes the concept of end user sites, vs Internal drupal api using composer 2.

Right now, users of drupal are fully compatible with composer 2.

This fix makes it so that internally, drupal core uses composer 2 in its tests, and *also* makes it compatible with php8.

Sounds like we want 2.0.1: https://github.com/composer/composer/issues/9331

I think we need to do a little work on the release note and CR to clarify that:

1. You can install and run Drupal sites using Composer 1 on your server (...this is true, right?...) and any limitations around that.
2. Some advice around troubleshooting Composer 2 issues so we don't get a flood of core bug reports for things outside our control. (I.e., if you get an error message like _____ that means that that other dependency on your site isn't compatible with Composer 2, so you should file an issue with that project, or try ______.)

Did some work on the CR: https://www.drupal.org/node/3178837/revisions/view/12106955/12108216 and also updated the release note as per #77, although if anyone is aware of specific edgecases that might be problematic, please add them to the CR. Restoring RTBC.

Note that I looked over the affected contrib from #65 again and most of them aren't compatible with Drupal 9 to begin with, so I think it's worth committing this to 9.1.x before beta1.

As 8.9.x is a LTS and, according to https://www.drupal.org/docs/understanding-drupal/drupal-9-release-date-a..., is supported until November 2021, with our European Commision project we plan to stick to 8.9.x months to come. But in the same time we want to migrate to Composer 2. Right now, composer/semver constraint prevent us to require composer/composer:^2. I think we should port this to 8.9.x.

Tagging.

@claudiu.cristea this patch is required for projects to have composer 2 as a dependency, but it's not required to use composer 2 to install and update Drupal core - which already works (with the likely exception of dependencies on composer 1.9 plugins). Are you trying to do the former or the latter?

We're discussing whether and how much to backport to 8.9.x, but it's not straightforward to provide test coverage until #2874198: Create and run dependency regression tests for core and related issues are dealt with.

@catch, what we need is that we change in 8.9.x (core/composer.json) from composer/semver:^1.0 to composer/semver:^1.0 || ^2.0 || ^3.0 so we can require Composer 2.

so we can require Composer 2.

Its not clear why you would `require` composer 2 in your projects? do you have custom modules that use composer's classes directly?

@Mixologic, we have some tools to build the distribution (composer --no-dev) and we use ./vendor/bin/composer to make sure it works on all environments

Just to pipe in on dependency conflicts we've been having outside of Drupal, Composer Plugin has been a source of frustration. My worry is that if Drupal requires composer 2, there will be a conflict where Drupal 9.1 refuses to install on composer 1, but cannot update the underlying site composer because it depends on composer plugin or some other composer package tied to composer 1.

If its entirely a dependency / version issue, then it'd be best to let Drupal work with both versions at least until 9.3.0 comes out, which should be when 8.x is EOL and Composer 2 has been out for a year. This allows some time for the more obscure downstream dependencies to get updated to composer 2.

Regarding 8.9: The committer team had a discussion on Monday regarding whether it is feasible to offer PHP 8 support for Drupal 8.9 at all, given the multiple BC breaks, major version upgrades, etc. we are having to negotiate due to upstream necessities.

We'll be announcing this more later, but we've agreed that it's simply not feasible for a long-term support release like 8.9 to offer PHP 8 support. Sites which wish to use PHP 8 should update to 9.1. See: #3156651: Prevent Drupal 8.9 and 9.0 from being installed on PHP 8

We did discuss whether we could loosen constraints in 8.9 to allow sites themselves to choose to update to the PHP 8-compatible dependencies and deal with the resulting disruptions, but this is also dangerous as we don't have min-max testing, would need to define upward bounds, etc. and all sorts of further maintenance issues that seem out of place for a branch that is going into a reduced support mode soon and will be end-of-life in a year. (All off-topic for this issue, though; please discuss further in #3156651: Prevent Drupal 8.9 and 9.0 from being installed on PHP 8.)

That said, Drupal 8.8, 8.9, 9.0, and 9.1 can all be used with either Composer 1 or Composer 2 on the command-line, both with and without this patch. I've tested this extensively at this point. Most problems people encounter are related to out-of-date third-party dependencies. It's perfectly feasible to use Composer 1.10 to manage a project that has composer/composer 2 among its dev dependencies, or vice versa (and remember that it's only a dev dependency; your project is free to define a different set of dev dependencies than core-dev.)

@greg.1.anderson, maybe you could add your suggestions in #88 to the change record?

Untagging for release manager review now that we've both reviewed it and agree it should go into 9.1.x.

Although the prospects for full PHP 8 support in 8.9.x are unlikely, there's probably more argument to widen the constraints for composer than for other dependencies, so keeping open for discussion and change record improvements.

This change has made Drupal core incompatible with Drush as it depends on composer/semver ^1.4,
https://github.com/drush-ops/drush/blob/10.x/composer.json#L36

@Chi Yes, as above, an issue was filed against drush. (If it wasn't this is an oversight and it should be filed now.) Edit: #3178950: Upgrade to Composer 2 was the issue. If drush core also declares a dependency (and not just this extension), then we need a PR made against Drush. (This is what happens when code is hosted off d.o; we don't find it when we look for things to fix.)

Did the change record updates I asked for in #90 get made?

If we don't strictly depend on composer/semver 3.x and can equally work with 1.x then maybe we should just relax the constraint to allow both for a little while? I know we don't have min-max testing but we could live with it for a minor or two while users get around to upgrading? Isn't this technically a BC break otherwise?

composer/composer shouldn't be an issue as that is only in require-dev.

Opened https://github.com/drush-ops/drush/issues/4580 for Drush itself, couldn't find an existing issue.

Isn't this technically a BC break otherwise?

According to semver.org it is not because it does not change public API.
https://semver.org/#what-should-i-do-if-i-update-my-own-dependencies-wit...

However, Drupal typically follows more strict BC policies. For instance, we did not update Twig to version 3 even in major release (Drupal 9).

Adding that related issue.

Filed PR to drush https://github.com/drush-ops/drush/pull/4582

@longwave re: relaxing constraints, the issue is that we specifically pin specific versions of our dependencies to avoid unpredictability and breakage when (e.g.) Twig makes a bad update and breaks people's Drupal sites. In my original issue (that was closed as a dupe) I tried specifying ^1 || ^2 || 3 for the same reasons you're suggesting, but at a minimum the tarballs will still need to get built with only one version, and that version needs to be compatible with PHP 8. Even for non-tarballs, we're introducing some unpredictability and risk if we don't use the strict constraint. We don't have min/max testing, upper bounds, etc.

It's worth discussing further but I've been given to understand that there's lots of problems each time we try to loosen a constraint.

https://github.com/drush-ops/drush/pull/4581 is the actual PR now (the other was closed as a dupe). It's worth considering whether we want to wait for that to be resolved (up to mid-window, at least) before we backport this to 9.1.x. Edit: For some reason I thought catch had committed this to 9.2.x only. I don't think it should be backported to 9.0.x. We'll discuss.

hmm I don't think this should go into 9.0.x either, but I thought we should discuss loosening the constraints on 8.9.x a bit more, so moving there. Seems strange to skip over 9.0.x but that's what LTSs are like..

I dont think it would hurt anything to loosen the constraints on 8.9.x, but if we're not doing so for php8 reasons, Im not sure that we should support the workflow described in #86. Thats a very atypical way to work with composer, and there are multiple other methods of accomplishing the same thing that do not rely on having composer as an actual requirement of your codebase. Granted, I haven't seen why or how of the specifics of #86, but it strikes me as a surprising, off label way to use composer.

Discussed further with @catch -- this issue isn't under consideration for 9.0.x since its final patch release is next week. However, for 8.9.x, it seems like there's further discussion needed about whether we could loosen constraints and what the implications of that would be.

Meanwhile, we also need:

1. More troubleshooting instructions in the CR, including recommendations from #88.
2. Monitor the Drush PR so that we can give users any needed Drush troubleshooting info for 9.1.x.

@catch and I agreed #3179647: Loosen composer/semver constraints to ^1 || ^3 should be discussed as a followup, so let's transfer discussion about loosening constraints over there. A looser-constraint version might be backportable to 8.9.x; however, the version in this issue definitely is not.

So setting it back to 9.1.x for the two points in #109 and changing to NW.

Thanks, and sorry for the noise.

Thanks @greg.1.anderson; I added this to the release note. We can update the text if/when a new release is created containing the Drush fix.

Did a bit more work on the CR, with an @todo for the info from #88. https://www.drupal.org/node/3178837/revisions/view/12109180/12111746

Im sorta not sure what to add to the CR for #88

Requiring composer/composer into a project and intermingling composer's dependencies with drupal's dependencies is generally considered an antipattern, because you mingle the code responsible for dependency management to be with your own sites dependencies and you can potentially get into a situation where you cannot upgrade because of a conflict between the upgrader itself, and your site. This is not to say that it cant be done, as someday composer/composer may actually ship with drupal to handle updates, but its not tested on it now, so.. not necessarily a good thing to do.

There are two issues that result from implementing this antipattern:

#1. Site owners, on drupal 9.1+ will be forced to upgrade to composer 2, and they may have incompatible plugins, meaning they cant upgrade to 9.1
#2. Site owners, that *do* wish to upgrade to composer 2, cannot do so until they are upgraded to drupal 9.1 without backporting these changes to 8.9.x (re: a href="#comment-13883847">#81)

Both of these issues have the same solution:

first, remove your dependency on composer from your site,
and either install composer 1 in a the typical, recommended way (as an external phar file, somewhere else in your path),
Or, if for some reason it *must* live underneath your project root, you can do another atypical method as follows:

Create a new directory in your project. Example:

$ mkdir projectroot/altvendor
$ cd altvendor
$ COMPOSER_BIN_DIR=../vendor/bin composer require composer/composer

You will see a prompt, answer "n"
No composer.json in current directory, do you want to use the one at /Path/to/projectroot? [Y,n]? n

This will leave you with a version of composer that can be run with ./vendor/bin/composer yet not be intermingled with the dependencies of your site.

The other option is to open up both dependencies to allow for broader acceptance (composer/composer: ^1.9.1 | ^2.0 and https://www.drupal.org/project/drupal/issues/3179647) but that comes with a lot of risk wherein we dont have any tooling to support that kind of openness, and thus may end up breaking somebody, sometime in the future.

Where does this stand currently? I've run into this issue and can't tell what's going on here right now. :(

@seanr this has been committed to 9.1.x (and above), it won't be backported to 8.9.x because that's in security-only mode now.

If you're running into an issue, you probably need to do something like #116.