Problem/Motivation
Support for Composer 2 was added in #3126566: Allow Drupal to work with Composer 2. However, the test suite still uses composer/composer ^1 internally for testing (via require-dev), as a stable release for the 2.x version of composer/composer and composer/semver (used by drupal/core, and required by Composer 2) are not yet available.
Proposed resolution
Once there is a stable release of composer/composer and composer/semver in the ^2 series, these dependencies should be updated for Drupal.
Also, the following test was conditionally disabled in the test suite when running the tests with the Composer 2 cli. This test installs components from packagist, so there must be a version of drupal/core on packagist that is compatible with Composer 2 before this test can be re-enabled. n.b. tagging a stable release with #3126566: Allow Drupal to work with Composer 2 is sufficient to allow this test to work with the Composer 2 cli; updating the internal composer/semver to 2.0 is not necessary. This step could therefore be done independently of this issue, if desired.
public function testTemplateCreateProject($project, $package_dir, $docroot_dir) {
+ $composerVersionLine = exec('composer --version');
+ if (strstr($composerVersionLine, 'Composer version 2') !== FALSE) {
+ $this->markTestSkipped('We cannot run the template create project test with Composer 2 until we have a stable version of composer/semver 2.x. The create project test installs drupal/core-recommended and the Drupal Composer plugins from Packagist, so these must also be compatible with Composer 2.x in order for this test to work.');
+ }
+
Remaining tasks
- Update composer/composer to ^2 in the root composer.json file
- Update composer/semver to ^2 in the drupal/core composer.json file
- Re-enable the test in core/tests/Drupal/BuildTests/Composer/Template/ComposerProjectTemplatesTest.php
User interface changes
None
API changes
None
Data model changes
None
Release notes snippet
In order to support PHP 8, Drupal core is now using composer/semver
version 3, and the tests are now using composer/composer
version 2 internally. For more information, see the change record on Drupal 9.1's internal composer requirements. Note: These are only internal dependencies. Site owners may use either Composer 1 or Composer 2 on the command line to manage their Drupal 8.8, 8.9, 9.0, and 9.1 sites. Drush users must update to the latest dev version for compatibility with this change in Drupal 9.1.
Comment | File | Size | Author |
---|---|---|---|
#64 | 3128631-64-9.1.x.patch | 15.7 KB | andypost |
#64 | 3128631-64-9.2.x.patch | 14.96 KB | andypost |
Comments
Comment #2
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedComment #3
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedcomposer/semver 2.0.0 stable is out!
Comment #4
catchBumping to critical and adding various tags - we need to decide whether we're going to try to squeeze this into the beta or not - gets harder after that.
Comment #5
longwaveDo we want to go straight to only
^2.0
or might we want to allow^1.9.1 | ^2.0
?Comment #6
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedIf this is backported to Drupal 8.9.x, then ^1 | ^2 should definitely be used there. For Drupal 9, we can choose to be flexible or not.
Also n.b. the current 1.x release of composer/semver is 1.5.1, and 1.5.0 is the version in the 8.9.x lock file.
Comment #7
longwave#3122112: Update dependencies for Drupal 8.9 updates the lock file versions of composer/composer and composer/semver.
Comment #8
longwaveIt doesn't look like Composer 2 will be stable in time for Drupal 9 release, as per https://github.com/composer/composer/issues/8726#issuecomment-618854271
@xjm suggested in Slack that we could still aim for
^1.whatever || ^2
Comment #9
longwaveThis bumps us to 2.0-dev release as there is no alpha let alone stable release yet, but let's see what happens.
Composer 2.0 does bring a new dependency,
react/promise
- does this have to go through the dependency evaluation process?Comment #10
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedI already did some testing with Composer 2 in #3126566: Allow Drupal to work with Composer 2; see patches there.
We should probably make composer/composer ^2 and composer/semver^2 separate issues, as the later can be updated now (if we want), whereas composer/composer ^2 won't be stable for a while, as observed.
Comment #12
hussainweb@greg.1.anderson, I am not sure if the latter can be updated right now in isolation. On 9.0.x branch:
It's clear we need to update `composer/composer` in order to use `composer/semver`. I tried it anyway, and as expected, I got an error.
Comment #13
hussainwebI tried switching the version of composer/composer and get a very similar patch as #9. As far as the tests are concerned, I think the fixes are in #3126566: Allow Drupal to work with Composer 2 anyway. It seems to me that they both might need to go in at the same time.
Comment #14
hussainwebJust to test, I combined the patch from #3126566-72: Allow Drupal to work with Composer 2 and #9 so that we can test.
Comment #15
hussainwebComment #17
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedThe testing I already did in #3126566: Allow Drupal to work with Composer 2 are in comment #18 of that issue.
I guess you are right that if we do not upgrade Composer to some ^2 version, that composer/semver will stay pinned at ^1.x, so we need to keep these updates together in this issue and not separate them.
Comment #18
hussainwebNow that the tests are passing in #3126566-83: Allow Drupal to work with Composer 2, I am merging that patch with the one here in #9.
Comment #19
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commented#18 is confusing. We should just postpone this issue until #3126566: Allow Drupal to work with Composer 2 is committed.
Comment #20
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commented#3126566: Allow Drupal to work with Composer 2 has actually been committed already; it was just Needs Work for the 8.9.x backport. Un-postponing here; #18 just needs a re-roll.
We probably don't want to merge until Composer ^2 is stable, so perhaps we'll end up going back to "postponed" here anyway. Leaving non-postponed, as a release manager should review and decide whether this should still be critical. Since composer/semver and composer/composer are tied together, and composer/composer is not necessarily going to be ready, I am guessing this will probably bump down to "major".
Comment #21
hussainweb@greg.1.anderson, certainly. I just wanted to verify that tests pass. +1 to postpone until composer/composer 2 is stable. Right now, the lock is pointing to `dev-master`.
Comment #22
andypostQuick re-roll of the patch, mostly everything already commited from previous one
So here just new removal of TODO
It's not clear which plugin API should be used
Added related issue which could be fixed same time - #3128641: Enable scaffold upgrade test for Composer 2
Comment #23
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedNote that Composer 2 now requires composer/semver:^3, so this patch is no longer bringing in Composer 2 or upgrading composer/semver.
I haven't tried yet; hopefully composer/semver:^3 will be usable here without a b/c break. Composer 1 cannot use composer/semver:^3. (Note: these comments only apply so using composer/composer and composer/semver as dev dependencies for our tests; both Composer 1 and Composer 2 from the commandline should continue to work regardless of which version of composer/semver we are using. I think / hope.)
Comment #24
andypostFor PHP 8.0 support it needs to be at least https://github.com/composer/semver/releases/tag/3.0.0
Comment #25
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedComment #26
jungleI haven't found the docs for the usage of single pipe (|) on getcomposer.org. Not sure if a single pipe (|) is treated as OR per the official docs here for Version Range.
Comment #27
volegerComments regarding single pipe usage https://github.com/composer/composer/issues/6755
Comment #28
jungleThanks, @voleger. then that means double pipe (||) is preferred.
Comment #29
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedPostponing this issue until Composer 2.0.0 stable is available. I expect this should be soon; I asked at composer/composer#9281.
Comment #30
andypost@greg.1.anderson while composer stable release date is not clear we could upgrade semver to
^1|| ^3
maybe spind up separate issue?Comment #31
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedYeah I was thinking about this issue this morning, and I don't think we need a separate issue. I think we need to de-scope composer ^2 completely, and make this issue only about composer/semver ^1 || ^3.
Consider our use cases for composer/composer:
* composer/composer is a require-dev dependency of drupal/drupal. It does not appear in Drupal sites at all.
* composer/composer is only a dependency of drupal/drupal so that tests can use Composer APIs.
* Upgrading to composer/composer ^2 would allow tests to use new Composer APIs.
In contrast, our use cases for composer/semver:
* composer/semver is used occasionally in drupal/core.
* None of the API breaks in ^3 are used by Drupal, so we can use either ^1 or ^3
* Upgrading composer/semver to allow ^3 would allow us to use composer/composer ^2 in our tests.
* If we upgrade composer/semver to allow ^3, then we allow composer/semver to be used in Drupal sites that might use dependencies that need it (present or future)
If we evaluate all of these drivers, only the last one is important. It is very frustrating to users when they cannot upgrade a dependency due to a Drupal limitation. Since Drupal updates its dependencies only rarely, these limitations can be lasting.
Consider also the testing aspects involved in these upgrades. If we allow composer/semver ^3, and put a 3.x version in our lock file, then all of our tests will run with composer/semver ^3. This is kind of a dangerous situation, because a future patch might use a composer/semver 3.x API, and we'd never know (unless very sharp eyes caught it -- unreliable), because all of the tests would pass. Writing a test to see if composer/semver 1.x still works wouldn't be practical; we'd have to run the whole test suite, or we would not protect against new code that might start using composer/semver ^3. If we don't test ^1, then we shouldn't say that it's allowed, and should just bump to composer/semver ^3. That could be a breaking change for some sites.
So, what I think we should do is change core/composer.json to allow composer/semver ^1 || ^3, but keep composer/semver 1.x in our lock file. To test composer/semver ^3, we could run a composer update and then re-run the Composer suite, which would be fairly quick (comparitively). In terms of what is POSSIBLE to go wrong, this scenario is the same as in the paragraph above; however, in terms of what is LIKELY to go wrong, I think this configuration is much safer. composer/semver ^3 does not break much from ^1, so it seems that someone introducing a deprecated use is less likely than someone picking up a new available API.
Is that safe enough? Hard to say. Drupal does not use any || constraints yet, so we do not have precedence. If this was only about composer/composer ^2 in our dev dependencies, I'd say that we should punt entirely, and wait until Drupal 10 for the composer/semver ^3 release. I have reservations, though, that sticking with composer/semver ^1 for that long would be more damaging than the risk of adding the ^3.
Maybe the safest thing to do would be to just bump up to composer/semver ^3 and omit the ^1 completely. While this might be breaking for some, it should be easy for sites (and their dependencies) to also allow composer/semver ^3. This level of change is allowed in a minor release, right?
Comment #32
andypostI read through https://www.drupal.org/about/core/policies/core-change-policies/drupal-8...
There's nothing about composer dependencies, but looking at #3110972: Update easyrdf library to 1.0.0 I guess we should split upgrade of semver out of composer because later is our dev-dependency
https://www.drupal.org/about/core/policies/core-change-policies/allowed-... states that it doable - jump to ^3 but it needs to check usage in contrib and decide about possible disruption (which I guess small)
Comment #33
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedI did a survey on contrib, and this is what I found.
One project requires composer/semver directly:
Several projects require composer/composer:
For the one project that uses composer/semver, we could post an issue in their queue requesting they upgrade.
For the second set, if we upgrade to composer/semver ^3 then they MUST upgrade to composer ^2, and if we do not then they CANNOT upgrade to composer ^2.
There may be other contrib projects that end up requiring composer/semver indirectly, but I did not analyze that.
Comment #34
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedWe cannot split off a separate issue to upgrade to composer/semver ^3, because once we do that we will only be able to use Composer 2 for our dev dependencies; upgrading composer/semver and leaving our dev dependencies on composer/composer ^1 would not work.
We COULD split off a separate issue to upgrade to composer/semver ^1 || ^3, and keep Semver 1 in the lock file. That still has testing impacts per #31.
We could move forward with this issue and upgrade to composer/semver ^3 in core/composer.json, and use composer/composer 2.0.0-RC1 if we think that's stable enough to use as a dev dependency.
Overall, though, I think we're probably best off if we just wait for Composer 2's stable release at the end of October, and do this all at once. Not sure how well that works with the Drupal 9.1 release date; 9.1.0-beta1 is slated for 2 November. There probably isn't time to get this patch in if we wait. If it is too much of a hardship to the modules mentioned in #33 (and any additional modules not yet identified) to be blocked out of Composer 2 use until Drupal 9.2, then we should consider trying composer/semver ^1 || ^3 and accept the testing risk.
I tend to think that given the limited contrib exposure, we could wait until Drupal 9.2 for this.
Comment #35
andypostMeantime https://github.com/composer/composer/releases/tag/2.0.0-RC2
Comment #36
jungleFrom the release note of https://github.com/composer/composer/releases/tag/2.0.0-RC2
Comment #37
Mixologicmarkdown -> from reading the issues there, semver isn't going to be an issue for d9 compatibilty - thats on a whole new branch, apparently.
composer project on drupal.org I believe is outdated and shoudl not be used. (5 sites report using that mondule)
violinist_projects is used on one site, violinist.io, which is a composer based tool for managing a site - They'll easily be able to incorporate this change.
beetbox is sort of an abandonware vm project aimed at being a lando like env. It never really took off, and has no releases
livezilla+azure_appinsights is in the require-dev section, thus sort of irrelevant to us conflicting with it.
There is also one other theme, that also has composer/composer as a require-dev: https://www.drupal.org/project/base_bootstrap_4
But base_bootstrap4/livezilla/azure_appinsights are all made by the same dev, so, I think thats not going to be hard to fix if need be.
*HOWEVER*
I did find one other project that has a dependency on
composer/semver
, and its not pretty:https://packagist.org/packages/composer/installers
composer/semver: 1.0.* || 2.0.*@dev
So if we make core just ^3 it wont work.
Granted, I think a case can be made for an upstream fix to composer installers. *but* composer/installers upgrades are the path to accidentally upgrading peoples sites if they have `prefer-stable:dev` in their minimum-stability.
Sooo....
Comment #38
longwavecomposer/installers
master is already compatible withcomposer/semver
1 and 3:https://github.com/composer/installers/commit/c462a693445b8f6913e95c2ea7...
So I think it just needs a new release? Also this is only in require-dev so does it even matter?
edit: as it also requires
composer/composer
1.6.* || 2.0.*@dev I suspect they might wait for Composer 2.0 final before upgrading that dependency too, and then doing a release.Comment #39
Mixologic@longwave so the problem is that composer/installers is pretty much the only upstream package that we have as a root requirement in the templates, which makes it a root requirement of most people's sites.
Originally the templates had 'minimum-stability: dev' in them, and 'prefer-stable:true'
Well we found out that when we upgraded composer/installers people's sites were resolving to upgrade drupal/core from 8.8.7->8.9.x-dev because that way it was able to get a more modern version of composer/installers. My theory is that this is due to it being a root requirement, and drupal/core is a transitive dependency of drupal/core-recommended
https://www.drupal.org/project/drupal/issues/3135247
So, we have a plan to prevent that from happening to future sites, but we dont have a good way to 'get the word out to everybody else' that they should change their minimum-stability setting.
Thus, if/when composer/installers puts out a new release, if we upgrade that, we run the risk of shuttling people to a version of drupal/core that they didnt actually want.
Comment #40
MixologicSo one thing Im noticing here is that our *internal* usage of both composer/semver and composer/composer isnt materially changing from composer 1 -> composer 2, and as such, isnt really a blocker for "composer 2 support"
The sole exception that I see here is in the Scaffold tests we have several invocations like the following:
$this->mustExec("composer
If Im not mistaken, that should cause the tests to rely on whatever version of composer the user has installed, and not rely on whatever version of composer core has a dependency on.
Given that, I think we should ensure that those exec's run out of core's composer directory
$this->mustExec("/vendor/bin/composer
Then, we can wait until composer has a stable release, composer installers has a stable release, before introducing composer2 support into core, because afaict, we dont really need it at all right now, and its not blocking anything, except some testing weirdness..
But as with all things composer, I can only remember 85% of the picture at any given time, so there might be some other reason.
Comment #41
MixologicPointed out in slack, however, was that I failed to notice that the semver requirement in composer/installers was a require-dev, so its a total non-issue.
Comment #42
Gábor HojtsyDo we still need to wait for Composer 2 to be stable? It does not quite match with our timeline as Composer 2 is planned to be stable in 2 weeks, and our beta is in 3 weeks. IMHO getting people to test this with the alphas would be better even with a Composer 2 RC2 that is so far available.
Comment #43
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commented#3135247: Composer's "prefer-stable" setting cannot be relied on to produce a stable release would prevent folks who create a site with drupal/recommended-project or drupal/legacy-project ^9.1 (which will have minimum-stability: stable) from using drupal/dev-dependencies if we put a non-stable version constraint in our drupal/drupal require-dev section.
If it is an acceptable workaround to make folks edit their minimum stability down to rc just until 9.1.1, then we could use composer/composer ^2 with no further adjustments. We could also make a (temporary?) follow-on adjustment to #3135247 to allow RC dependencies in stable Drupal releases.
Comment #44
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedAlso, there's no problem if Composer stable is on time, ahead of the Drupal stable release, and we update our version constraint before our stable.
Comment #45
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedAlso, just for clarity for those following along with this issue, I will reiterate that we are talking about Composer 2 usage in the Drupal tests. If we don't get this issue in, it do not prevent folks from using Composer 2 with Drupal 9; this already works.
Comment #47
PasqualleComment #48
catchThis seems reasonable to me - I will also talk to @xjm about it since it is a bit borderline. We'd need to be prepared to undo the change if composer gets delayed.
Also unpostponing because we are not necessarily waiting for composer here any more.
Comment #49
xjmOh, I wasn't aware of this issue and had filed #3159967: Allow Drupal 9 (including dev builds) to use Composer 2, part 2.
Comment #50
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedComposer 2 stable is out, so we can go ahead and bump to composer/composer ^2 and composer/semver ^3 here.
Comment #51
longwaveComment #52
andypostNow it needs upgrade to 2.0.1 https://github.com/composer/composer/releases/tag/2.0.1
Comment #53
andypostHere's
Comment #54
andypost9.1.x patch
Comment #55
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedNice to see the skipped test being included again, and similarly the removal of the re-testing with Composer 2 should speed things up a bit.
Comment #56
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedComment #57
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedI also updated the release note snippet and added a change record.
Comment #58
catchSo the disruption would only be for composer/semver, used by a handful of contributed projects. Given those projects are built around composer, they're going to have to update for composer 2 themselves anyway or not be very useful, so I think it's worth doing this and I don't think we have a choice for PHP 8 support.
Untagging for release manager review.
Comment #59
andyposthttps://github.com/composer/composer/releases/tag/2.0.2
Needs reroll for it
Comment #60
catchComment #61
ayushmishra206 CreditAttribution: ayushmishra206 at OpenSense Labs commentedWorking on this.
Comment #62
ayushmishra206 CreditAttribution: ayushmishra206 at OpenSense Labs commentedRerolled the patch for 9.2.x. Please Review.
Comment #63
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedPerhaps #59 should not have said reroll, because what is being requested here is that the patch be updated to require composer/composer ^2.0.2. I don't know if that is actually necessary under the circumstances, since composer/composer is only a dev dependency of Drupal, and I don't think that any of the bug fixes in the Composer 2.0.2 release will affect the Drupal tests.
So, I think we could go back to RTBC here by re-copying #53 and #54 to a later comment. We cannot go with #62, though, as it does not appear to be correct to me. I'm not able to spend more time on this right now so I can't be more specific, & I think it would be good to have another pair of eyes before acting as I have suggested.
Comment #64
andypostFixed patches to use 2.0.2 as ATM this is latest bugfix release
Comment #65
catchgoing back to #33 again.
So with these projects they're all going to get disrupted, the question is when. In the meantime it would be worth opening issues against those to warn them they'll need to update to composer 2.
If we only commit this to 9.2.x, then they have ~ 8 months to update, but we won't be able to properly support PHP 8 until 9.2.0. If we commit this to 9.1.x, they only have a few weeks but there's a small chance we'll be able to finish PHP 8 support.
Checked the stability and usage of each of those projects:
violinist_projects - dev only, no usage recorded.
composer - stable - 5 users.
beetbox - dev only, no usage recorded.
livezilla - alpha - 22 usage
azure_appinsights - alpha - 6 usage
Retagging for release manager review for 9.1.x backport discussion, but I think this can go into 9.2.x as soon as it's ready.
Comment #66
catchComment #67
longwavebeetbox, livezilla and azure_appinsights have
composer/composer
in therequire-dev
section, so I think this is a non-issue - users will be unaffected, only developers of the modules themselves will run into the clash and they can solve this in their own time.In violinist_projects I can't see any usage of Composer in the code itself. Raised #3178947: Drupal core is upgrading to Composer 2.
composer is a Drush extension which allows use of Composer commands from inside Drush. I am not really sure I see the point of this module when you can just use Composer from the CLI already, but raised #3178950: Upgrade to Composer 2 anyway.
Comment #69
andyposttest requeued and follow-ups are filed
Comment #70
xjmThere's also https://github.com/cweagans/composer-patches which could affect many sites that use that tool. It probably needs to be reviewed/tested for Composer 2 compatibility as this sounds like the author isn't confident:
https://github.com/cweagans/composer-patches/commit/7ee3bb5cd61ac2f9b26d...
Comment #71
xjmAdding back the followup tag for that one. Also adding credits from the dupe I filed (when you close duplicates, please ensure relevant credits get transferred to the other issue).
Comment #72
andypostI'm using
cweagans/composer-patches ^1.7
last 2 weeks with composer 1 and 2 and no issues faced yetComment #73
Mixologichttps://github.com/cweagans/composer-patches 1.7 has been out for a while, and that, again, mixes the concept of end user sites, vs Internal drupal api using composer 2.
Right now, users of drupal are fully compatible with composer 2.
This fix makes it so that internally, drupal core uses composer 2 in its tests, and *also* makes it compatible with php8.
Comment #74
xjmSounds like we want 2.0.1: https://github.com/composer/composer/issues/9331
Comment #75
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commented#64 is already pinned to 2.0.2 or later.
Comment #76
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedReviewed the patches in #64, and they look good.
Also removing the "Needs followup" from #71 per #72 and #73. I am also of the opinion that cweagans/composer-patches is working fine and needs no further notification in this area.
Comment #77
xjmI think we need to do a little work on the release note and CR to clarify that:
Comment #78
xjmDid some work on the CR: https://www.drupal.org/node/3178837/revisions/view/12106955/12108216 and also updated the release note as per #77, although if anyone is aware of specific edgecases that might be problematic, please add them to the CR. Restoring RTBC.
Note that I looked over the affected contrib from #65 again and most of them aren't compatible with Drupal 9 to begin with, so I think it's worth committing this to 9.1.x before beta1.
Comment #79
xjmComment #80
xjmComment #81
claudiu.cristeaAs
8.9.x
is a LTS and, according to https://www.drupal.org/docs/understanding-drupal/drupal-9-release-date-a..., is supported until November 2021, with our European Commision project we plan to stick to8.9.x
months to come. But in the same time we want to migrate to Composer 2. Right now,composer/semver
constraint prevent us to requirecomposer/composer:^2
. I think we should port this to 8.9.x.Comment #82
claudiu.cristeaTagging.
Comment #83
catch@claudiu.cristea this patch is required for projects to have composer 2 as a dependency, but it's not required to use composer 2 to install and update Drupal core - which already works (with the likely exception of dependencies on composer 1.9 plugins). Are you trying to do the former or the latter?
We're discussing whether and how much to backport to 8.9.x, but it's not straightforward to provide test coverage until #2874198: Create and run dependency regression tests for core and related issues are dealt with.
Comment #84
claudiu.cristea@catch, what we need is that we change in 8.9.x (
core/composer.json
) fromcomposer/semver:^1.0
tocomposer/semver:^1.0 || ^2.0 || ^3.0
so we can require Composer 2.Comment #85
MixologicIts not clear why you would `require` composer 2 in your projects? do you have custom modules that use composer's classes directly?
Comment #86
claudiu.cristea@Mixologic, we have some tools to build the distribution (composer --no-dev) and we use ./vendor/bin/composer to make sure it works on all environments
Comment #87
japerryJust to pipe in on dependency conflicts we've been having outside of Drupal, Composer Plugin has been a source of frustration. My worry is that if Drupal requires composer 2, there will be a conflict where Drupal 9.1 refuses to install on composer 1, but cannot update the underlying site composer because it depends on composer plugin or some other composer package tied to composer 1.
If its entirely a dependency / version issue, then it'd be best to let Drupal work with both versions at least until 9.3.0 comes out, which should be when 8.x is EOL and Composer 2 has been out for a year. This allows some time for the more obscure downstream dependencies to get updated to composer 2.
Comment #88
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commented@claudiu.crestea: Re #86, it would be better for you to provide your site-local copy of Composer in some other location than the `vendor` directory. It is not needed there, and as you have observed, could cause problems if you mixed it in with your Drupal site. You could `composer require composer/composer` in a subdirectory of your site (create a second nested vendor), and set COMPOSER_BIN_DIR to install the binary symlink at ./vendor/bin/composer. You wouldn't even need to alter how Composer is called if you did that.
@japerry #87: As long as composer/composer is not a dependency of your site or your module, you may continue to use either Composer 1 or Composer 2, either with this patch (presumably in Drupal 9.1) or without it (Drupal 9.0 and earlier, if not backported).
The more salient question is, which old versions of Drupal do we intend to support PHP 8.0 on? If we expect to allow PHP 8 on Drupal 8.9.x LTS, then that is a reason to backport. Again, except for a few edge cases, it would probably be fairly safe to do so, and would not require old sites to stop using Composer 1.
Comment #89
xjmRegarding 8.9: The committer team had a discussion on Monday regarding whether it is feasible to offer PHP 8 support for Drupal 8.9 at all, given the multiple BC breaks, major version upgrades, etc. we are having to negotiate due to upstream necessities.
We'll be announcing this more later, but we've agreed that it's simply not feasible for a long-term support release like 8.9 to offer PHP 8 support. Sites which wish to use PHP 8 should update to 9.1. See: #3156651: Prevent Drupal 8.9 and 9.0 from being installed on PHP 8
We did discuss whether we could loosen constraints in 8.9 to allow sites themselves to choose to update to the PHP 8-compatible dependencies and deal with the resulting disruptions, but this is also dangerous as we don't have min-max testing, would need to define upward bounds, etc. and all sorts of further maintenance issues that seem out of place for a branch that is going into a reduced support mode soon and will be end-of-life in a year. (All off-topic for this issue, though; please discuss further in #3156651: Prevent Drupal 8.9 and 9.0 from being installed on PHP 8.)
That said, Drupal 8.8, 8.9, 9.0, and 9.1 can all be used with either Composer 1 or Composer 2 on the command-line, both with and without this patch. I've tested this extensively at this point. Most problems people encounter are related to out-of-date third-party dependencies. It's perfectly feasible to use Composer 1.10 to manage a project that has
composer/composer
2 among its dev dependencies, or vice versa (and remember that it's only a dev dependency; your project is free to define a different set of dev dependencies thancore-dev
.)Comment #90
xjm@greg.1.anderson, maybe you could add your suggestions in #88 to the change record?
Comment #91
xjmComment #92
xjmComment #95
catchUntagging for release manager review now that we've both reviewed it and agree it should go into 9.1.x.
Although the prospects for full PHP 8 support in 8.9.x are unlikely, there's probably more argument to widen the constraints for composer than for other dependencies, so keeping open for discussion and change record improvements.
Comment #96
Chi CreditAttribution: Chi commentedThis change has made Drupal core incompatible with Drush as it depends on composer/semver ^1.4,
https://github.com/drush-ops/drush/blob/10.x/composer.json#L36
Comment #97
xjm@Chi Yes, as above, an issue was filed against drush. (If it wasn't this is an oversight and it should be filed now.) Edit: #3178950: Upgrade to Composer 2 was the issue. If drush core also declares a dependency (and not just this extension), then we need a PR made against Drush. (This is what happens when code is hosted off d.o; we don't find it when we look for things to fix.)
Did the change record updates I asked for in #90 get made?
Comment #98
longwaveIf we don't strictly depend on
composer/semver
3.x and can equally work with 1.x then maybe we should just relax the constraint to allow both for a little while? I know we don't have min-max testing but we could live with it for a minor or two while users get around to upgrading? Isn't this technically a BC break otherwise?composer/composer
shouldn't be an issue as that is only in require-dev.Comment #99
longwaveOpened https://github.com/drush-ops/drush/issues/4580 for Drush itself, couldn't find an existing issue.
Comment #100
Chi CreditAttribution: Chi commentedAccording to semver.org it is not because it does not change public API.
https://semver.org/#what-should-i-do-if-i-update-my-own-dependencies-wit...
However, Drupal typically follows more strict BC policies. For instance, we did not update Twig to version 3 even in major release (Drupal 9).
Comment #101
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedYeah I agree. I went looking and I know that Drush uses composer/semver and I sometimes use Drush and I forgot and missed it. Doh. Not saying we should move Drush back to d.o. tomorrow, but might be something to consider as the gitlab merge request features solidify.
I'll work on a PR to fix Drush to work with composer/semver ^1 | ^3
Comment #102
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedOh it would also be cool to require Drush in the project templates so that it can be included in the Drupal tests.
Comment #103
MixologicAdding that related issue.
Comment #104
andypostFiled PR to drush https://github.com/drush-ops/drush/pull/4582
Comment #105
xjm@longwave re: relaxing constraints, the issue is that we specifically pin specific versions of our dependencies to avoid unpredictability and breakage when (e.g.) Twig makes a bad update and breaks people's Drupal sites. In my original issue (that was closed as a dupe) I tried specifying
^1 || ^2 || 3
for the same reasons you're suggesting, but at a minimum the tarballs will still need to get built with only one version, and that version needs to be compatible with PHP 8. Even for non-tarballs, we're introducing some unpredictability and risk if we don't use the strict constraint. We don't have min/max testing, upper bounds, etc.It's worth discussing further but I've been given to understand that there's lots of problems each time we try to loosen a constraint.
Comment #106
xjmhttps://github.com/drush-ops/drush/pull/4581 is the actual PR now (the other was closed as a dupe).
It's worth considering whether we want to wait for that to be resolved (up to mid-window, at least) before we backport this to 9.1.x.Edit: For some reason I thought catch had committed this to 9.2.x only. I don't think it should be backported to 9.0.x. We'll discuss.Comment #107
catchhmm I don't think this should go into 9.0.x either, but I thought we should discuss loosening the constraints on 8.9.x a bit more, so moving there. Seems strange to skip over 9.0.x but that's what LTSs are like..
Comment #108
MixologicI dont think it would hurt anything to loosen the constraints on 8.9.x, but if we're not doing so for php8 reasons, Im not sure that we should support the workflow described in #86. Thats a very atypical way to work with composer, and there are multiple other methods of accomplishing the same thing that do not rely on having composer as an actual requirement of your codebase. Granted, I haven't seen why or how of the specifics of #86, but it strikes me as a surprising, off label way to use composer.
Comment #109
xjmDiscussed further with @catch -- this issue isn't under consideration for 9.0.x since its final patch release is next week. However, for 8.9.x, it seems like there's further discussion needed about whether we could loosen constraints and what the implications of that would be.
Meanwhile, we also need:
Comment #110
xjm@catch and I agreed #3179647: Loosen composer/semver constraints to ^1 || ^3 should be discussed as a followup, so let's transfer discussion about loosening constraints over there. A looser-constraint version might be backportable to 8.9.x; however, the version in this issue definitely is not.
So setting it back to 9.1.x for the two points in #109 and changing to NW.
Thanks, and sorry for the noise.
Comment #111
greg.1.anderson CreditAttribution: greg.1.anderson at Pantheon commentedThe composer/semver fix for Drush has been committed, so folks can now use the latest dev release with Drupal 9.1.
Comment #112
xjmThanks @greg.1.anderson; I added this to the release note. We can update the text if/when a new release is created containing the Drush fix.
Comment #113
xjmComment #114
xjmDid a bit more work on the CR, with an
@todo
for the info from #88. https://www.drupal.org/node/3178837/revisions/view/12109180/12111746Comment #115
xjmComment #116
MixologicIm sorta not sure what to add to the CR for #88
Requiring composer/composer into a project and intermingling composer's dependencies with drupal's dependencies is generally considered an antipattern, because you mingle the code responsible for dependency management to be with your own sites dependencies and you can potentially get into a situation where you cannot upgrade because of a conflict between the upgrader itself, and your site. This is not to say that it cant be done, as someday composer/composer may actually ship with drupal to handle updates, but its not tested on it now, so.. not necessarily a good thing to do.
There are two issues that result from implementing this antipattern:
#1. Site owners, on drupal 9.1+ will be forced to upgrade to composer 2, and they may have incompatible plugins, meaning they cant upgrade to 9.1
#2. Site owners, that *do* wish to upgrade to composer 2, cannot do so until they are upgraded to drupal 9.1 without backporting these changes to 8.9.x (re: a href="#comment-13883847">#81)
Both of these issues have the same solution:
first, remove your dependency on composer from your site,
and either install composer 1 in a the typical, recommended way (as an external phar file, somewhere else in your path),
Or, if for some reason it *must* live underneath your project root, you can do another atypical method as follows:
Create a new directory in your project. Example:
You will see a prompt, answer "n"
No composer.json in current directory, do you want to use the one at /Path/to/projectroot? [Y,n]? n
This will leave you with a version of composer that can be run with ./vendor/bin/composer yet not be intermingled with the dependencies of your site.
The other option is to open up both dependencies to allow for broader acceptance (composer/composer: ^1.9.1 | ^2.0 and https://www.drupal.org/project/drupal/issues/3179647) but that comes with a lot of risk wherein we dont have any tooling to support that kind of openness, and thus may end up breaking somebody, sometime in the future.
Comment #117
seanrWhere does this stand currently? I've run into this issue and can't tell what's going on here right now. :(
Comment #118
catch@seanr this has been committed to 9.1.x (and above), it won't be backported to 8.9.x because that's in security-only mode now.
If you're running into an issue, you probably need to do something like #116.