Problem/Motivation
Composer v2.2 includes a new security feature https://getcomposer.org/doc/06-config.md#allow-plugins
As of Composer 2.2.0, the allow-plugins option adds a layer of security allowing you to restrict which Composer plugins are able to execute code during a Composer run.
As a result of that update, without an implicit config setting in composer.json, commands including composer create-project drupal/recommended-project drupal9
and composer update
will not complete running until the user replies y
multiple times during the operation.
Creating a "drupal/recommended-project" project at "./drupal9"
Installing drupal/recommended-project (9.3.0)
- Installing drupal/recommended-project (9.3.0): Extracting archive
Created project in /var/www/html/testy/drupal9
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Package operations: 62 installs, 0 updates, 0 removals
- Installing composer/installers (v1.12.0): Extracting archive
composer/installers contains a Composer plugin which is currently not in your allow-plugins config. See https://getcomposer.org/allow-plugins
Do you trust "composer/installers" to execute code and wish to enable it now? (writes "allow-plugins" to composer.json) [y,n,d,?] y
- Installing drupal/core-composer-scaffold (9.3.0): Extracting archive
drupal/core-composer-scaffold contains a Composer plugin which is currently not in your allow-plugins config. See https://getcomposer.org/allow-plugins
Do you trust "drupal/core-composer-scaffold" to execute code and wish to enable it now? (writes "allow-plugins" to composer.json) [y,n,d,?] y
- Installing drupal/core-project-message (9.3.0): Extracting archive
drupal/core-project-message contains a Composer plugin which is currently not in your allow-plugins config. See https://getcomposer.org/allow-plugins
Do you trust "drupal/core-project-message" to execute code and wish to enable it now? (writes "allow-plugins" to composer.json) [y,n,d,?] y
Steps to reproduce
Using composer v2.2 and above, run composer create-project drupal/recommended-project drupal9
or composer update
.
Proposed resolution
Add the new required config setting to Drupal core's composer.json
"config": {
"sort-packages": true,
"allow-plugins": {
"composer/installers": true,
"drupal/core-composer-scaffold": true,
"drupal/core-project-message": true
}
},
This update should be ported to all Drupal core versions.
(It also affects installation of Drupal 8 with Composer 2.2, but since Drupal 8 is EOL, I am not sure if this update would happen)
1. July 2022: The patch has been committed to the repo, and will apply on the next release 9.4.2 and forward.
Remaining tasks
User interface changes
API changes
Data model changes
Release notes snippet
Composer 2.2 introduced a new security feature that requires Composer projects to authorize plugins. This change means that Composer commands to install and update Drupal projects will fail unless either the required plugins are authorized in the project configuration or the user manually replies "y
" to a prompt to authorize the plugin. The prompt can break continuous integration and deployment workflows, so it is recommended that projects authorize core's required plugins (and projects using development dependencies should authorize development plugins).
This release configures drupal/recommended-project
to authorize core's required and development plugins by default. Existing projects may also need to update their configuration to authorize these plugins. For more information, review Composer 2.2+ authorized plugins.
Comment | File | Size | Author |
---|---|---|---|
#88 | interdiff_70-87.txt | 6.24 KB | Spokje |
#87 | 3255749-87-d93.patch | 9.94 KB | Spokje |
#78 | interdiff_72-78.txt | 1.11 KB | Spokje |
#78 | 3255749-78-d10.patch | 7.78 KB | Spokje |
| |||
#71 | interdiff_65-70.txt | 2.74 KB | Spokje |
Issue fork drupal-3255749
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
shaalComment #3
shaalAlternative ways to bypass this issue -
composer create-project drupal/recommended-project drupal9 -n
Keep in mind that you'll have to add `-n` for every composer command afterwards.
allow-plugins: true
to composer.json, and it won't check for the permission of each composer plugin:composer config allow-plugins true -n
Comment #4
shaalThis composer update can break exiting Drupal projects' automatic builds in CI.
@larowlan mentioned on Slack that he started seeing quite a few RTBC patches knocked back to "needs work", and that perhaps it's related (ie.
https://www.drupal.org/pift-ci-job/2273030)
Comment #6
SpokjeComment #7
Martijn de WitIf tests are failing because of this, should this be a "critical" ?
Comment #8
Martijn de WitComment #9
cilefen CreditAttribution: cilefen commentedFWIW I don’t see an issue like this as a bug but rather, a task.
Comment #10
heddnThis is causing many, many test failures. Because of that, I think it should be a bug.
Comment #11
cilefen CreditAttribution: cilefen commentedI disagree—this is a new feature in Composer that defaults to "on". That's a BC-break on their part. It's critical either way so that's fine.
But the testbots are only one concern. Soon site owners will be opening "bugs" about this. They will actually have to change their own composer.json files. So at some stage the issue summary above should be such that we can direct site owners to it. Maybe a "Recommendations for Site Owners" at the top would be useful.
Comment #12
alexpottThis issue has become about two different problems. One if the fact that composer create-project requires user input and the other is that core tests are failing. Here's a patch that deals with core tests.
Comment #13
alexpottI opened a new issue for the test failure - #3255836: Test fails due to Composer 2.2 - let's leave this issue to cope with the problem reported.
Comment #14
alexpottI suggest that this issue adds the config to the project templates as added by #3253683: Improve compatibility with composer 2.2
Comment #15
benjifisherDo we want #3255836 to target 10.0, not 9.4?
I am not sure whether I should add the following here or on that issue:
The first commit on MR 1579 did what the issue summary says, also adding
drupal/core-vendor-hardening
. The test fails at line 84 ofcore/tests/Drupal/Tests/Composer/Plugin/Scaffold/Functional/ComposerHookTest.php
:That bit about "You have until July 2022 to add the setting. Composer will then switch the default behavior to disallow all plugins." agrees with #11: it looks like a bug in Composer.
The second commit on the MR reverts the first one: maybe that was an accident? It also adds a few lines to
composer-hooks-fixture/composer.json.tmpl
. Now the test fails at Line 79. I tried restoring the first commit, and it still fails at Line 79. I addeddrupal/core-composer-scaffold
and got back to Line 84, with the same message as before, with or without restoring the first commit.I do not have time to keep working on this.
Comment #16
SpokjeAs far as I can tell the introduction of PHP 8.0.13 on Drupal CI introduced this test failure, which seems to only happen on the PHP 8.0 TestBots.
So I think we should target all supported branches.
By the looks of it using the
--no-interaction
when calling composer will automagically answer "Yes, I want to give access to the plugin", but it can't hurt to explicitly grant it.Comment #17
cilefen CreditAttribution: cilefen commentedOh this only happens with “create-project”? That’s not as disruptive as I thought.
Comment #18
benjifisher@alexpott is right in #12 and #13: there are two separate problems here. The fix for one of them on #3255836: Test fails due to Composer 2.2 looks good to me. Let's return this issue to making sure that
composer create-project
works. See the "Steps to reproduce" in the issue summary.The Critical issue is #3255836. I am moving this one back to Major. I would not object to Normal.
Comment #19
shaal@cilefen, in existing projects, as long as composer.json missing allow-plugins config settings - ANY composer command will display the warning/question to the user.
Comment #20
AaronMcHaleCan confirm what Ofer said in #19.
I hit up against this today when running
composer update
, so it's not specific tocomposer create-project
.While we're on the topic of semantics:
How significant is something before it's marked as critical, I feel like this is probably a "significant regression".
Comment #21
AaronMcHaleUpdated titled and IS to note that it also impacts
composer update
.Comment #22
AaronMcHaleFrom IS:
Does it work if we add it to only Core's
compose.json
, I presume it's the root package that needs this setting, so should it be added todrupal/recommended-project
anddrupal/legacy-project
?Comment #23
cilefen CreditAttribution: cilefen commentedIt's
composer install
too. It's most things.If you pass the
--no-interaction
flag to Composer, it does not ask, but it shows this warning:Most CI pipelines should pass
--no-interaction
but who knows whether they do.Comment #24
mglamanThe problem is that
composer show
also causes hangs when TTY in a CIComment #25
neclimdulSeems like tests are still broken.
Comment #26
klonosJust noting that I've tried installing latest D8.9.20 (I know it's EoL - just for testing things) and have the following to report:
--no-interaction
(without the patches provided here) reduces the prompts from 4 or 5 down to only 1hirak/prestissimo
, and even if I add that to the.config.allow-plugins
section of my composer.json, I still get prompted about it.Hope this helps.
Comment #27
rfayUse of --no-interaction will be obsolete this summer, as they'll cease to allow that approach as I understand it.
I'm sure you know that hirak/prestissimo was made completely obsolete by composer 2, at least as far as I understand it.
Comment #28
SpokjeSo I'm seeing the message
on all automated test branches of
9.3.x-dev
,9.4.x-dev
and10.0.x-dev
near theDrupal\Tests\Composer\Plugin\Scaffold\Functional\ManageGitIgnoreTest
test except when testing 10.0.x-dev with PHP8.1I hope the attached patch (which seems to apply on all mentioned branches) will prevent that message.
I've tried hard to create a test, but somehow the output message above wasn't showing up in the output of the scaffolding tests nor with
$this->assertExpectedOutput()
.I bumped the version of this issue to
10.0.x-dev
and added the widely fearedNeeds tests
tag.Comment #29
longwaveUsing
2>&1
works locally - the message is printed to stderr, whichexec()
does not capture otherwise - and switching tomustExec()
avoids the output hanging locally while Composer waits for input.Nit: this should be indented two characters, not four.
Comment #30
SpokjeAh, that's where the #WindowsUsingDev (=me) always fails, command-line *nix trickery.
Thanks @longwave, if your fail-test returns red, I'll wrap up a new version including test and correct indentation.
Comment #31
SpokjeAs expected, the test-only patch from #29 fails on
Drupal\Tests\Composer\Plugin\Scaffold\Functional\ManageGitIgnoreTest
on both9.3.x-dev
with PHP 7.4 and10.0.x-dev
with PHP 8.0, but passes on that branch with PHP 8.1I've created a new patch, which includes the test and fixed my incorrect indentation from patch #28.
Comment #32
longwaveDo we know what the difference is with PHP 8.1? Is it just that the PHP 8.1 container on DrupalCI has an older version of Composer?
If so, #3182188: Core composer tests should use local composer vs host composer would make this more consistent.
Comment #33
SpokjeThe point in time we all fear: Intelligent questions are asked...
Latest composer version out there: 2.2.6.
From the land of
10.0.x-dev PHP 8.1 & MySQL 5.7
:Downloading composer/composer (2.2.4)
From the land of
10.0.x-dev PHP 8.0 & MySQL 5.7
:Downloading composer/composer (2.2.4)
But seeing that the drupalci/php-8.1-apache:production was last pushed 3 months ago, and all other
drupalci/php-x.x-apache:production
all were pushed 2 months ago, I think that the local composer versions are indeed in play.Local composer is pulled in during the image building:
/bin/sh -c curl -o /tmp/composer-setup.php https://getcomposer.org/installer && curl -o /tmp/composer-setup.sig https://composer.github.io/installer.sig && php -r "if (hash('SHA384', file_get_contents('/tmp/composer-setup.php')) !== trim(file_get_contents('/tmp/composer-setup.sig'))) { unlink('/tmp/composer-setup.php'); echo 'Invalid installer' . PHP_EOL; exit(1); }" && php /tmp/composer-setup.php --filename composer --install-dir /usr/local/bin && curl -Lo /usr/local/bin/drush https://github.com/drush-ops/drush/releases/download/8.3.5/drush.phar && chmod +x /usr/local/bin/drush && /usr/local/bin/drush --version
Comment #34
cilefen CreditAttribution: cilefen commentedComment #35
cilefen CreditAttribution: cilefen commentedWe are starting to get help reports from people who answered "no" to the plugin prompts.
Comment #36
SpokjeComment #37
SpokjeRight, so the current state of the MR solves a related problem on the drupal CI environment, but _not_ the actual problem which is:
Following the instructions here: https://www.drupal.org/docs/develop/using-composer/starting-a-site-using-drupal-composer-project-templates#s-how-do-i-set-it-up will require the user the approve installing currently unauthorized plugins by answering "
y
es" several times to questions likeLet's split the (semi-)related drupal CI environment issue about that into a separate issue and work on the above issue here.
Comment #38
SpokjeMy water-muddling, side-stepping, mildly related patch/MR now lives on in #3277025: For additional security you should declare the allow-plugins config with a list of packages names that are allowed to run code..
The solution from @shaal in the IS is absolutely right, I "just" need to get a test up-n-running to confirm that.
Am very close to that and will be cleaning up this issues MR/patches within 20 hours from now (scouts honour!)
Comment #39
SpokjeComment #40
Mile23Was going to comment earlier about the need to rescope some of this.
For all our packages, if they require plugins, then we should explicitly allow those plugins in the config section.
We should also have a test for all of our packages which fails if Composer asks to add a new plugin. That way we know what our dependencies are requiring on our behalf, and we can address it as needed.
Here are the packages. I'll say that there's not a test, but that's not from looking, only deduced because we have this issue. :-)
drupal/drupal: Addressed by #3253683: Improve compatibility with composer 2.2 but does not have a test.
drupal/core-dev: Generated from drupal/drupal.
drupal/core: Does not require any plugins, does not have a test.
drupal/core-recommended: Generated from drupal/core.
drupal/recommended-project/legacy-project: Uses plugins and does not allow them, does not have a test.
I'm not sure why we need to spin off #3277025: For additional security you should declare the allow-plugins config with a list of packages names that are allowed to run code. but we do need to scope some work within the generated project templates and also add some tests.
Comment #41
SpokjeSwitching back to patch workflow so it can be tested more easily against different branches.
Comment #42
SpokjeComment #43
SpokjeAbove patches should prove
drupal/recommended-project/legacy-project
now has test-coverage for non-allowed composer plugins and that both projects now don't require manual "y
es-sing" to allow any composer plugins any more.Blissfully ignored the rest of #40, so there might be need for follow-ups. I think this one alone is disruptive enough to get in separately and first.
Comment #45
SpokjeFail patches gonna fail (all of #42)
Fail in #41 for
9.4.x-dev
is an unrelated JS test failure.Fail in #41 for
9.2.x-dev
is a CSpell failure on the Composer plugins, let's tackle that when we have agreement if this patch is the way to go and if it should be backported as far back a 9.2Comment #46
SpokjeComment #47
alexpottI don't think we should be allowing plugins for things that are not part of that versions build.
phpstan is only part of the Drupal 10 builds.
I'm not too sure when phpcodesniffer-composer-installer became part of coder but it is quite recently - and for sure not part of Drupal 9.2.
Comment #48
SpokjeComment #49
SpokjeComment #50
SpokjeComment #51
SpokjeThanks @alexpott.
1.
Indeed, removed in
3255749-d9.4_and_d9.3-49.patch
2.
I was as surprised as you probably will be, but https://www.drupal.org/pift-ci-job/2368237 proves it _is_ already in D9.2.
3.
From (at least) D9.2.x the dev dependency is part of the default package for drupal/recommended-project and drupal/legacy-project.
Since there's no mention about a no-dev option in the docs here: https://www.drupal.org/docs/develop/using-composer/starting-a-site-using-drupal-composer-project-templates#s-how-do-i-set-it-up, I think it's safe to assume that's what ~90% of the users will get when they c/p the
composer create-project drupal/recommended-project my-project
command.Comment #52
SpokjeThe above leaves us with 2 patches.
1.
3255749-d10.0-49.patch
: For10.0.x-dev
, allows:for
drupal/recommended-project
andfor
drupal/legacy-project
2.
3255749-d9.4_and_d9.3-49.patch
For9.4.x-dev
and9.3.x-dev
, allows:for
drupal/recommended-project
andfor
drupal/legacy-project
Note:
3255749-d9.4_and_d9.3-49.patch
should probably work on9.2.x-dev
, but there seems to be a change in the workings of cspell between 9.2 and above, since in 9.2 *.json files seem to be spellchecked somehow.I don't feel enough urgency to find out what and why changed.
Comment #53
alexpottRe #51.2 Well that's an interesting thing because the test is running 9.2.x but according to the logs it is install 9.3.12! I think that this is an interesting thing - maybe the test should use
create project --no-install
and thenupdate --prefer-lowest
- I dunno. I think we need to take steps to ensure we're testing the codebase provided and it looks like we're not.Comment #54
alexpottThis should be
copy($this->getDrupalRoot() . '/' . $package_dir . '/composer.json', $composer_home . '/composer.json');
so the test can be run locally from the root directory of a core checkout. ATM it has to be run from inside ./core/Comment #55
Spokje#54: Nice catch @alexpott!
Let's see what fails if we use that in a test-only-apocalypse-now patch
Comment #56
alexpottHere's a better test that tests the current codebase under test.
It addresses the following issues with the current test:
Comment #57
alexpottFor reasons I can't quite explain yet it only catches when
drupal/core-composer-scaffold
anddrupal/core-project-message
are not present. And it does not current fail when any ofcomposer/installers
,phpstan/extension-installer
ordealerdirect/phpcodesniffer-composer-installer
and not present. I guess this is down to how \Drupal\BuildTests\Composer\Template\ComposerProjectTemplatesTest::makeVendorPackage() works - maybe we are missing something there.Comment #59
SpokjeComment #60
SpokjeComment #61
alexpottPatch attached fixes \Drupal\BuildTests\Composer\Template\ComposerProjectTemplatesTest::testTemplateCreateProject() so that composer plugins are triggered as expected by it.
Comment #63
SpokjeComment #64
SpokjeComment #65
SpokjeComment #66
SpokjeComment #67
demeritcowboy CreditAttribution: demeritcowboy commentedThanks for this. Commenting that the patch works for me:
Comment #68
longwaveTested the patch on 9.5.x and 10.0.x - this fixes the current fails in HEAD in ComposerProjectTemplatesTest caused by the Composer plugins time bomb expiring today.
Comment #69
SpokjeNot showing in the daily QA-runs (yet), but there are also 2 failures here:
Uploading patch which should fix both.
Comment #70
SpokjeComment #71
SpokjeComment #72
SpokjeComment #73
SpokjeComment #74
SpokjeComment #76
Spokje3255749-d10-64.patch
is old hatComment #77
longwaveI think the 10.x patch also needs to allow
phpstan/extension-installer
as in #64.Comment #78
SpokjeYep, nice catch.
Comment #79
SpokjeUnsure if we want to backport this as far as
9.3.x
, but if we do we need another patch.Comment #80
SpokjeComment #81
longwaveThanks @Spokje.
Random JS fail on 10.1.x so triggered a retest. Otherwise this is good to go to fix HEAD in 9.4.x, 9.5.x, 10.0.x and 10.1.x.
As 9.3.x still receives security support I think we should backport there to keep tests green, but the fail looks a bit strange so let's investigate that separately - I sent it for a retest in case it is a random thing.
Comment #85
catchs/drupal/Drupal
Something off with the indentation here, fixed both on commit.
Committed/pushed to 10.1/10.0 and 9.5/9.4 respectively.
I think we should get this in 9.3.x so moving there for backport.
Comment #86
SpokjeComment #87
SpokjeProblem with
9.3.x
seems to be that #3277025: For additional security you should declare the allow-plugins config with a list of packages names that are allowed to run code. never made it into that branch.Attached patch is a combination of
3277025-3.patch
and3255749-70-d9.patch
, with the fixed-on-commit remarks from #85 rolled into it.Comment #88
SpokjeComment #89
mikelutzJust tested #87 locally, it both clears the prompts on composer install, and fixes the test, which was broken for me on the old patch. LGTM if the tests pass.
Comment #91
catchCommitted/pushed to 9.3.x, thanks!
Somehow I had completely missed there was a July time bomb with this one...
Comment #92
chingis CreditAttribution: chingis commentedCan someone who has permissions update the default branch in https://github.com/drupal/recommended-project to 9.4.x. Currently it's 9.1.x and still has this bug.
Comment #93
ressa CreditAttribution: ressa at Ardea commentedI still get the
Do you trust "drupal/core-project-message" to execute code ...
prompts with Composer 2.3.6:Comment #94
SpokjeThe fix in the patch has been committed to the repo, but will only apply on the next releases, 9.4.1 (the current release for 9.4.x) was released 13 days ago, so before this issue landed.
9.4.2 (and up) _should_ be smooth sailing.
Comment #95
benjifisherI get something similar to #93 even when I ask for the dev version:
Comment #96
ressa CreditAttribution: ressa at Ardea commentedThanks for clearing that up @Spokje, as well as your efforts here, I really appreciate it. I have added a note in the Issue Summary.
Comment #97
shaal@benjifisher I created #3294205: [Packaging broken!] Composer v2.2 prompts to authorize another plugin when stability=dev to fix issue #95 that you have found.
As far as I can tell, it is not the same issue as in #93
Comment #98
benjifisher@shaal:
Thanks for looking into that. That makes sense: the plugin I saw in #95 is not the same as the one @ressa reported in #93.
Comment #99
Mile23Just confirmed:
Branch 9.4.x contains the plugin permissions: https://github.com/drupal/recommended-project/blob/9.4.x/composer.json#L32
Tagged release 9.4.1 does not: https://github.com/drupal/recommended-project/blob/9.4.1/composer.json#L28
I'd assume that tagged release of recommended-project 9.4.2 will contain the new config. The schedule says that could be tomorrow... :-)
Comment #100
xjmSoooo this issue broke packaging, hooray! #3294205: [Packaging broken!] Composer v2.2 prompts to authorize another plugin when stability=dev seems to have fixed it, we think.
Based on this experience, adding it to the release notes for both the bugfix release and the alpha. TBD what we'll do about 9.3.x, but it would bad if packaging were also broken there and we did not find out until a security release.
Comment #101
xjmComment #102
xjmThis could do with a CR.
Comment #103
NickDickinsonWildeDrafted https://www.drupal.org/node/3294646 (unpublished)
Comment #104
xjmAdded a release note to the IS that should cover both this and #3294205: [Packaging broken!] Composer v2.2 prompts to authorize another plugin when stability=dev.
Comment #105
xjmComment #106
SpokjeI've split the example code in the CR into a 9.x and 10.x example. Since PHPStan is a 10.x-only thingy, the plugin
phpstan/extension-installer
doesn't have to be allowed on 9.x.Since the CR was seemingly the only reason this issue was set back to
Needs Work
and then toNeeds Review
, and seeing the CR was published, I'm putting this one back toFixed
.Comment #107
NickDickinsonWildeComment #108
masipila CreditAttribution: masipila as a volunteer commentedI'm not sure if this thread is an appropriate place to mention this observation, but there is a difference in the drupal/recommended-project composer.json and the change record of this d.o issue.
https://github.com/drupal/recommended-project/blob/9.4.x/composer.json#L32
vs.
https://www.drupal.org/node/3294646
The change record has this, which is currently not present in drupal/recommended-project:
I'm not an expert on this topic so I don't have a position which one is correct and which one is not, but I think it would be nice that the change record would be aligned what we have in the drupal/recommended-project
Cheers,
Markus
Comment #109
catch@masipila that's because #3280589: Deprecate Composer Vendor Cleanup Scripts and related issues landed. I think we can just remove that line from the CR.
Comment #110
ressa CreditAttribution: ressa at Ardea commentedI just installed the latest version 9.4.2 released July 7th, and can confirm that it is smooth sailing again, which is really awesome:
Comment #112
Nicolas Bouteille CreditAttribution: Nicolas Bouteille commentedFYI: composer self-update (from 2.2.5 to 2.4.1) solved it for me on Drupal 8.
The allow-plugins options is not enabled by default anymore, there's just a message at the beginning:
For additional security you should declare the allow-plugins config with a list of packages names that are allowed to run code. See https://getcomposer.org/allow-plugins
Comment #113
phenaproximaI'd like to see the change record updated to provide commands for people to set the appropriate changes in composer.json, without needing to know how to edit JSON.
The commands are:
This should also be added to the release notes.
Comment #114
DamienMcKennaGreat idea @phenaproxima!
The commands appear correct.
Comment #115
quietone CreditAttribution: quietone at PreviousNext commentedI have updated the change record per #113, so I am removing the tag. I don't think the command line commands need to be in the release note now that they are in the change record, which is linked in both the 9.4.2 and 9.3.18 release notes. Therefor removing another tag.
That addresses the points in #113 so I am restoring the Fixed status.