Remove some minor constraints from core-recommended

Would this have consequences for testing infratructure / matrix / setup on drupal.org? The recommended project is explained in its own readme as:

Require this project instead of the drupal/core subtree split in order to guarantee that all of the dependencies from drupal/core will be included in your Drupal site at exactly the same version that was tested with the version of Drupal you are currently using.

Remove the minor constraints from core-recommended and have it be essentially identical to running drupal/core.

Makes sense to me as long as the core-recommended Git repo is archived, or otherwise marked as deprecated. Archived repos still function and no sites would be affected.

+1 on this. We're having discussions internally about adopting an ADR to always use drupal/core and never drupal/core-recommended due to how many times we've been blocked on these security upgrades. Improving this upstream would be much better.

+1 to the general idea. This approach was fine until the sudden recent flurry of upstream security updates and I don't think those will decrease from here on in. And the problems that led us to this approach in the first place have largely been mitigated by removing the problematic dependencies altogether.

+1 to #3, we can still keep the namespace around and un-abandon it if we want to reintroduce something again in the future.

We all bought into the stability (even me who lives tends to live in -dev). I have always had a competing argument (with myself) that I’d rather have a broken site than a hacked site.

Supply-chain attacks slow my (addict level) updates down too.

I am on board with essentially moving back to Drupal/core but wonder if we can ~ the versions more (haven’t read the constraints yet, typing on phone) to let more patch releases through?

MRs look good, but I'll try to actually manually test tomorrow. Unless someone else RTBCs first.

I like that this keeps the metapackage around since it doesn't require us to change all projects to remove core-recommended. That makes sense to me.

OK, manually tested with these steps:

1. Pulled down branch for MR 16071
2. Created a new project in separate folder: composer -n create-project drupal/recommended-project:dev-main my-project
3. Observe the version numbers of packages installed in the new project, such as guzzlehttp/psr7 (2.10.4)
4. Edit composer.json in the new project and added this to repositories, which points to the checked out MR's core-recommend composer.json:
   

             {
                 "type": "path",
                 "url": "../drupal/composer/Metapackage/CoreRecommended",
                 "only": ["drupal/core-recommended"]
             }
     

5. Ran composer update guzzlehttp/psr7 drupal/core-recommended
6. Observe guzzlehttp/psr7 can now update to 2.11.0

This RTBC +1 for me.

Edited: I also did a minor revision of the CR.

Thanks, this is great news and will make updates easier for casual Composer users. Adding issues as related, so that this issue is shown there as well.

+1 for #9, let's initially ship this with 11.4.0 to remove the pain quicker.

Not yet sure about backporting to patch releases. Maybe if we don't hit anything unexpected in 11.4 and we get bitten by this again in some dependency in 11.3/10.6 then we could consider backporting at that point, given we'd be bumping dependencies anyway at that time.

Have not reviewed the MR or change record.

I think this is incredibly hasty and not an appropriate release target for 11.4. I think if we wanted this in 11.4 it should have been decided like 3 months ago.

I am still not convinced we are not overreacting to the pain upstream is causing us and throwing the baby out with the bathwater. Like what is even the point of core-recommended without the minor constraints?

I think this could also be solved by having better documentation/naming.
People probably use core-recommended because it is the recommended/better option but we should explain that there is no best option, it is a choice between stability and security.
We should make sure people using core-recommended are OK with waiting for a core release to apply some security patches (or overriding some constraints if needed).

+1 @xjm and @prudloff in #20 and #21. It feels like an emotional reaction to the pain and better understanding of the choices (and workarounds/pain mitigations). Take a breath, sit with pain. Recall the breaking minor changes that needed a lot of work to untangle.

For our teams, I can see we've hit this class of issue 3 times since January. I think that's enough to warrant a change or improvement, whatever that is. From a new-to-Drupal (but not PHP) standpoint, I think the real confusion is that nearly all Drupal sites use drupal/recommended-project to start. If you review that composer.json you see broad constraints like ^11 you're likely to assume transient dependencies follow the same pattern, until you start work one day and everything is broken in your team's pipelines.

Given the core and Drupal CMS strategies, what if the spirit behind locking dependencies moved to Drupal CMS, leaving core free to follow Composer best practices and allow a wider set of versions?

I feel like a change or solution of some sort is warranted. AI is increasing the frequency of security updates in dependencies, and it's something we have to adapt to. It's incredibly frustrating when I run a composer command to update a module, but I get errors because some core dependency is vulnerable. I have to manually look up what the workaround is each time and add it, and then remove it later when it's fixed. This has happened for me 3-4 times in the last month, and I have to do it for each site I want to update anything on.

I think this is incredibly hasty and not an appropriate release target for 11.4. I think if we wanted this in 11.4 it should have been decided like 3 months ago.

There is some sound logic there.

I'm not a fan of core-recommended(I've suggested many times on Slack use drupal/core for flexibility) however it has for years had a defined purpose, tightly pin the release. I would not recommend changing that in a minor release, especially with near zero lead time to inform site owners.

As a site owner (if I was using the package) I would hope for one of the following:

• Marking abandoned would be best as it informs me that this project no longer does what it has for years.
• Waiting to D12 and giving a significant PR announcement to increase chance that I'm aware of it would be second best

There is a significant of community documentation out there both on D.O. and not D.O. that needs to be considered as impacted by whatever decision is made here.

I am still not convinced we are not overreacting to the pain upstream is causing us

A good point, as I have seen a few times lately feel like knee jerk reactions.

We might decide to add constraints back at a later date... Or if not, we could deprecate it at a later date. But I don't think we need to decide that now...

Whatever the package future is I would suggest the core team should have that defined and documented for users. It should be clear to a site owner when and how it may be updated in the future and not a 'lets decide this later' if this change were to be adopted.

The team should likewise consider that any blocked upgrades will, no matter how the package is advertised, likely be expected to be remedied in short order, especially with a name containing "recommended", rebranding the package may be the teams best option.

The current issues are arguably more of a problem for Drupal CMS sites because it's aimed at people who are less comfortable with composer - e.g. they're not expected to make a decision to switch from core-recommended to core, or allow-list CVEs temporarily, or necessarily to run composer from the cli at all. It's not even possible to install Drupal CMS at the moment.

To me that describes exactly why Drupal CMS should be responsible for locking requirements not core, they are intended to be the "hand holding" project, they can evaluate "is any recipe we include impacted" or "is any combination of 5 modules a threat vector for this flaw", they can make more controversial choices such as "lets de-register that class".

Drupal CMS is inherently an "opinionated" deployment, they can make these decisions that core can't/shouldn't.

guzzlehttp/psr7 has a new SA today and corresponding new minor release: https://github.com/guzzle/psr7/security/advisories/GHSA-vm85-hxw5-5432.

The SA has not yet hit Packagist, so composer isn't blocking installs yet, but presumably it will eventually.

#3603733: Update guzzlehttp/psr7 to 2.12.1 is issue on updating the package.

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

I had suggested in Slack that Drupal CMS could switch from depending on drupal/core-recommended to depending on drupal/core directly. I think that would pretty much put these problems to bed for our target audience.

However, I didn't want to do that without agreement from the core release management team. Drupal CMS can make these kinds of decisions, but we really shouldn't do it unless we're all on the same page. So I think there might still be some discussion needed before Drupal CMS makes this change, because if core-recommended decides to loosen up, we don't need to change anything.

And for the record, our strategy would not be to pin any core dependencies more tightly; we would be more likely to just let drupal/core's dependency constraints do their thing, treating them the same way we already treat our normal contrib dependencies. We have found it very helpful that, when a contrib dependency has a security release, our official line has been "no worries, you're safe, we don't pin dependencies". I think that's best for end users and keeps friction and frustration to a minimum.

How about a half-way between removing all the restrictions from drupal/core-recommended and staying as we are?

For those dependencies where it's known that minor updates are likely to or have caused issues in the past because of how they are maintained or otherwise, like twig, those are kept pinned by core-recommended, but other dependencies that are more stable, aren't pinned.

As a user of drupal/core-recommended I very much appreciate that someone from the core team has reviewed and asserted that the new version of the dependency works with Drupal, and I very much appreciate that stability.

We've been avoiding drupal/core-recommended for a long time without issues, but that's due to extensive e2e-pipelines that we run prior deployment. It's probably not a good path forward for the less technical users.

However, the pain is real. And it's not just those DMCS users who are not really into the console, let alone composer.json. We've see many tech savy Drupal users in Slack and social media who really struggled with that stable-vs-security balance in the last few months. And there doesn't seem to be any sign that this is going away any time soon.

Changing the dependency from drupal/core-recommended to drupal/core in project templates and in DMCS may be a solution for new installations in the future. But that leaves all existing installations still alone, feeling the pain, and getting frustrated with Drupal, although the issues comes from somewhere else.

Therefore, keeping the recommended project and either remove the constraints or updating that project more frequently would be my preference. More frequent updates as mentioned in #32 sounds really tempting as it still goes through some vetting and users still get something that's recommended by the maintainers.

I'd be in favor of this, getting around a single dependency is pretty straight forward, but the more recent one where there were two projects that needed the version as override that themselves had conflicts made it far harder to update.

It'd be good for recommended to know the minimum tested versions, e.g. don't go below x major.

Presumably at some point going forward ANY composer updates required will be added to the RELEASE page, exactly like this:

composer update "drupal/core-*" --with-all-dependencies

For example, from the enormous cascade of error messages when I ran the above, I need to do something about guzzle. But this:

task: #3603733 Update guzzlehttp/psr7 to 2.12.1 and guzzlehttp/guzzle to 7.12.1

is not actionable without doing a lot of detective work. This:

composer update guzzlehttp/guzzle

would be actionable, but that's not there.

The alert says "you should update immediately!"

Glad to, as I have been, if you give me the tools to do that. Same goes for updating composer.json and all the other suggestions for tweaks I saw floating about.

+1 for the idea of more frequent releases of drupal/core-recommended and +1 for adding an extra component to the version string (as in Comments #32, #34).

Famous last words: how hard can it be?

The generated repo for drupal/core-recommended is just one file (composer.json; two files, if you count README.md in the default and master branches). So other changes to the main repo will have no effect: all that matters is composer/Metapackage/CoreRecommended/composer.json.

All we need is a CI job, and it seems to me that the only questions are implementation details. For example: what triggers theCI job? Some possibilities:

1. A core committer manually triggers the CI job.
2. A 4-part tag (like 11.3.12.1) is pushed to the repo. (Do we have to update other triggers to ignore 4-part tags?)
3. A commit is made one one of the release branches that changes composer/Metapackage/CoreRecommended/composer.json.

If we want a simpler, short-term fix until that can be implemented, then I vote for relaxing the constraints on the known trouble makers: twig/twig, guzzlehttp/*, maybe symfony/polyfill-*.

I left some comments on the MR. I see the issue status is already NW.

When the MR is updated and the tests still pass, I will re-review unless someone else does it first. If I set the status to RTBC, it will be with the understanding that the release managers still have to decide whether this short-term fix is a good idea.

The downside to this is it's going to be a fair bit of work to set up, but I think we could initially relax the constraints as in this issue, then add that tooling/pipeline on top.

Tagging for a followup for this comment.

I updated the issue summary with the latest proposal.

I prefer this more nuanced approach than the more drastic, 'remove all constaints'. My concern remains communication and documentation. How will this change be communicated to sites so they understand what it means when updating, the risks and the advantages. And what about the README for core/recommended. How does this change that?

I reviewed the changes in MR !16151, and they match the Proposed resolution. Thanks to @quietone for updating that. (A refinement to that section: we are keeping the minor-version constraint for guzzlehttp/promises but not the other two guzzlehttp packages.)

I had in mind sorting within each section. I see that the MR now sorts the entire list, which is at least as good.

For testing, I realize that composer/Metapackage/CoreRecommended/composer.json is a generated file. In the top-level composer.json, I see the following in the scripts section:

        "post-update-cmd": [
            "Drupal\\Composer\\Composer::generateMetapackages",
            "Drupal\\Composer\\Composer::generateComponentPackages"
        ],

So I hacked that file, adding another line to the scripts section:

        "gmp": "Drupal\\Composer\\Composer::generateMetapackages",

Then I deleted composer/Metapackage/CoreRecommended/composer.json and regenerated it with composer gmp. There were no changes.

I now see how the hyphens in the Builder class are related to the two guzzle packages not being removed from the generated composer.json. (See my comment on the MR, now resolved.)

And here is the follow up, #3606820: Add a separate release process for core-recommended

I reviewed the change record and it was out of date. I have updated putting the change first and listing the affected dependencies. Back to NR for just that part.

Looks good to me. I would only say: symfony/yaml is listed twice, and IIUC we were pinning to patch releases, not minors

That is from @phenaproxima in Slack after reviewing the change record.

I have removed the duplicate line and nothing to do for the other item as we did pin to minor. Therefore restoring the RTBC

The new approach looks good to me but there is a comment out of place and we will need a separate MR for 11.x as we should include the PHP 8.4/8.5 polyfills too (and also I am not sure the main MR will apply to those branches anyway).

Committed 0ed786c and pushed to main. Thanks!

Will commit the backport as soon as it's green.

Committed and pushed 8db15f2346c to 11.x and e257fc7d212 to 11.4.x. Thanks!

Also published the change record.

Discussed with @catch whether to change the current approach to wildcards, because we will have to remember to update this in future if new transitive dependencies get added; it would be cleaner to be able to say symfony/polyfill-* for example, but let's deal with that in a followup.