Add a composer plugin that supports 'composer require-lenient' to support major version transitions

This will occasionally result in modules being declared compatible with new major versions of core when they're not, but this seems like a much better problem to have.

One of the biggest battles I have in larger projects is dependencies declaring >= compatibilities. A side effect of this for e.g. composer is that if a newer version actually says "Oh wait, I'm not yet compatible" that package managers will happily install older versions, essentially downgrading you as a surprise, even though those older versions are also not yet compatible, they were just not specific enough in their version declaration.

I've seen this especially with declarations of PHP compatibility and expect similar problems when doing this for Drupal core compatibility. I think while this may seem like a good short-term fix for modules which don't get a lot of maintenance, it's going to cause us headaches if we decide we want a bigger breaking change in Drupal 11 or 12, because now all of the sudden my package manager thinks I can run Drupal 11 and tries to install it, even though modules still need updates.

The suggested change here essentially means that I can no longer trust the core_version_requirement.

Could it be #3230209: Remove requirement for core_version_requirement for module to be compatible with Drupal 9 you were thinking of @catch?

I agree, let's encourage >=9 and anyone who knows they have conflicts / want to have different major/minor branches for Drupal versions can control upper limits.

+1000

I've been pushing for this at Acquia for the past few months. In fact, back around Christmas I wrote up a whole blog post as to why we should make the switch: https://japerry.medium.com/the-case-for-drupal-9-dbfa45c52f9e

tldr; core, being a stable dependency for all of the Drupal ecosystem, works much like PHP now in the scope of contrib modules. All of Nicolas' arguments are valid here: https://twitter.com/nicolasgrekas/status/1263023258938548225

The biggest hurtle for D9 readiness was just getting modules to install via composer, even if they otherwise were compatible. An artificial barrier is made making it much harder to upgrade modules without going through each dependency.

There isn't a perfect solution, but IMHO allowing modules to break, causing specific, fixable, bug reports in issues, is much more preferred than the generic "please make this one line change so we can see what breaks" policy we've been dealing with.

As a site admin and a module maintainer: -1 from me for same reasons as expressed by KingDutch. I've seen those issues raised in the #maintainers Slack and they are messy to deal with.

^9 is already actually a bit loose for many projects that depend on 'non api' code (such as extending controllers and non-base classes) without realizing it.

I want to reiterate what KingDutch said in another way, this would be equivalent to saying a module written for Drupal 8.0 should have indicated it is compatible with D10.

I've already seen some modules that need to open new branches for D10 support because changes are significant and require API breaks that can not really be kept BC.

If sites are having issues with maintainers taking a long time to release new updates that is a an indicator the module is either under maintained OR has a stricter release policy than the site wants to accept and in my opinion not a reason to encourage maintainers to maintain the modules even less.

Also not a fan of this, fully agree with the comments by @cmlara, @Kingdutch.

Yes, it can be annoying. Also seeing that again right now as I start to make my modules D10 compatible, but the require-dev network is complex and sometimes there are even cross-dependenciec betwen projects (ERR has a diff.module tests, paragraphs has 9 other projects as require-dev for integration tests).

But that is IMHO not worth the downsides. I've had those exact cases too, that this is something that you can never take back, once a release is out there and claims to be D10 compatible, that will be possible to install. forever. And as mentioned, D10 has some hard breaks that in some cases require you to do a new major version, then you do not want existing sites to update to incompatible versions and create support requests.

As (more or less active) maintainer of a lot of big projects like token, pathauto, redirect, ERR, paragraphs, I have no intentation of switching to this. It might be useful for small modules that are unlikely to break, but all these big modules need a non-trivial amount of changes for D10.

Neither option is 'fullproof' but what this issue is advocating for is that the downsides for using >=9 is less than the downsides of doing ^9|^10.

The biggest downside to moving to >=9 is that modules which aren't often updated can break in interesting ways if the user doesn't perform adequate pre-launch testing. This is partially true (minor versions can also break them), and is the primary reason why the Drupal community has collectively advocated for ^9 in the past.

However, the side effects of this decision has been, imho, much worse:
* Users cannot even test a module on Drupal.org or find the problems without getting a maintainer involved. Unless you're a composer expert and can spoof things beforehand, maintainers have to change the upstream branch to ^9|^10 BEFORE the issues are fixed. This effectively negates the whole argument about being compatible.
* It takes away the ability for the community or F/OSS contributors to be effective.

The worst part of >=9 is a module becoming incompatible with the next version of Drupal due to a deprecation. However, even there -- the community can step in and offer patches, which can be added to composer. Composer itself cannot be patched.

In the long run, the automated tools on Drupal.org (that already provide deprecation patches to modules) can say if a module is truly compatible. However, in the short term, using >=9 will enable the community to update modules faster for not just D10 but versions after as well.

The biggest downside to moving to >=9 is that modules which aren't often updated can break in interesting ways if the user doesn't perform adequate pre-launch testing

We're moving in the direction of having a project browser, capable of downloading modules that are supposed to work with your site. If that module could potentially use an api that is gone, and your site now whitescreens after you use the project browser, you're going to have a bad time.

* Users cannot even test a module on Drupal.org or find the problems without getting a maintainer involved.

This used to be true, but it isnt anymore. We solved this with the introduction of the lenient facade https://www.drupal.org/docs/develop/using-composer/using-drupals-lenient.... That allows for developers to do whatever they want without composer interfering with what they want to install. Granted its *current* definition is that "you havent set a core version requirement at all" vs "even if there is one, we want an even less constrained version of this package".

Because ultimately this doesnt have to be a tradeoff between one or the other. We have both now, and the developers have a choice of "Please take off the safety belt, I know what to do if this module breaks my site".

Using an open ended constraint will eventually lead to a scenario where an unmaintained project with an unfixed deprecation gets installed to an end users site and whitescreens them, despite not having any indicators that the project is not actually compatible.

* Users cannot even test a module on Drupal.org or find the problems without getting a maintainer involved. Unless you're a composer expert and can spoof things beforehand, maintainers have to change the upstream branch to ^9|^10 BEFORE the issues are fixed. This effectively negates the whole argument about being compatible.

I haven't' had a project that I have tested this workflow on D.O. yet, however I wonder if issue forks are a method to solve this? They can be opened by the community, can be installed using composer (they are just a git repo) and keep the stable branch 'clean' until the merge is ready.

Setup is a little more than "composer require drupal/something" however IMHO it is far from being a composer expert and doesn't require the facade or other 'dark magic'.

Once an issue fork is open patches can be rolled as needed to allow full support until the new major version is ready to be merged into the real repository. Even an abandoned project can use this method while it awaits a new maintainer being approved by the D.O project ownership team.

@Mixologic: I think your answer got formatted as part of the quote.

> Granted its *current* definition is that "you havent set a core version requirement at all" vs "even if there is one, we want an even less constrained version of this package".

I guess this was sufficient for D9, but for D10, it probably makes sense to make it even "lenienter" as D9 compatible modules now have an explicit core compatibility definition?

@xjm:
> - Must opt-in to use lenient facade, difficult to discover, no help for tarball sites

While this is technically true, tarball sites don't have the problem that the lenient facade is solving in the first place. Nothing is stopping them from downloading whatever version they want manually and then applying a patch, manually. The lenient facade will just bypass composer version constraints, you still need a patch to actually update the core compatibility definition in the .info.yml file.

> However, the side effects of this decision has been, imho, much worse:

I don't believe this is worse. The problem we have/had is that people can't install/update an incompatible module, preventing them from updating. Working around that became easier with the lenient facade as well as merge requests that are exposed to composer. It's annoying, but nothing is immediately broken and people can research and learn how things work. However, if an inexperienced user manages to install an incompatible module as @Mixologic said then their site might be dead with a fatal error and especially if they used an upcoming composer UI integration, then they would have no easy way to recover from that?

I don't believe this is worse. The problem we have/had is that people can't install/update an incompatible module, preventing them from updating. Working around that became easier with the lenient facade as well as merge requests that are exposed to composer. It's annoying, but nothing is immediately broken and people can research and learn how things work. However, if an inexperienced user manages to install an incompatible module as @Mixologic said then their site might be dead with a fatal error and especially if they used an upcoming composer UI integration, then they would have no easy way to recover from that?

But users can be faced with fatal errors that cause certain sections of their site to be dead all the time, this can happen in minor version updates, when other dependent contrib modules make an unintentional backwards breaking, or even a new version of PHP. And I don't think putting a hard block in composer is helpful for end users because they a) won't be using lenient facade, and b) cannot use what is likely an available patch in d.o to fix deprecations.

I don't see lenient facade really a solution here at all since its not meant for end users. And we definitely should NOT be recommending this for end users who are trying to upgrade their sites to D10, and then forgetting to remove it later on. Also, as @berdir pointed out:

As projects add core_version_requirement, they will be removed from the lenient endpoint, and will fall back to the standard packages.drupal.org."

-- so @mixologic, it seems like this wouldn't even work?

It seems long term that its more important that modules have minimum requirements and announce its supported compatibility. Users who install modules that don't yet support a major version of core will get a warning that things might not work as expected. But this is a feature, not a install breaking thing. There are few other areas where applications themselves will automatically refuse to work with more modern APIs, like older Windows, OSX, Linux, and Android apps.

Re #19 Thanks, fixed. accidentally removed a stray >

I don't see lenient facade really a solution here at all since its not meant for end users.

It is meant for end users, but our definition of end users may differ.

-- so @mixologic, it seems like this wouldn't even work?

This is true for the d9->d10 scenario, and at the time was a can that we kicked down the road that we're now coming up on. We have the ability to adjust what we put in the lenient facade and whether we do put something in there that is 'lenientier'.

Open ended constraints are really not much different than no constraints. It would only be useful to have minimum requirements in the rare scenario where somebody has created a project that will only work on d10, during a timeframe when d9 sites are still being built, and would only prevent d9 sites from installing d10 modules, but not d11 sites from installing d10 only modules.

Users who install modules that don't yet support a major version of core will get a warning that things might not work as expected.

Only if that doesnt fully break their site...

Ultimately the problem we're looking at solving is that the drupal module ecosystem has a quiver of rock solid, heavily maintained optional features with attentive, dedicated maintainers, and a *very* long tail of one and done projects that solved a problem for some agency for some project that one time and now nobody is getting paid to continue to shepherd that feature into the future.

The lenient facade allow informed choice for the end users to take the risk that those unmaintained projects will continue to work with their sites, and that with a couple of patches submitted by the community, would get them there. This does mean that the long tail of abandoned projects becomes somewhat visible to folks, and perhaps encourge somebody who is deriving value from a particular project may take on ownership.

Open ended constraints dole out that risk without the end users having any warning that what they are doing is risky.

I dont think I agree with this sentiment about the lenient facade:

- Encourages bad behavior (not updating a module to declare explicit compatibility because a work around exists)

The maintainers don't have any behavior to encourage, because the maintainers are gone, regardless of whether the workaround exists or not. They're gone because there isn't any continued benefit for them to keep maintaining whatever it is they shared.

But users can be faced with fatal errors that cause certain sections of their site to be dead all the time

That used to be more true, but the whole raison d'être of composer, semver, and api compatibilities is to prevent the main reason they used to happen.

Two data points:

• According to the latest Drupal 10 readiness data (form a week ago), of the 7.6k Drupal 9 compatible projects, 3.3k will only need an info.yml file change.
• According to Dries’ last yearly data at https://www.drupal.org/blog/who-sponsors-drupal-development-2020-2021-ed..., of ALL of the community projects ALL of the Drupal community worked on 3.8k projects in a one year period.

AFAIS this shows that likely thousands of projects are left behind yearly, nobody is working on them and nobody is updating their compatibility. One solution for this is the lenient facade that we created as it was impossible to change the codebase of thousands of projects at once in time for rolling out Drupal 9. However, it is evident that a long term plan for the lenient facade and/or more premissive composer limitations is in order because the nature of how Drupal.org projects are being worked on will necessitate it.

The above is forward looking data. There is some current data about the Drupal 8 to 9 transition that @Mixologic provided on the use of the lenient facade in the past 15 days:

composer has asked for /packages/8/packages.json 2,050,555 times and /packages/lenient/packages.json 7242 times. So some usage, but not ‘widespread’ [...] 607 client ips using lenient facade. 79612 using the default

I don’t know if the 607 IPs compared to the 80k IPs is covering those who would want to use the thousand of projects that never became compatible with Drupal 9, or are people giving up updating or postponing indefinitely. We tried a co-maintainer drive that was not very successful and also marked Drupal 8 releases unsupported to send a better signal that projects need to move on. There were 3598 projects we had releases marked unsupported (vs. 7277 projects that became Drupal 9 compatible) around the end of last year. So that was 33% of all Drupal 8 compatible projects where a year and a half was not enough to become compatible. Out of those 1431 only had info file problems, so 40% of the releases we marked unsupported could have lived on with a more permissive composer requirement and/or on the lenieent composer facade.

All in all I think its a great thing that a lot of contributed projects get published that people use building Drupal sites. We cannot expect the humans that posted them to sign up for maintaining them for however long. Ideally the humans would be open to maintainer transfers and other humans would want to take over the projects. That did not seem to work out on a massive scale. So looks like we need to solve this with non-humans.

While this is technically true, tarball sites don't have the problem that the lenient facade is solving in the first place. Nothing is stopping them from downloading whatever version they want manually and then applying a patch, manually. The lenient facade will just bypass composer version constraints, you still need a patch to actually update the core compatibility definition in the .info.yml file.

Or 50% of the modules could just work on the D10 site, with no patching needed, since (let's be honest) patching is a barrier for the typical tarball site owner as much as the façade or other workarounds are for a Composer user.

But users can be faced with fatal errors that cause certain sections of their site to be dead all the time

That used to be more true, but the whole raison d'être of composer, semver, and api compatibilities is to prevent the main reason they used to happen.

That was the theory. The practice is that there are just as many fatal errors and unplanned breaking updates as there ever were, along the whole dependency tree; they just come on a more predictable schedule. Source: The people responsible for triaging critical regressions after each core minor release.

If we make this change, we will obviously have documentation that explains the pros and cons of using a >= constraint for your contrib module, alongside common alternatives. Maintainers of complex projects, maintainers who can commit to doing an update once every two years, those maintainers can set their own appropriate constraint. We're obviously not going to mandate using >=. We're just going to make it the default for people who have low-maintenance modules with low-maintenance needs.

Using >= will put the power in the hands of site maintainers if they want it. Might the module still be broken? Sure, but then they're no worse off than they were. And there's a 50% chance it will just work (see Gábor's data above about D9 projects), and tools like Upgrade Status can warn the site owners if there are problems.

It's the right choice for the sustainability of contrib, for the major release cycle going forward, and for making upgrades easier for site owners, which is our top priority for Drupal.

For new sites the discovery should be improved with project browser and ideally on drupal.org too IMHO. I don't think that sets a direction either way for composer requirements of composer projects honestly.

Reminder that this suggestion is also just for core compatibility. We would not recommend dependencies on contrib or other third-party libraries have the open constraint. It's only for core major updates, because they will be highly predictable and increasingly low-impact for smaller contrib projects.

For new sites the discovery should be improved with project browser and ideally on drupal.org too IMHO. I don't think that sets a direction either way for composer requirements of composer projects honestly.

I think it'd be great for Project Browser metrics to take into account the module's upgrade path status, using the data we're already collecting. It could demote modules that allow the current version in their constraint but have known compatibility issues. That way, existing site owners can get on as best they can, but new sites can be steered toward modules with more active maintenance. (Which also should eventually be part of the metrics.)

@catch
> This will resurrect some 8.x-only modules from the dead, although I guess they've already been marked unsupported now so might be excluded anyway?

Maybe I don't understand your argument, but my comment is just about the composer facade, not the .info.yml definition. And exactly those 8.x only modules already work there (that was the whole purpose), but D9-compatible modules won't be installable in D10 based on what I understood.

> According to the latest Drupal 10 readiness data (form a week ago), of the 7.6k Drupal 9 compatible projects, 3.3k will only need an info.yml file change.

I do wonder how that data looks if we limit it to the 10% or so most frequently modules. There's also a considerable grey area of things that you're currently not seeing with static parsing. I haven't used these tools yet for D10 but pretty sure entity query access is one thing. A lot of non-trivial modules are going to have at least one entity query that needs a change. Anything that has at least one test needs change too. file_create_url() and drupal_get_path() are examples of 9.3 only depreciations that affect a ton of modules too. I'd assume that data is driven by a _lot_ of tiny and also rarely used modules.

Return and argument type changes from Symfony sometimes make it hard/impossible to to be D9 and D10 compatible at all and I assume that trend will continue into D11/12 (Entity Reference Revisions is such an example, only avoided a new major release because I decided to drop the problematic HAL integration)

I can just repeat what I said in #9, I have no intention of doing this in any of "my" non-trivial projects. And even for tiny modules there's always a change that even that one single hook it implements is going to change. If we for example decide to replace hooks with a new system in D11/12 then that's going to leave _a lot_ of modules broken. Another fun example would be to move at least part of info.yml into composer.json, your >= 9 compatibility definition would mean nothing if we don't even look for that in .info.yml anymore ;)

If we do want to go in that direction then I'd suggest to just clearly document the advantages and disadvantages somewhere. It already works, anyone who wants to do it is free to do so. And maybe wait out another core release to see how things evolve?

> It's only for core major updates, because they will be highly predictable and increasingly low-impact for smaller contrib projects.

FWIW, not sure I agree with that. A predictable release timeline doesn't imply predictable *changes*?

I wanted to note about entity queries requiring access checks, and other breaking changes: I have discovered some deprecations only use runtime @trigger_error and not @deprecated annotation. The ones I found I have opened issues for and I believe they have been fixed.

But entity query is a nightmare scenario. There's a ticket for phpstan-drupal to catch this with analysis: https://github.com/mglaman/phpstan-drupal/issues/338. However, it's undetected right now until you WSOD. There are other runtime specific changes that are probably not detected.

TBH I don't think there is much overlap with the least found deprecated APIs and the long tail of contributed projects. I would not expect that they appear only needing an info file change because some of those deprecated APIs are not found. Its unlikely that would be the only deprecated API in them.

#32 that'd make me more comfortable. But also, is it worth it? I know just about every Commerce site and module will be impacted. And there are like 300+ of them. Will they get fixed? Maybe. Will this force them to get fixed? For sure. Is there enough time? Doubt it.

Either way, that problem isn't tied to >=9, because it's happen with ^9 || ^10 anyways.

That's why I am all for >=9. It's going to be a mess no matter what for the one-off minimally maintained modules. For the more actively maintained ones, they'll have appropriate constraints.

@Berdir You are the ideal example of an active contrib maintainer, so it is totally appropriate and expected for you to use more accurate constraints. The top 10% of projects will also include many other such maintainers, who can use their best judgment to pick the best constraint.

The recommendation is for projects that mostly work fine without maintenance, where the maintainer may be absent during the development of a new major release, e.g. because Drupal is just one thing they deal with occasionally rather than a central part of their work. Basically, the maintainers/projects for which we could just automate Project Update Bot.

I would discourage everyone using a module where the maintainer can't update 1 line of code every 2 years.

Modules should not automatically declare compatibility with future Drupal major versions (where backward compatibility is broken, by definition).

It is even documented for composer:
https://getcomposer.org/doc/faqs/why-are-unbound-version-constraints-a-b...

I would discourage everyone using a module where the maintainer can't update 1 line of code every 2 years

There's no way of knowing which projects are going to end up with that type of maintenance, and which projects are going to be maintained in perpetuity by responsible maintainers.

We had a really nice round table here at drupalcon and have coalesced around ambitions for a different plan.

Some key takeaways:

1. There are *two* constraints that are used in different contexts, both of them derived from core_version_requirement. The first is the build time dependency constraint on composer, which translates the core_version_requirement to drupal/core, and prevents users from *downloading* a non-compliant project. The second constraint is the *runtime* dependency that drupal itself implements that prevents *installing* a module that doesn't match the criteria.

2. Encouraging open ended constraints suffers from the same problem that we have in encouraging maintainers to update their code to be compatible. In order to unblock end users/site owners, we have to rely on behavior/culture/action of the maintainers, and we're not empowering the blocked site end users/owners to be able to move forward with their upgrades.

3. Most of the risk of "open ended modules" breaking a site only really occur at major version upgrade time (with, of course some edge cases at minors, but the risk is less as time goes on).

4. We can write a composer plugin that gives end users/site owners the ability to effectively do what the lenient facade does -> relax the drupal core constraint to allow a project to satisfy the drupal/core requirements, so that a module can be downloaded and patched in order to be compatible with d10. This could take the form of something like composer require-lenient drupal/some-project that would add some-project to a data array in the extra section of the composer.json. This plugin would then alter the drupal/core requirement to something along the lines of ">= 2versionsback, <= current installed core version" so it would maybe change a contrib module like so:

• Current: core_verison_requirement: ^9
• Becomes: drupal/core >=9, <=10

The proposed composer plugin is essentially an alter hook for composer. Its great because

• We no longer need to have an additional lenient facade, and we do not have to make service changes to that facade in order to make it work with 9->10 (current lenient facade only supports the 8->9 pattern of missing core_version_requirements)
• The power to move forward would be entirely in the site owners hands, and not dependent upon whether or not maintainer X of project Y has adopted a new heuristic or actually maintained their project at all.
• It keeps a list, in the composer.json of a site, of modules that are deliberately overridden in this manner.

It *does* still require that end users patch their modules to be compatible if the module's core_version_requirement is still incompatible. For composer users this is the typical pattern.

For tarball users, well, they will still be able to download a project and patch it, and continue to be encouraged to stop being tarball users. ¯\_(ツ)_/¯

+1 from me.

It solves the composer-specific problem and only that. It's a deliberate decision for a specific module and still requires you to patch it, preventing inexperienced users from downloading incompatible modules and possibly breaking their sites.

Making this a composer plugin that empowers end-users without weakening the reason the constraint exists sounds like a great solution! 👏

+1 from me too (Yay consensus!)

My top concern is the trouble both end users (site maintainers) and developers (module/theme maintainers) have when updating modules to be d10 compatible. Especially when we go down the dependency tree, the d8 to d9 upgrade was very painful in some cases.

This is entirely handled by creating a composer plugin. Hopefully we can iterate on that asap.

From #17:

I haven't' had a project that I have tested this workflow on D.O. yet, however I wonder if issue forks are a method to solve this? They can be opened by the community, can be installed using composer (they are just a git repo) and keep the stable branch 'clean' until the merge is ready.

Yes, that works. I wrote a blog post about using the issue fork this way. I cannot find it now, but @damienmckenna wrote a similar post before I wrote mine. (One advantage of Damien's post is that it recommends using the only key, or maybe the exclude key.)

So that confirms one more way to handle the problem that this issue wants to solve. FWIW, I agree with the arguments against using >= 9, so my vote is -1 for the proposed resolution.

Yes, that works. I wrote a blog post about using the issue fork this way. I cannot find it now, but @damienmckenna wrote a similar post before I wrote mine. (One advantage of Damien's post is that it recommends using the only key, or maybe the exclude key.)

Note: This has exactly the same security risks as using a patch directly from a merge request. Anyone can gain push access to a merge request and directly change the project you are building. You could combine it with an explicit commit hash I guess, but not even that is a 100% guarantee. I don't think we should promote that.

So that confirms one more way to handle the problem that this issue wants to solve. FWIW, I agree with the arguments against using >= 9, so my vote is -1 for the proposed resolution.

I'm confused by this. >=9 is no longer the proposed solution, #36 is, and everyone seems to be happy with that so far?

Here we go! https://github.com/mglaman/composer-drupal-lenient

    "require": {
        "composer/installers": "^1.2",
        "drupal/better_exposed_filters": "^5.0",
        "drupal/cdn": "3.x-dev",
        "drupal/charts": "^5.0",
        "drupal/core-composer-scaffold": "^10@alpha",
        "drupal/core-project-message": "^10@alpha",
        "drupal/core-recommended": "^10@alpha",
        "drupal/devel": "^4.0",
        "drupal/facets": "^1.8",
        "drupal/sqlsrv": "^4.3",
        "drupal/token": "^1.8",
        "drupal/tvi": "^1.0@RC",
        "drupal/upgrade_status": "^3.10",
        "drupal/webform": "^6",
        "drush/drush": "^11",
        "mglaman/composer-drupal-lenient": "*",
        "nikic/php-parser": "^4.11"
    },

And like a charm, I've got Drupal 10 and random D9 contrib.

➜  contrib-dev composer show drupal/core | grep versions
versions : * 10.0.0-alpha1

and then

➜  contrib-dev composer show drupal/token | grep versions
versions : * 1.10.0

EDIT: Works for new project as documented. Having issues updating existing project.
Thank you very much @mglaman, going to try it.

This works for the first step as mentioned by @mixologic: Introducing the code using Composer to your project. See key takeaway 1 in #36. The second step, installing module for your sites, still requires action. Drupal doesn't allow installation. You need to patch or use the module to ignore core_version_requirement.

Module drupal/token 1.10 is not ready for D10:

docker@d92:/var/www/web/modules/contrib/token$ cat token.info.yml
type: module
name: Token
description: Provides a user interface for the Token API and some missing core tokens.
core_version_requirement: ^8.8 || ^9

# Information added by Drupal.org packaging script on 2021-12-04
version: '8.x-1.10'
project: 'token'
datestamp: 1638619778
docker@d92:/var/www/web/modules/contrib/token$

@mglaman: Nice. I think that does it for every requirement you have now, correct? My understanding of the proposal is that you would opt-in to this behavior per module, so you would need to keep a list of those somewhere in the settings. So that you only get that for modules that you explicitly deal with, coming to @mmjvb...

@mmjvb: Yes, that is on purpose. Now you have to use a patch to make token.module actually compatible. But you *can* use a patch now, and without this, you can't do that as composer will not allow you to download the incompatible module in the first place.

@mmjvb: yes, the existing lenient composer facade also does not change anything about the code and this plugin is meant to replace that with a future-proof solution, since the lenient facade does not work from 9 to 10 due to how it is architected.

Indeed, works as designed, design sucks!

Just provided the information as it wasn't clear to me from #44 what it was about. Only when diving in, it made sense to me, hence the explanation.

Making modules compatible with new major is something maintainers should do. Site builders should just check outdated and update accordingly. Provided of course there is a need to make it compatible. They are likely already compatible, just lacking proper maintenance. Which this plugin is trying to resolve. With drupal/backwards_compatibility installing in Drupal (second step) is no longer the problem. Obviously, updating to compatible version is preferred.

Just as with the lenient endpoint/backwards_compatibily, ignoring requirements sucks. It should extend requirement on drupal/core for major version when not mentioned in current requirement. That means the meta information for both Composer and Drupal need to do what maintainers fail to do. Currently, the meta is adjusted just in time to resolve as expected. The meta of the repository should be adjusted for Composer to have a consistent view of the meta. The core_version_requirement adjustment should also allow installation in Drupal site. Piece of cake to do that manually for site builders or continue the use of drupal/backwards_compatibility. Very annoying though.

On the new project with the lenient plugin:

docker@d92:/var/www/d10-3$ composer show drupal/token
name     : drupal/token
descrip. : Provides a user interface for the Token API, some missing core tokens.
keywords :
versions : * 1.10.0
type     : drupal-module
license  : GNU General Public License v2.0 or later (GPL-2.0-or-later) (OSI approved) https://spdx.org/licenses/GPL-2.0-or-later.html#licenseText
homepage : https://www.drupal.org/project/token
source   : [git] https://git.drupalcode.org/project/token.git 8.x-1.10
dist     : [zip] https://ftp.drupal.org/files/projects/token-8.x-1.10.zip 8.x-1.10
path     : /var/www/d10-3/web/modules/contrib/token
names    : drupal/token

support
source : https://git.drupalcode.org/project/token

requires
drupal/core *
docker@d92:/var/www/d10-3$ composer show drupal/token -a
name     : drupal/token
descrip. : Provides a user interface for the Token API, some missing core tokens.
keywords :
versions : 1.x-dev, * 1.10.0, 1.9.0, 1.8.0, 1.7.0, 1.6.0, 1.5.0, 1.4.0, 1.3.0, 1.2.0, 1.1.0, 1.0.0, 1.0.0-rc1, 1.0.0-beta2, 1.0.0-beta1, 1.0.0-alpha2, 1.0.0-alpha1, dev-1.x
type     : drupal-module
license  : GNU General Public License v2.0 or later (GPL-2.0-or-later) (OSI approved) https://spdx.org/licenses/GPL-2.0-or-later.html#licenseText
homepage : https://www.drupal.org/project/token
source   : [git] https://git.drupalcode.org/project/token.git 8.x-1.10
dist     : [zip] https://ftp.drupal.org/files/projects/token-8.x-1.10.zip 8.x-1.10
path     : /var/www/d10-3/web/modules/contrib/token
names    : drupal/token

support
source : https://git.drupalcode.org/project/token

requires
drupal/core ^8.8 || ^9

Expect to see drupal/core ^8.8 || ^9 on show drupal/token, resolving on D10 is due to plugin. Maybe expressing (lenient) here. Possibly drupal/core ^8.8 || ^9 || ^11 when D11 comes out.

@mmjvb:

That means the meta information for both Composer and Drupal need to do what maintainers fail to do. Currently, the meta is adjusted just in time to resolve as expected. The meta of the repository should be adjusted for Composer to have a consistent view of the meta. The core_version_requirement adjustment should also allow installation in Drupal site. Piece of cake to do that manually for site builders or continue the use of drupal/backwards_compatibility.

We don't know automatically if a project needs only the info file changed or various other changes made, so after the composer require step is a good point to stop and figure that out manually. (Also we could not fix those incompatibilities automatically in this scenario).

As for maintainers should do this. Well, yes. But as per https://dri.es/who-sponsors-drupal-development-2020, the 12 month period before Drupal 9 was released,

the Drupal community worked on 4,195 different Drupal.org projects

Compare this to the number of Drupal 8 compatible projects on Drupal 9's release date which was more than double that number. So half of the Drupal 8 compatible projects received not even a single commit in the 12 months prior to Drupal 9's release, meaning that you cannot reasonably expect the existing maintainer would make a release. That is where the whole requirement for this command roots in the first place.

@Berdir see readme, not implemented. Tossed together quick to prove it's possible

correct? My understanding of the proposal is that you would opt-in to this behavior per module, so you would need to keep a list of those somewhere in the settings.

You'll specify an allow-list in Composer manifest extras

We don't know automatically if a project needs only the info file changed or various other changes made, so after the composer require step is a good point to stop and figure that out manually. (Also we could not fix those incompatibilities automatically in this scenario).

Actually, we do know: https://www.drupal.org/project/guestbooks/issues/3147730#comment-13671180. And raise an issue for it because current policy refuses to implement it. Yet again, putting work on those that don't do it.

meaning that you cannot reasonably expect the existing maintainer would make a release

Sounds like a perfectly reasonable request to the existing maintainer. The current proposed solution (lenient plugin) deals with the situation where maintainers don't respond to the requests.
In addition to nobody taking over these projects and not having a group of maintainers doing such work for unmaintained projects. It is not that there are no volunteers prepared to do it, it is current policy that blocks it. But +10000000 for lenient plugin whitelisting modules and adjusting meta. Both Drupal and Composer meta.

It's more than just policy, its *culture*, and the definition of maintainership, and where that overlaps with ownership, and what it even means to contribute an extension to the drupal community.

Fixing/changing that is a very, very large undertaking, and unlikely to find a real consensus for the forseeable future.

A perfectly reasonable request, it is still a request that the maintainer spend a nonzero amount of time paying attention to something that they no longer care about maintaining. But we don't have a great way of figuring out who does, or does not care about maintaining their projects, but we *do* have a policy that allows somebody else to take over ownership.

Problem is we *do not* have volunteers prepared to take over ownership of various projects, we only have volunteers prepared to do a minimal amount of effort to make a project work with a more recent version, so that they can upgrade it on their own sites and move on. Theres probably a significant number of projects out there with patches like you listed for which there is no volunteer because nobody is using that project and isn't incentivized to try and upgrade it.

Anyhow, there is no engineering solution to the people problem of the varied degrees of maintainer commitment. Best we can do is allow the end users a way to work around needing a project that lacks maintainer commitment.

Moving from Slack discussion into this thread:

At Open Social we've internally banned use of >= because we ran into the exact problem of "Oh there's actually a breaking bug here, but now composer is happily using an older version which claims to be compatible but is not".

Providing people with a way to say "I know what it says on the tin, but I know better, listen to me" is the way to go IMO (as long as that doesn't become the "I think this always works :pray:" default).

Addition after reading some of the messages here and briefly looking at the created plugin:

If our developers start using composer require-lenient by default (because it has lower friction), or if the plugin just makes composer require ignore Drupal core versions (Sorry Matt! It's not clear to me if that's what now happens or whether conscious action is required besides installing the plugin), we'll probably have to ban the tool for the same reason. The composer version constraints are important and should not be disabled by default (be that using >= or through a plugin).

One mitigation for this would be that the composer-drupal-lenient plugin updates the extra config in the root composer.json to have a list of overridden packages. This provides an important aspect of auditability (e.g. in PRs) (and forces this conscious decision to be made on a per-package basis).

@mixologic Agree with everything you said in #53.

With the changed development strategy, semantic versioning, the chance a module just works is much higher than before. Maybe the security group of volunteers are prepared to take on this responsibility. Not taking over ownership and proper maintenance, just release an alpha for the next major of Drupal. Possibly releasing stable bug fix after a period of no related bug reports. Rector would be able to do that, but there is stil resistance to go further than providing patches.

This lenient plugin is removing the unnecessary roadblocks for site builders to keep current on Drupal. These roadblocks are from the past, based on different strategy, no longer valid. Obviously, alternative solutions need to be found for unmaintained modules. Not something you want for your site. This plugin allows you to do that on your own time schedule.

As part of the effort to update the project update bot that generates patches for the contrib module to make them compatible with drupal 10, another possible path forward here emerged.

If we focus on #36.1 for a moment:

There are *two* constraints that are used in different contexts, both of them derived from core_version_requirement. The first is the build time dependency constraint on composer, which translates the core_version_requirement to drupal/core, and prevents users from *downloading* a non-compliant project. The second constraint is the *runtime* dependency that drupal itself implements that prevents *installing* a module that doesn’t match the criteria.

There are really two problems we’re trying to solve and they each have different risk profiles.

1. Can we *get* the code we want.
2. Can we *use* the code we get.

The risk of #1 is limited, in reality there probably is not any consequences if you are on a drupal 10 site, and composer require drupal/some-d9-project-with-deprecations. The risk associated with composer require drupal/next-major-version is also known. If you are doing a major version upgrade, all of the instructions should likely point you to “ensuring your modules are updated to work with d10” and that the whole process of getting your modules to work with d10 (either by upgrading them, or patching them) should be well documented.

The risk of #2 is really where the main concern is. I think we might have been thinking about downloading *and* installing as one thing, but really, the `core_version_requirement` in the info.yml files is the place where we mitigate the risk of actually breaking/whitescreening a site.

Perhaps we can relax #1 without relaxing #2.

1. packages.drupal.org would serve open ended constraints for projects drupal/core dependency.
2. If a project has a constraint on drupal/core in it’s composer.json, override the open ended constraints for releases of that project, and use the closed constraints instead. This gives maintainers a way to opt-out of having to support open ended constraints. This would also apply to all releases of that project as we wouldnt want older relases to satisfy requirements.
3. Leave core_version_requirement as is in the info.yml file. This allows drupal to gatekeep installing an incompatible modules that might whitescreen somebodies site.

So, this proposal could:

1. Eliminate the problem we faced before which is “composer is preventing site owners from taking the steps necessary to upgrade”.
2. Still keep a safety net that prevents breaking a site with a module that *isnt* compatible. For the project browser example, it could still download, but then drupal can say its not ‘compatible’, without putting people into a bad space.
3. Wouldn’t require any action at all from site owners at the composer level - no plugin, no lenient facade. No extra steps beyond adding whatever patches are necessary to their composer.patches extra section.

Downsides:

1. The modules that only need an info.yml patch will still be required to be patched to declare compatibility. Given that sites more than likely use. a combination of these modules in addition to modules that already required rector patches to run, I dont see this adding much addition burden to site owners during major upgrades.
2. Some weird scenarios possible where somebody adds a d9 only module to their d10 site, then the maintainer adds constraints to their composer.json that deny install on d10, effectinvely removing it from the site. Very edge casey.
3. This might hamper efforts to using composer.json exclusively for modules, but that might not be something we can actually achieve given the build-time/run-time split characteristics of drupal.

Let's say you have 1.0 supporting d9 and 2.0 supporting drupal 10. With normal dependency management this would mean when you require the module you'd get 1.0 if you are running d9 and 2.0 if you are running d10. If we open up composer this would mean it cannot resolve the dependency normally anymore.

Going this route means you pretty much downgrade composer to a download tool.

You can still update inside minors, but getting the right version then means you MUST be specific when adding the module instead of letting composer figure out the version. This will mean so much overhead while developing drupal AND will mean drupals composer implementation is degraded to being a unicorn.

I feel that is a step back in the quest to get with the program and work like the whole php world works.

I think that could work, but personally need to think more about it. Can we expose an extra value with the "real" info yaml contents though? That could help tell the two apart without getting the package itself and trying it. (Also eventually expose it for project browser I think?). That way there is still the option for optionally considering that in a client.

I'm not sure if this would be a solution. People will need time to upgrade, so this would be a way that people would be forced to upgrade at some point or are no longer to use composer to update dependencies.

The fact is you are then dynamically changing the requirements. I really wonder what would happen then to existing lockfiles. I mean, when the requirements of the package have changed and you run composer update, even if this is targeted to some other package, would it reevaluate the whole chain? If so, when Drupal decides that for those old modules the new version of Drupal is supported only, this would mean you are stuck completely.

The fact is, its quite outside the scope of composer. In the end the version of a package is no more than a commit hash. So technically the Drupal endpoint could change the requirements, but this will be messy, especially around the time of the change. You might have local cache, you might have run CI tests locally and are publishing between the timeframe.

To me it feels like the best solution is something explicit. Make sure that if you want to install something that is old and no longer supported you make that choice manually.

In composer.json this could be the solution proposed with the lenient plugin. But explicit, so only allow packages marked as lenient to install, this would be great. Then there still is the problem of actually installing the package in Drupal, but when you have an installed codebase, you could just use the patches provided by the upgrade bot since composer installs the package then.

We need to stay clear of trying to change the default way of serving packages as done normally with composer. There are good reasons package versions are set in stone, this means you as a developer know that the result of downloading a package is the same every time, and things are stable, even if you download it later down the road. If we need something special for Drupal, to help modules migrate to new versions, keep it separate of the default workflow, make it something in extra's like the lenient plugin provides.

A conscious and explicit decision is the only way to go here in my humble opinion.

I might've misread something there regarding removing the support of the previous Drupal version. Lets say d11 comes out, the modules last version is d8, this might mean d9 is now no longer supported. The reality is, not everyone is gonna upgrade, but perhaps would like to receive updates after that. So they would be forced to upgrade. Reading through your post again, this seems like a mis assessment of the proposed solution.

Technically this could work if we try and container this to the last supported version. That would at least give a way out. But I'm quite afraid of the implications of changing those requirements dynamically in regards to composer dependency tree handling and their cache and such.

Tried finding out how (and if) composer does any checks while installing other than platform checks. If that is not the case there is a chance this is in the clear. But unfortunately, my 15 minutes didn't result in much.

I still stand with what I said though, that making the site owner be explicit about it is a better way to handle it.

The risk of #2 is really where the main concern is. I think we might have been thinking about downloading *and* installing as one thing, but really, the `core_version_requirement` in the info.yml files is the place where we mitigate the risk of actually breaking/whitescreening a site.

Isn't the info.yml only checked on module install? If the code gets upgraded/downgraded via composer I believe the module is still enabled and loaded without the safety check. I believe that nullifies Comment #57 safety positive #2 "Still keep a safety net that prevents breaking a site..."

A couple scenarios I can think of off the top of my head are:

• The module constraint no longer prevents a core upgrade (which may happen for other reasons of a not tight pinned core, or upgrading core thinking its safe)
• Contrib ModuleB has a hard limit on Contrib MoudleA that is already installed, However ModuleB requires an older version with a conflict on newer minors which forces ModuleA to downgrade to a no longer compatible with Drupal Core version, the facade allows the install to occur

Are there checks in here that I'm missing that would prevent those two scenarios?

Personally I'm in support of the plugin still, I would prefer its an explicit choice to override.

Let's say you have 1.0 supporting d9 and 2.0 supporting drupal 10

If the maintainer has a project like that, *and they express that in their composer.json for their project*, then everything would work the way it does now. We take maintainer intent, and advertise it at the facade.

If a maintainer has a 1.0.x branch, and it only has dependencies in info.yml, and its current core_version_requirement says '^8 | ^9', then that is the kind of project where we wouldnt advertise a requirement on core at all, or advertise a dependency on the minimum version).

Now, if that same maintainer adds a 1.1 release that adds a composer.json that says it require's drupal/core '^8 | ^9' then we would alter the composer metadata that the facade advertises to use the composer.json and the core version requirements for all the releases in that project.

Essentially the key indicator that we are observing is 'whether or not a maintainer has been explicit about a build time dependency requirement' - if so, trust the maintainer. If not, allow for more flexible installation.

So, one edge case weird scenario could end up like this:
1. site owner upgrades core to d10.
2. site uses a d9_example_module but it works with d10 fine.
3. maintainer of d9_example_module adds a composer.json that says it is d9 only
4. composer would remove d9_example_module from the users d10 site.

#3 is a very contrived scenario but Is *a* way this breaks.

Re #65:
There are two types of upgrades: upgrading *core*, in which case yes, we expect people to have projects that do not work in that scenario, but that is by design, so that people are able to patch them.

The second type is upgrading a project. Now, in order for this scenario to break, the project you are upgrading would have to *introduce* new usages of deprected functions from one release to the next, and also not explicitly declare itself to be compatible with d9/D10.

Im not sure I understand the second scenario described in #65. define "hard limit" -> requires an older version.. of Module A? of Core?

I haven't been following this closely, but I just published 1.0.0 of https://packagist.org/packages/mglaman/composer-drupal-lenient, so I can start using it in D10 readiness work.

I have struggled for weeks now upgrading sites to D10. A fresh install worked on one site, but generally I want to upgrade. All my modules are either upgraded or patched (using cweagans/composer-patches). But this still does not work. So I tried composer-lenient, and the laminas package installed by upgrade_status still stops the upgrade. It should not be so difficult. Users should not have to munge around in composer.json.

Drupal 9.5.3 requires symfony/psr-http-message-bridge, but the version used does not support php 8.2

A fundamental flaw in Drupal is that a small change can give a php error that kills the site, and sometimes even a db restore does not work.

Module patches sit unreleased for months. Drupal.org needs to push developers to at lease release dev versions immediately that a patch works.

It looks to me like there's a majority for relaxing the recommended core_version_requirement, which I agree with. (I recently created and closed the duplicate issue #3357582: Always allow downloading and installing contrib module dev-release in next major version).

I really like @japerry's method outlined in Navigating core_version_requirement, and think it would sense to make this the recommended model:

What should core_version_requirement look like?

My personal preference is to work with the next ‘beta’ of a major version. That allows developers to immediately work with a module, but doesn’t allow it to automatically work with the next stable release:

If your module supports Drupal 8, 9, and 10:
core_version_requirement: >=8.9 <11.0.0-stable

If your module supports Drupal 9 and 10:
core_version_requirement: >=9.2 <11.0.0-stable

This could reduce future contrib module upgrade bottle necks, and allow contrib modules to be easily installed and worked on in the next major release while it's underway, but not yet released.

In the issue summary's "Proposed resolution" section, it is still proposed that an open constraint (>=) is used as the core_version_requirement. IMHO, this is a bad idea, for all the reasons pointed out by Kingdutch, cmlara, Berdir and Pasqualle. I think this link: Why are unbound version constraints a bad idea? bears repeating.

However, when reading the discussion, my interpretation is that consensus gravitates towards adding the Lenient Composer Plugin as the preferred solution.

Perhaps the "Proposed resolution" section of the issue summary needs updating?

Installing and configuring a Composer plugin is easy for developers (the majority of users in this issue), not so much for the Drupal beginner. As @xjm puts it, and I agree with:

Using >= will put the power in the hands of site maintainers if they want it. Might the module still be broken? Sure, but then they're no worse off than they were. And there's a 50% chance it will just work (see Gábor's data above about D9 projects), and tools like Upgrade Status can warn the site owners if there are problems.

It's the right choice for the sustainability of contrib, for the major release cycle going forward, and for making upgrades easier for site owners, which is our top priority for Drupal.

I think we need to discuss the option of using a "Range Constraint", with a minimum and maximum Drupal core, for example:

core_version_requirement: >=9.2 <11.0.0-stable

I agree that installing and configuring a Composer plugin may not be easy for inexperienced developers. That's just the way it is. If they're that inexperienced, maybe they're better off not using unstable versions?

You quote @xjm's argument from comment #24, that says:

Using >= will put the power in the hands of site maintainers if they want it.

Unfortunately, if an unbounded version constraint is used, the site maintainer will get this "power" whether they want it or not. The version constraint is not something the site maintainer can control, it is put into the project's .info.yml by the project maintainer.

And IMHO, the "power" they get is akin to a shotgun aimed at their own foot. Yes, it gives them the "power" of being able to easily install the dev-version of an unmaintained project on their website, but if it blows up due to unhandled deprecations, they're stuck with a WSOD. And if they are inexperienced, maybe they shall not be able to recover from that.

To me, the downside of increasing the risk of the site breaking outweighs the upside of letting site maintainers easily install an unmaintained project that just might work anyway.

That's a great suggestion @catch. It would definitely be a step in the right direction, towards making it easy to download a contrib modules in the next major release. The user still needs to define the allowed modules, and also patch the *.info.yml file, and add || ^11 though ...

Ideally, to lower the bar of entry for new Drupal users, and just overall make it easier to run modules in the next major release, if the Lenient plugin was added in Drupal Core, it would be fantastic if a new Leniency toggle feature was added, allowing both the download of any module with Composer, but also a subsequent installation in Drupal 11, by running this command:

composer config extra.drupal-lenient.allow-all true

After running this command, a module with core_version_requirement: ^10 should be able to be downloaded, as well as installed in Drupal 11.

It could also be split into one setting for downloading via Composer, and one for installing in Drupal:

composer config extra.drupal-lenient.allow-download-all true
composer config extra.drupal-lenient.allow-install-all true

It sounds like the core_version_requirement in the module should just be deprecated in favor of composer.json constraints, period.

Thanks for sharing your opinion @mglaman, I added it as an option in the Issue Summary, and in the "Persona | Site owner | Maintainer" table together with "Open Constraint >= 9", since those two options share a lot of traits.

Should the title be updated to something like "Make major version transitions easier"?

#3363604: Add Drupal 11 support :)

A small update in the CSS and adding || 11 under the core_version_requirement's, and Admin Toolbar works well in Drupal 11.

Since people are linking to this issue as justification to put open ended version constraints like ">=9.0" in their contrib module composer.json files, I think we should better explain the dangers of doing this in the issue summary.

Folks here might already know this, but I only recently learned (via a Bun.js release announcement) that npm supports an overrides field and yarn supports a resolutions field that let you achieve what the composer-require-lenient (issue title refers to it with this name, but are we referring to https://github.com/mglaman/composer-drupal-lenient?) plugin lets you achieve. I think this adds to the argument that Composer itself should add similar functionality upstream, but if they don't want to, I'd support #74's suggestion of adding it to Drupal core.

Since Drupal Lenient Composer Plugin has replaced the lenient packages endpoint, maybe we could soon consider including the Lenient Composer Plugin in Drupal core, as @catch suggested in #74?

I think this issue is the most important thing that we could do to make Drupal more competitive with WordPress. WP requires much less effort to maintain long-term. Even with the relatively smooth transition that Drupal has with D10 to D11, WP is still much less. WP's approach of "backwards compatibility forever" has not caught up with it yet (and maybe never will).

While we've done so much to make Drupal major upgrades smooth, I think we still need to do more to remain competitive.