Commit Project Update Bot patches to under-maintained projects for Drupal 9 compatibility

Thanks Damien for creating this.

Determine if this is something the community should do.

This is my +1 vote, but are there any real drawbacks? I think the second task above (i.e. determin which solution) will help inform if there is any reason why not to do this.

See also: the initial conversation thread on Slack

For option #2, there's the existing process for taking over abandoned projects, which has a two week grace period before full ownership can be granted: https://www.drupal.org/node/251466#procedure---own-project

But option #2 is less drastic than that, it's temporary granting co-maintainer access to apply one patch and potentially cut a release.

I'm a big fan of the abandoned modules process. It's a great process for creating new project owners.

It's a lot of effort if you're pushing a 1 line code fix that every module on Drupal.org will need.

One of the reasons people don't apply for ownership over a project is that it implies a commitment to maintain the module for years to come. So, getting folks to agree to do that is a big ask. By comparison a temporary access doesn't grant that kind of expectation.

What if we said that if anyone moved an Upgrade Bot-supplied patch to RTBC the bot could go ahead and merge it in? Those patches were first generated a while ago so any module that is genuinely still maintained will have had plenty of time to test the patches by now

I think it's worth at least calling out that sometimes modules are abandoned for a reason:

1. Sometimes better solutions come along. Do we want to promote incorrect/competing solutions for the sake of achieving 100% module compatibility?
2. Sometimes "good ideas" never come to fruition because of changing priorities. Do we want to keep dragging around incomplete modules for the sake of history?
3. Sometimes modules only solve very narrow use cases, and don't provide a more generic purpose. In those cases if the maintainer isn't updating it, then that use case basically no longer exists, thus that module should just kinda disappear with it.

By autopatching a module that is rightfully obsolete and forcing continued support, when there shouldn't be, it's implicitly endorsing it as a "valid solution" when it's not. Do we as a community want to keep carrying old technical debt around, simply for achieving some arbitrary benchmark of compatibility?

All that to say, I'm not against the idea, but there need to be some sort of qualifications as to when this would apply? For example:

1. If a module is marked as "obsolete" it should be excluded.
2. If it's marked as "unsupported" it should be excluded.
3. If zero sites report using that module, it should be excluded.
4. I'd assume sandbox-only modules are already excluded, but still worth calling out.

There is nothing stopping anyone else from requesting maintainership of these modules, or forking it into a custom module inside their own website, if they want to just maintain it for their use case.

I think it's also a good opportunity to issue a PSA to all module maintainers out there to mark obsolete modules as obsolete. I just did that prior to writing this message. I had a "good idea" that never amounted to anything and at this point I think it's irrelevant.

        {
            "type": "package",
            "package": {
                "name": "drupal/cache_control_override",
                "type": "drupal-module",
                "version": "1.0.0",
                "source": {
                    "type": "git",
                    "url": "https://git.drupalcode.org/project/cache_control_override.git",
                    "reference": "8db91684a427366d8f9c51f60cbac10c2d586d95"
                }
            }
        },

Per https://getcomposer.org/doc/05-repositories.md#packages, composer will use the data in the package repo rather than from the https://packages.drupal.org/8 endpoint when resolving versions. Thanks to @dpi and @larowlan for sharing this idea.

Thanks @DamienMcKenna for creating this issue and for everyone else chiming in. I've updated the issue summary to include two most relevant Slack threads for those who want more context of the discussions.

I opened #3169163: Send email to project owners with projects near Drupal 9 compatibility for a maintainer-friendly solution. Dumped in the list of 1945 projects/versions that only need an info.yml change as of yesterday AND have a bot patch at least in the processing artifacts. @tedbow needs to run the bot part to make sure all of those end up on the right projects. Then @drumm will do the email to all the maintainers of the 1945 projects (one email per maintainer, possibly multiple projects). Feedback is welcome there.

Suggest to create an endpoint on the facade for migrating d8 modules. It should claim them to be compatible with d9. Allowing them to be used on D9 and find out which patches are needed for their circumstances.

Another note about #3169163: Send email to project owners with projects near Drupal 9 compatibility which unless concerns are raised will be sent out to maintainers of 1900+ projects soon. With so many people here, some of you may be interested in taking another look at the email template there :) Thanks!

Another fringe issue we're seeing is that maintainers may have committed the one-liner already to their main development branch (eg 8.x-1.x branch), but never actually enabled the release in Drupal.org UI for the development branch, so people cannot actually do a composer require drupal/modulename:1.x-dev in order to get the Drupal 9 support. In this case, I guess it makes sense to just open an issue to request the maintainer enable a release for the development branch.

Making "auto" commints won't make the projects maintanable. Also I belive adding support for new major Drupal version still requires some manual testing. As of Composer Patches the workaround could be creating a temporary fork of the D8 project in a custom repository.
We can keep all such forks in one place somewhere on GitHub or GitLab to prevent any work duplication.

Actuall there was an attempt to implement this but now it is kind of abandoned.

https://github.com/D9-ports

Making "auto" commints won't make the projects maintanable.

Making the project maintainable wasn't the goal here. People are still free to do work in the issue queues, and submit patches to undermaintained projects. The problem this solves is simply that one cannot even submit patches (easily) due to the packagist.drupal.org blocking composer from letting you even install the project in the first place. That is all we're trying to do here. Forking the modules to github was mentioned in the original slack discussion (see link on issue summary above), but ultimately that is not a good solution because it drives people and processes away from Drupal.org where all the community contrib should take place -- were it not for this one little one-liner issue.

One might even argue that getting the one-liner in on drupal.org would actually end up leading to module being picked up and maintained eventually as opposed to being abandoned or worse someone creating a NEW PROJECT in its place to provide the D9 integration.

This issue is about clearing away debris from a self-inflicted road block.

Did you folks try a combination of composer version aliasing (see https://matthewgrasmick.com/posts/screencast-updating-drupal-9-15-minutes) for getting the project on the site and https://www.drupal.org/project/backward_compatibility to let Drupal 9 install modules that are normally not compatible? The combination of the two should avoid creating a bunch of forks that would need to be kept up to date with the original projects.

Ok composer version aliasing is not the best ideas as it will make composer think you are actually running Drupal 8. What about a combination of pulling in the git repo of a project and applying the Drupal 9 patch to it? Essentially a combination of:

1. "Require Any Git Repository with Composer" section from https://www.daggerhart.com/composer-how-to-use-git-repositories/ and
2. https://github.com/cweagans/composer-patches to apply the patch?

Maybe we should have a way to autogenerate those composer snippets, eg. in upgrade status to help you move forward? Does that sounds workable?

I saw there's a new Merge Request feature being tested on Drupal.org. Could that be utilized somehow?

#3152637: Opt-in to the Drupal.org Issue Forks and Merge Requests beta

Maybe steps like this:

1. Find all of the projects you want to add the one-liner for, then for each:
2. Fork the project git repo
3. Apply the one-liner change
4. Create a merge request
...

@solideogloria as discussed above the fork process is optional and maintainer opt-in. If a maintainer does not commit a one line patch they are not likely to opt into an entirely new way of working on their project. Therefore the suggestion above to combine the git composer method with applying the patch. If issue forks become a general thing, then the patches would be a git repo by themselves, but the special composer git repo method would still be required to use them.

#3168047-9: Commit Project Update Bot patches to under-maintained projects for Drupal 9 compatibility seems like the cleanest workaround here - no need to fork, and not really tricking composer as such - just overriding packagist a bit.

Yeah but #9 still needs https://www.drupal.org/project/backward_compatibility to enable the module if only the info yml file changed or https://github.com/cweagans/composer-patches to patch it then right? I would be happy to put some snippet generator feature into upgrade status or another sensible place to help people do figure out this "best workaround" easier if we can agree on which one is better :D

@catch Sounds like you understand what is being said there. Care to elaborate as it doesn't make sense to me.
What needs to be done for composer and drupal accepting modules that don't state they are ready for the platform you are running?

@mmjvb: this issue is about projects that are mostly Drupal 9 compatible other than their info.yml files or their own composer.json files defining they are. There are two problems with that:

1. You cannot obtain them with composer because composer will consider them incompatible.
2. Even if you could obtain them somehow, Drupal will not let you enable them.

Most of these projects have patches waiting from humans and/or Project Update Bot on drupal.org, but not all maintainers are available to commit them and make releases. The proposal of this issue was to force commit those, but that has various problems, so we went on to discuss how to overcome this locally.

1. The composer problem can be overcome by either forking the project somewhere else and making it a package there (not very future proof and lots of work for a one line problem). Instead @heddn posted a snippet you can copy-paste into your site's composer.json file's requires section. Adjust the name and url properties based on the d.o project name. The reference is a git hash, which is the unique identifier of the commit that is known to be fine. This let's composer pull in the project and understand it as using the standard Drupal package (eg. for other projects that are dependencies).
2. The problem of Drupal not letting you enable it is still present. If it really only lacks the info file change, then you can use https://www.drupal.org/project/backward_compatibility to force that to not matter. You need to know what you are doing here because if the project is genuinely not compatible with Drupal 9, it could cause problems, eg. whitescreen the site. Alternatively (I think) you can also use https://github.com/cweagans/composer-patches to patch the git checkout and fix whatever issues it has.

As more projects get Drupal 9 compatible, you would convert them to the actual versions of the projects that are Drupal 9 compatible instead of using a git checkout workaround and the Drupal install workaround.

Does this help?

@Gábor Hojtsy
Thank you for elaborating. For the record the issue itself was not the problem, although I very much like your description of it. It is why I suggested a d9 endpoint. Just like the current one it would adjust the information in both composer.json and module.info.yml.

Understand the amount of work that needs to be done Just asking the facade to do it for us. Understand also that claiming to be compatible is not being compatible. Obviously, there will be modules that are not 100% compatible. This way is much easier to find out. No longer limited to the handful of people that understand the details. Setting your repostory to d9 instead of d8 currently is something anybody should be able to do. That is not the case for the amount of work you need to do for selective modules.

@mmjvb: I went for an extensive description in hopes others find it useful too :) I think a blanket Drupal 9 compatible endpoint would be tricky because eg. some projects have branches that are Drupal 9 compatible while others are not. Projects may have known problems in their codebase that our detection does not surface and the maintainer is working on fixing it. As with all patches for projects, ideally Drupal users would not resolve it for their own local site only but collaborate with maintainers. They are benefiting from the project being there the first place, so contributing to getting it better would be the ideal outcome instead of an official way to avoid contributing. Sometimes that contribution is saying in the issue it worked, marking it RTBC, etc. sometimes its taking over maintaining a project.

@Gábor Hojtsy thanks! Moreover not every maintainer will be happy if some AI-bot will commit changes without review

I think d-org can enable issue forks for this projects or even re-use previously created issues by info-file-patch-bot

Then everyone can use fork repo/branch to test/approve d9 compatibility

I mean it will allow to everyone to use #9 approach with template (issue forks are predictable names)

@Gábor Hojtsy
Your description is very much appreciated. It makes very clear what the actual issue really is. Just wanted to make clear that it is not me that is not understanding the issue. I did fail to understand the impact of the proposed solution, thank you again for explaining that.

Don't like the proposal because it is far too much / complex work for most people. Having the facade doing very simple things allows a lot more people to participate in the process to get the modules they use ready for d9. When things are made that difficult, they wait for others to solve it.

But, I'll refrain from participating here as there seems to be no interest and I am not sure whether it is according the rules for this project.

@mmjvb:

Don't like the proposal because it is far too much / complex work for most people. Having the facade doing very simple things allows a lot more people to participate in the process to get the modules they use ready for d9. When things are made that difficult, they wait for others to solve it.

I don't think if there is a proxy that makes it look like there is no problem that will make people more likely to seek out if there were problems and go solve them. It would rather encourage to ignore the problems IMHO.

I am generally against forcing modules to be D9 compatible.

There are too many unmaintained modules on d.o. Making them D9 compatible will just make the situation worse. In most cases, using unmaintained code base is worse than using a dev release.

If a module is still missing a one line change, then the module is not maintained. If you want to use such module, the only reasonable way forward is to find someone, who will be a responsible maintainer of that module.

We used to have a blue box pointing to the contrib porting tracker for unported Drupal 7 projects. I think that may be gone at this point (I went to check some projects that are not ported and it does not seem to show up anymore on the project page). We can figure out some heuristic to show a call to action to claim maintainership of projects to expedite the process. For example:

• If a project is not yet Drupal 9 compatbile.
• And there was no commit in the past year (or since the Drupal 9.0 release or some other metric of "unmaintained")
• The project is not marked Obsolete
• The project has a Drupal Project Update bot issue.

=> Show a box with something along the lines of: "A [link1]Drupal 9 compatibility patch exists for this project[/link1], however there were no changes in the past [time]. This project likely needs a new maintainer. [link2]Claim maintainership through the unmaintained projects process[/link2]"

Some D7 projects could be stable, even with no recent commits. Maybe if there are RTBC issues (needing a commit) but no recent commits, that could be a metric?

I would recommend going even further with #32, if possible. Something along the lines of using those metrics to identify unmaintained projects and simply removing the maintainer unless they opt-in to keeping it, rather than making people go through the unmaintained module process.

It's frustrating for somebody that wants to maintain something that they care about to be held up for a couple of weeks by red tape. If there haven't been any commits, issue updates, closed issues, etc. by the maintainer in some amount of time, then why not let somebody else jump in right away?

If a person isn't willing to wait a couple of weeks to take over ownership of a project, they probably aren't willing to maintain it for years to come. The ownership transfer process removes the credit of the original owner as the owner of the module and gives you trust to change the code on thousands of websites without external review. So, it's not a small step.

For trusted folks, it would be ok to have a shorter process, but for a newer community member, having a robust and detailed process helps ensure that new owners are actually working, and not just asking to become the owner without doing anything afterwards.

The other big advantage to the current process is that it forces interested people to wake up the old maintainers which does bring contribution back to the community.

The current process has a big hole in that it requires the security release permission checkbox which is a checkbox new users cannot get which does break the process quite a lot. But, everything else in the process has been super helpful.

#2573661: Show message linking users from project pages to the d8 contrib project. I think this is the issue Gabor mentioned.

If someone wants to get immediate action, they should ask for co-maintainership. Co-maintainer status (for "forgotten" modules) is granted very quickly in most cases, especially when the person has a history on d.o, and the request is delivered to the maintainer.

If we do this, I think we should commit a version constraint like >= 8.8, so that the same modules are 'compatible' with Drupal 10.

It's quite easy to apply a patch dealing with deprecations - same as any other patch, but it's hard doing the composer trickery to get a module installed at all. Additionally the Drupal 9->10 changes are less than 8->9, so the number of modules that stay compatible with Drupal 10 should in general be higher.

Further to #37, we've run into a similar issue with Laminas components recently. While Drupal contrib is different, it's close enough #3180207: Update laminas-diactoros, laminas-escaper and laminas-feed for PHP 8 compatibility. Laminas assumes it will be incompatibile with major PHP version releases, even if it is, preventing any dependency from updating to PHP 8. If instead it assumed it was compatible, then there's a risk of declaring compatibility prematurely, but it enables wider testing as well as potentially requiring zero changes anyway.

With Drupal 6 and Drupal 7, it was possible to run an abandoned module for many years - in Drupal 7's case, theoretically more than a decade. With Drupal 8 and especially 9, that time period is getting shorter. For some modules this is real and they'll actually need compatibility updates, but we know that a significant minority of modules required no changes whatsoever - and the number of required changes is decreasing between 9->10 vs. 8->9. If a module does need changes, then patching for those is easier than tricking packagist.

The downside is the possibility that on a new install or recent update, you run into Drupal 9->10 porting issues that otherwise you'd have been 'protected' from, but these aren't the only bugs that people experience from contrib modules.

If we made a new release automatically, would it use an alpha or beta semantic version if the info yml patch issue isn't RTBC? And will there be an indication on the release that it was automatically generated?

If there were a release, it would definitely be an alpha. And yes, it would be created by project update bot and probably link the policy handbook page in the release notes.

We would not create a release from a non-RTBC patch, I don't think

We agreed in a call with the core committers and DA infrastructure that the next step here is to draft a policy of what would (and wouldn't) be committed, the specific process, etc. Once that is finalized, affected maintainers will be emailed again letting them know this will happen to their projects. It will be possible to opt out just as it was with the issues themselves, and the text will also include a call for (non-robot) maintainers.

Great to see some initiative here. Maybe the newly compatible version should be clearly marked as only having passed the auto-bot tests. This way people aren't left with the impression that a module is tried and true in Drupal 9, just because it's marked as compatible.

This way those are interested in the module would be warned of potential issue, but could still easily install it. This would of course help facilitate the community's ability contribute helpful information about the module's performance in Drupal 9.

Draft handbook page about this plan:
https://docs.google.com/document/d/1OHGrKbgZZ8hLdUk3aHlW0T7qixi9B_BW3r7b...

Thanks for the draft. I think this initiative is a very good idea and it is also explained nicely in the proposed documentation page. Just a few quick thoughts:

Condition 1. No branch of the project is not compatible with Drupal 9.

I think this was meant to be "No branch of the project is compatible with Drupal 9."

Maintainers of affected projects will be emailed at least two weeks before any commits are made by Project Update Bot, so they have the opportunity to either create a Drupal-9-compatible version themselves or to opt out.

Maybe link "to opt out" to an anchor for the last section, where the process of opting out is explained in detail?

Action 6. A release node will be created and associated with the alpha1 tag

There is also a field "Short description" for releases that is displayed alongside the release information on the project page. It might be a good idea to add a suitable short description as well, so that users can see the release was created by update bot right on the project page. This would also work with the existing implementation on drupal.org, so no need for the infrastructure team to implement something special.

Action 6. b) The development version will be marked "Supported" but not "Recommended so that the version is available on the project page.

I think this should refer to the alpha release here, not the development version?

Thanks @FeyP! Incorporated your suggestions.

Regarding:

There is also a field "Short description" for releases that is displayed alongside the release information on the project page. It might be a good idea to add a suitable short description as well, so that users can see the release was created by update bot right on the project page. This would also work with the existing implementation on drupal.org, so no need for the infrastructure team to implement something special.

I mistakenly thought this was a core-only field. As a core release manager, I'm quite familiar with the way the custom download button rendering on the core project node breaks when it is left blank for the latest release! 😂 But didn't realize that it is also displayed somehow-or-other for contrib. Added a proposed short description to the doc.

We had a call with myself, @effulgentsia, @Gábor Hojtsy, @drumm, and @Mixologic and identified that there are two kinds of efforts being discussed together here:

• Making it easier for people to install, test, and patch projects that don't yet have D9 compatibility indicated in their info files.
• Making it easier for site owners to use Project Update Bot versions (that include info file changes but also other rector fixes)

This issue has evolved into a proposal for the second group. @Mixologic has a proposal to address the first group differently, without any new branches or commits to the project, using the composer façade and possibly https://www.drupal.org/project/backward_compatibility. He will post a related issue about that.

The proposal for the first group is to essentially unblock, en masse, all of the stranded d8 projects that composer prevents from being downloaded. This is essentially what @mmjvb suggested in #12.

Details can be found here:
#3227031: Add a major-version-agnostic façade endpoint for stranded d8 projects.

I have reservations about creating actual releases,

1. When it comes to what version numbers we should/can use to label them, we're forced to open a new MAJ version of the project.

Site owners will have to do major version upgrades to get those releases, which requires a composer require step thats different than the usual composer update.
If maintainers ever come back to their modules, they'll have a new major branch that they'll either have to merge into and move forward on, or leapfrog (requiring site owners to again do a major version upgrade with composer require). We'd probably be safe in assuming that most of the maintainers aren't really going to come back anytime soon and do anything with these modules, as they're already sitting on a patch that hasn't been touched in almost a year now.

Creating a new Maj branch has a downstream effect on drupal.org as now we'll probably likely need to add special cases everywhere that users interact with releases.

• Does this new Maj alpha show up on the project's download pages?
• What metadata do we tag this new branch with regarding whether its supported and/or recommended?
• Do we alter git's default branch?
• do we create a new -dev release for the branch in conjuction with the alpha release?
• Do these releases change the metadata we use to determine whether or not to show the project in a list of "drupal 9" modules on search pages?
• If any of these projects have security coverage, do they lose it by upgrading to drupal 9 essentially? (I guess they wont have it regardless once d8 is EOL, but not sure what goes on the project pages regarding the Shield)

Im not very enthused about any solution that requires us to add in a new "category" of release to drupal.org, because that has cascading impacts everywhere we take "normal" release behavior for granted. So if we're restricting it to just updating the short description field, that would be good, but we should treat these releases as 'just as official' as if the maintainer released it themselves, (i.e. it would get advertised in the updates.xml stream etc.)

Im skeptical of the impact that having actual releases is going to have, based on how many projects are going to meet the criteria we're establishing for what projects are *eligible* for a release. Theres a very long tail of seldom used modules that all add up to the reason why so many sites are stuck on d8. We'll either have to loosen our criteria, or come up with some sort of plan to at least check that the site doesnt whitescreen with a patch applied to the module. If we constrain this to only RTBC projects, this will be a lot of work for very little impact, and also a lot more ongoing work to try and solve what is essentially the problem of "The module ecosystem has a long tail of useful but abandoned modules"

Re @catch in #37

If we do this, I think we should commit a version constraint like >= 8.8, so that the same modules are 'compatible' with Drupal 10.

Looking at the draft doc it looks now the idea would be >= 8.9

Here is another idea that could make this process safer so we have less projects marked with >= 8.9 that then when Drupal 10 comes out will be read as compatible but still could be using things deprecated in the 9.x cycle.

We could do another pass of Project Update Bot/Rector for all projects that are in this batch and determine for each project if it is Drupal 10 compatible as of now by determining if they aren't using any functionality deprecated in D9 already.
Then use >= 8.9 && <11 or just >= 8.9 if it is not using new known deprecations and >=8.9 || ^9 if we know they are using things that will removed in D10.

Since there were not as many deprecations in Drupal 9 more projects would get >= 8.9 && <11 or just >= 8.9 but then we use >=8.9 || ^9 when know the other is not true.

Of course there could still be deprecations in Drupal 9 after we do this so it is not perfect solution and we would still have some projects that are marked as >= 8.9 && <11 but aren't actually compatible with Drupal but it should be less projects.

This additional run with Project Update Bot/Rector directly could be done when running the process to make the commits to repos rather than by posting additional patches to the issue.

Does this new Maj alpha show up on the project's download pages?

Yes. This is already documented in the doc.

What metadata do we tag this new branch with regarding whether its supported and/or recommended?

This is already documented in the doc: Supported yes, recommended no. It's an alpha.

Do we alter git's default branch?

No.

do we create a new -dev release for the branch in conjuction with the alpha release?

We have to -- This is how d.o works. Also already documented in the doc.

Do these releases change the metadata we use to determine whether or not to show the project in a list of "drupal 9" modules on search pages?

No.

If any of these projects have security coverage, do they lose it by upgrading to drupal 9 essentially? (I guess they wont have it regardless once d8 is EOL, but not sure what goes on the project pages regarding the Shield)

They would lose it under the same circumstances they currently do: A reported security vulnerability is not fixed by the maintainers in a timely fashion, or the maintainer manually opts out or unsupports the module themselves.

For posterity, marking this as postponed since the Drupal Association is working on #3227031: Add a major-version-agnostic façade endpoint for stranded d8 projects. following our discussion of cost/benefits.

Shall we close this now?

#47 pointed out there are two threads in this issue. The first one is for Drupal 9 and thus outdated. I like to think that that discussion could have lead to the formation of the Project Update Working Group project. The second was was dealt with in another issue, as stated in #48.

https://www.drupal.org/project/puwg was launched in the meantime indeed, so I agree this can be closed as won't fix.