Concerns about beta and RC contrib without security coverage

Moving to a more correct queue.

This also touches on (or is more in the domain of) security team policy and d.o policy, but the TWG is probably a reasonable entity based on my reading of the governance. But definitely not CWG. :)

The best thing you can do is probably contact the maintainers of these modules and suggest they look at creating full releases and opting into security coverage (especially if you offer to help out where possible).

Retitling to a title that is more neutral and also more specific.

The TWG had a spirited discussion about this issue today and we may need to have another meeting or two before we can post a thoughtful response. We will update this issue as we progress.

The TWG discussed this issue on several of our recent internal calls. First of all, we completely agree that it's a problem for it to be such common practice for production sites to deploy with modules whose security vulnerabilities can be discussed in public prior to getting fixed, as is the case with alpha/beta/rc releases. So thank you for opening this issue in which to have this discussion.

force alpha / beta / rc releases into the security cycle if they meet a usage threshold, say 5%

This option had some initial appeal in our discussion, except replacing the word "force" with a voluntary opt-in system, where the contrib maintainer (with security team approval) could choose to enter a pre-full-release project into the private security issue tracker.

However, upon further discussion of that idea, we were concerned that this doesn't address the deeper problem of a communication mismatch between project maintainers and site owners. When a module maintainer has a module in alpha/beta/rc, what they're saying is "this module is not yet suitable for production usage". When 7%, 19%, or 50% (based on the numbers in this issue's summary) of Drupal sites use the module in production anyway, that's a sign that a large number of site owners are disagreeing with the maintainer's assessment. Ideally, when such a situation occurs, that disparity between the maintainer's assessment and the site owners' assessments should get resolved within a reasonable timeframe. If the maintainer were to opt-in to placing a module that they do not yet believe to be suitable for production usage into the private security issue tracker, it seems to us that that would further confuse the message rather than clarify it.

Therefore, our recommendation is for maintainers of pre-release modules with a large number of users to do one of two things: either tag a full release, or communicate clearly on their project page what's holding up a full release. Here are some common things that hold up a full release, and how we propose handling them:

• The maintainer still wants to improve the APIs: If there's a really bad problem with an API, we suggest linking to an issue that explains the problem and gives the community an opportunity to comment on their thoughts about that problem and potentially help with getting that API ready. If the problem isn't too bad, we suggest cutting a full release anyway. The APIs will never be perfect, and we think it's better to tag a release, and then branch the next major version (e.g., admin_menu 7.x-4.x), and continue improving there. We'd love to see #1612910: [policy, no patch] Switch to Semantic Versioning for Drupal contrib extensions (modules, themes, etc) get resolved, so that potentially this would only require branching the next minor version, instead of major version, but until that's resolved, we think it's fine to increment major version numbers for this purpose.
• There remains a Critical (e.g., data integrity) issue that's only triggered in edge case situations: This situation could easily explain why a large percentage of sites are happily using the module while the maintainer is still reluctant to consider the module stable. We suggest that in such situations, if it doesn't look like a fix to this will happen anytime soon, that the maintainer release the module anyway, and have prominent release notes that warn about the edge case.
• The module is still broken in some really bad way, even under common, non-edge-case usage: This seems unlikely to be the case if the module is being actively adopted by tens or hundreds of thousands of sites, but if the maintainer believes this to be the case, then we suggest writing up some clear communication about this on the project page.
• The module is employing an ugly workaround to a core (or other upstream) bug and doesn't want to create a full release until the upstream bug is fixed and the workaround can be removed: If no upstream fix is in sight within a reasonable timeframe, we suggest creating a full release anyway, and removing the workaround in the next major (or point release) version after the upstream bug is fixed.

These are our initial thoughts. We'd love to hear feedback about this from people following this issue, and especially from module maintainers. Thank you!

At least one proposal here I have a pretty strong disagreement with, let's juxtapose these two paragraphs:

The TWG discussed this issue on several of our recent internal calls. First of all, we completely agree that it's a problem for it to be such common practice for production sites to deploy with modules whose security vulnerabilities can be discussed in public prior to getting fixed, as is the case with alpha/beta/rc releases.

There remains a Critical (e.g., data integrity) issue that's only triggered in edge case situations: This situation could easily explain why a large percentage of sites are happily using the module while the maintainer is still reluctant to consider the module stable. We suggest that in such situations, if it doesn't look like a fix to this will happen anytime soon, that the maintainer release the module anyway, and have prominent release notes that warn about the edge case.

For an example, here's a module of mine that has nearly 15,000 users and is still in beta: https://www.drupal.org/project/drafty

Drafty has zero UI or configuration, at all, not even a drupal_set_message(), no routes, no access handling. Doesn't make it immune from security issues but it's in a low risk category for the usual XSS/CSRF stuff.

Meanwhile workbench moderation had multiple stable releases with a very severe data integrity issue/race condition, currently at around 30,000 users. This resulted in a security release https://www.drupal.org/node/2824455 - the fix for the security release was refactoring the module to be based on Drafty (since Drafty was specifically written to avoid the data integrity issue/race conditions that led to the information disclosure in workbench moderation, see #2376839: Rewrite to use the new Drafty module). That issue was fixed in 2016, one of the data integrity issues it fixed was reported in 2012.

In my opinion, workbench_moderation should not have had a stable release in the first place with that issue outstanding. The policy proposed here would suggest that even if that race condition and data integrity issue had been identified during alpha/beta then it should have moved to stable.

Drafty doesn't currently have a known data loss issue, but it creates a lot of revisions which #2730265: Try to avoid saving a new revision when restoring the published revision and #2579205: Delete redundant revisions are trying to ameliorate - both risk re-introducing data loss, so should ideally go in before a release candidate (or be won't fixed if unfixable).

tl;dr - some data integrity issues are more serious than some security issues, so I see no reason to mark a module stable when it's subject to known data loss, just because a security issue might be reported in public.

a voluntary opt-in system, where the contrib maintainer (with security team approval) could choose to enter a pre-full-release project into the private security issue tracker.

The security team essentially does this already, or at least used to do it until recently. People can create private issues on security.drupal.org for any module, and the security team would invite the maintainers and let them decide whether or not they wanted to keep the issue private. If so, the security team wouldn't get directly involved in fixing it, and there wouldn't be a security advisory, but the maintainer could still fix the issue privately and put out a release tagged "Security" which would show up as a security update in Update Status on people's sites. See https://www.drupal.org/drupal-security-team/security-team-procedures/sec...

All of the above is still mostly possible on a technical level, but recent changes on Drupal.org have essentially deprecated it and put out the idea that if it's not a stable release it should be treated as totally unsafe with security issues always discussed publicly:
#2861822: Redirect the Report a security vulnerability on projects that have not opted in
#2861585: Disable option to make security releases for project that have not opted into advisories

There is some value in being honest about the policy, but there is also value in having a "gray area" like existed before. Perhaps a way should be found to reintroduce something like that.

In terms of the real fix/deeper problem (encouraging more maintainers to put out stable releases that are fully supported by the security team) I think it would be difficult to achieve unless Drupal develops more of a market for ongoing commercial support. If you post a module on drupal.org that was written for a particular client, it's reasonable to start it off as an alpha or beta, because no one else has looked at it or used it yet. But there is not much incentive for you or anyone else to come back later and do the work to push out a stable release. So there is a tendency for the module to never get a stable release at all. Sometimes it does happen, but many people are not going to jump in and volunteer for that kind of responsibility.

Is it legit to have (let's say) 7.x-1.0-beta1 and 7.x-1.0 be exactly the same code, with no commits between them? That always seemed wrong to me... but if we can encourage that, maybe it is more likely people in the above scenario will just put out a 1.0 release, if they see that the issue queue isn't full of incredibly serious bug reports - without having to to review patches or bothering to make new changes.

I think the recent change to only showing the most current release isn't helping. Now, not now showing other stable previous releases makes it harder for newer devs and site-builders to track what they should then be using.

Speaking as a longer term site-builder/module-dev: it tool a while to understand the dev release cycle. When actually many fixes are left on dev lines where then only someone tracking issues and actual commits can understand module stability (assuming a larger security issue isn't the direct issue).

Explaining that to our end client is always a difficult thing... certainly should the client/admin my happen across

Another large concern is what we now see with abandoned and un-maintained modules. Where an issue and patch may exist but are never 'closed'. Or even, is committed on dev but broken..

beta/dev/rc tags and Semantic Versioning should be (re)addressed and documented more clearly.

I think the recent change to only showing the most current release isn't helping.

What change are you referring to? As far as I know, the download page was redesigned recently, but I don't think there's been any change regarding which releases are shown there.

it had the ability to show server of the release for a module. ie:

8.x-1.1
8.x-1.0

7.x-2.1
7.x-1.3

now only shows the most recent tagged release.

I'm really not seeing the difference between before the redesign:
https://web.archive.org/web/20161102005623/https://www.drupal.org/projec...
https://web.archive.org/web/20161208035201/https://www.drupal.org/projec...

and after the redesign:
https://www.drupal.org/project/admin_menu
https://www.drupal.org/project/media

Both before and after, it shows the most recent release from each supported branch only, but you can get to the other releases via the "View all releases" link.

RE #14... The project page has always had the ability to show different *branches*, such as 8.x-1.x, 7.x-1.x, 7.x-2.x. But it always only showed the most recent tagged release within that branch, plus the development branch if the module maintainers decided to show it.

The format looks like it has changed in the last week, or maybe last few days. But the information seems to be the same to me?

before

--edit
sorry, i was sure multiple releases where shown...

almost regardless. perhaps not showing 'other stable releases' is a problem.

You mean you think that older releases should be shown, such as 7.x-1.1 if 7.x-1.5 is the current release? I don't think any module maintainer would agree with you. They wouldn't want people using the older versions and potentially filing bug reports about bugs they fixed in the latest stable release.

But maybe you meant something else when you said "perhaps not showing 'other stable releases' is a problem." ?

Note: Even in the new design, there is still a link at the bottom that says "View all releases" where you can see a complete list of the old versions and their release notes.

Thanks for everyone's comments. We discussed them on our last TWG call and are still digesting them. Meanwhile, adding #2907162: Add $setting to specify minimum module stability allowed to be installed as a related issue.

[Edit: Lol. I cross-posted with #19]

Leaving aside the question of established maintainers who will not cut a release with security coverage when nagged (cough), the barrier for new contributors now is that you still need to apply and get reviewed to opt in to security coverage, then somebody with godlike powers needs to approve the application. The security review queue suffers from the same problem as the old project application review queue. RTBCed security opt-in applications:

https://www.drupal.org/project/issues/search/projectapplications?text=&a...