[policy, no patch] Changes to the security team process / tools to reduce workload from increased project applications

Most of this has been discussed, both in #2453587: [policy, no patch] Changes to the project application review process and numerous other issues over the course of the last few years ... I'd suggest that the portions of this which would fall under the TWG mandate are already flagged in the other issue, and otherwise, the security team tools and processes are the mandate of the SWG themselves.

Flagging this issue for the next TWG meeting, and will be recommending it be closed (partially as duplicate, and partially as out of scope).

Well I think this should be discussed somewhere again, even if it's not in this issue or if the project needs to be changed. As pointed out in #2453587-126: [policy, no patch] Changes to the project application review process an SA went out for a module with 2 users. Usually when a module has two users, one of them is the development install of the developer who wrote it, and the other is the staging install of the site they wrote it for.

Time spent doing that directly takes away from dealing with security reports for more highly used modules, or core, or reviews of new projects, or really, anything else.

Several years ago I suggested changing the security team support threshold from 'stable tagged release' to 'stable tagged release and more than 1,000 users', similar proposals surfaced several times in the main project review issue.

There were concerns about how you know whether a module is security team supported or not, especially since usage figures fluctuate, but I think that could be handled by the suggestion on the other issue of having boilerplate on project pages for what level of support the module gets. Potentially it'd need a flag on the project node that gets triggered once usage hits 1,000, since once a module is supported it shouldn't become unsupported again just because usage slips to 999 the next week.

What security support means is:
1. Issues should be reported in private
2. Issues should be fixed in private using s.d.o
3. An SA is released when the fix is released

That's really it. Points #2 and #3 necessarily involve work from a much smaller pool of people than almost any other category of Drupal issue. Recent changes to s.d.o like adding access to project maintainers improve things, but it's still necessarily a lot of manual work by people with elevated permissions.

Another way to reduce workload would be seriously looking at what categories of issues we fix in private: https://securityblog.redhat.com/2015/06/10/the-hidden-costs-of-embargoes/

In my experience, even stable (sometimes especially) modules written by high profile contributors can have serious data loss/race conditions/fatal errors which are much more likely to bring a site down or cause data loss than a good portion of security issues that we publish SAs for.

Allowing less-critical security issues to be fixed in public would likely result in a faster turnaround rate for actually getting them fixed - and by extension allow some of that effort to be spent on the backlog of more-critical ones that are still handled privately.

From a TWG perspective, we consider any security working group policies and processes outside of our particular scope, unless i) an existing policy or lack thereof was causing a specific issue for the Drupal community, ii) it was perceived that the security working group was either refusing to or unable to address the concern on their own accord, and iii) a community member was bringing the lack of action on that specific issue forward as a technical problem with a significant impact to the rest of the Drupal project. (And, in this case, I'd suggest that the issue would probably be immediately escalated to Dries as opposed to being handled by the TWG directly.)

If there are specific process changes which the community wishes to proposed to the Security working group directly, it was suggested that those issues could be opened in the Security Drupal.org module to bring them to their attention.

This answer is not intended to discourage the discussion and research of options or tools as suggested in the issue summary ... but this activity is left as an exercise for the Security Working Group or other interested community members, and will not be undertaken by the TWG directly at this time.

If the research does lead to suggestions for a particular policy change, tool recommendation, or conversation requiring TWG input, we would certainly welcome a followup issue to explore those specific discussions.

Ok, let's do it there.

+1 to stop security support for modules with less than 1000 installs.

I don't support using module installs as a gate, or certainly not with such a high number. I don't have confidence in the usage data at that level.

If we wanted to have a usage threshold I'd state very low - like 10 or 20 to distinguish essentially experimental or site-specific modules from ones that are useful to more people.

Even more important is the gate to going from beta/RC to 1.0

That's really the point we want to focus on reviewing the code rather than during the initial project application.

Has the security team kept stats on project usage for the projects that have had SAs? That might be helpful to see where the "why are we bothering?" threshold is.

+1 for setting a limit on install
+1 for the RC » stable being a gate

I like the limit of installs being a gate, the RC/Stable is a good one as well but it doesn't always work, e.g. the Field Collection module has 140k installs yet avoids security processes by being in beta.

We could have >1k installs and stable or >10k installs regardless of stability. Something more along these lines where eventually we step in once the adoption becomes significant would be a good improvement.

In #7 it is suggested that:

+1 to stop security support for modules with less than 1000 installs

I don't agree with this at all.

As a Drupal site builder, I know that "stable release" means that the project has security support. This is something I rely on when picking projects. Having to check the number of installs (which fluctate from week to week and may not even be reliable) to know whether a project has security support is a step in the wrong direction.

In #11, it is suggested that:

We could have >1k installs and stable or >10k installs regardless of stability.

Having the security team monitor ">10k installs regardless of stability" will increase the workload on the security team, not reduce it (so it is off-topic in this issue thread). I think it is a bad idea, but if you really want to discuss this, I suggest you open up a new issue where it can be on-topic.

This issue has the title "Changes to the security team process / tools to reduce workload from increased project applications".

As a solution, it is proposed (in #5) to make 'stable tagged release and more than 1,000 users' the gate for security support.

This is a solution to a non-problem. The revamped PA process will not allow non-vetted users to create stable tagged releases. To quote from #2035235: Add a permission for creating stable releases, and grant to “git vetted” users:

We do not want these users to be able to create anything more than a -dev release on these projects.

Frankly, I do not see the point of this issue at all. As long as the security team only review stable releases (its current policy), allowing non-vetted users to create full projects but not mark them as stable is not going to increase the workload on the security team. There is really nothing to "fix".

I vote that this issue is closed (works as designed).

@gisle #2035235: Add a permission for creating stable releases, and grant to “git vetted” users and #2666584: [Community Initiative Proposal] Project Applications Process Revamp are, as stated in the issues such as #2453587-198: [policy, no patch] Changes to the project application review process short-medium term proposals to reduce the backlog of project applications. It's unlikely to be the final state.

The 'vet users/projects before a stable release can be made' part of that proposal has been retained in large part to avoid a sudden explosion in unvetted stable project releases, which would very likely increase the number of SAs and security team workload. However as discussed in the issue, there are still drawbacks to vetting at that stage (there could still be a backlog for manual reviews, just at a different stage of project creation, there are still workarounds if you're made a co-maintainer etc.). Also that proposal was made on the basis that the current security team support policy would not change at all, it's impossible to change it further without having that discussion, so please do not try to shut down this issue, since that's where it's happening. Note that most of the people participating in this issue so far are security team members.

Replying to #12, if we switch to a measure other than tagged stable releases for security team involvement, there'd need to be some indication on project pages as to whether security issues in a module should get reported in public or private (which is what 'supported by the security team' means in the first instance) if we change this.

However are you suggesting you don't look at project usage to decide whether to use a project? I'd personally install a module like field_collection with tens of thousands of users and eleven betas with a lot more confidence than one with 100 users and a 1.0 release.

@benjy >10k installs regardless of stability also brings into confusion whether development releases of high usage projects would get SAs - for example an 8.x port of a module might be in alpha with 10 installs, while the 7.x version might be stable with 100k installs. At the moment the 8.x alpha would not get an SA at all, and I don't think we should change that. I think what you mean is >10k installs and never had a stable release, but projects like that are generally an exception.

I understand that idea from the point of view of minimising the number of sites which might get hacked due to a flaw in any particular module, but don't see an easy way to differentiate.

> I think what you mean is >10k installs and never had a stable release, but projects like that are generally an exception.

There are lots of projects like that, sadly. Field Collection is just one of them. I really think that once you've got something like 500+ installs, it doesn't matter whether the developer calls it an alpha or beta -- it's de facto stable because that many people are using it and their sites aren't exploding.

Download numbers have meaning of their own that I don't think we should necessarily tie into the security team process.

For example, the paranoia module has relatively few users, but it is intentional that it has a stable release so that security issues will be handled in a private-coordinated-disclosure manner.

I think it's possible to make a similar statement that the release version/status has a meaning of its own which doesn't necessarily need to be tied together completely with security support. What if we let maintainers select the branches that have security support the same way they can choose which are "supported" at all?

On https://www.drupal.org/node/807766/edit/releases branches that have a tagged release could check a box saying that it uses a private security process or not. Dev releases would never be able to be security supported. See this annotated screenshot:

And then on the download page it could look something like this:

The thumbsup/down is not a final suggestion, just indicative that there would be some kind of an icon that conveys "covered/not". There could be hover text with a quick explanation that "thumbsdown" releases will have security issues reported in the public queue and thumbsup will get to the private queue which means the advisory process and help/support from the security team for the maintainer as they fix the bug.

@greggles isn't that likely to increase the number of projects supported rather than decrease it? Apart from that I like the idea though.

Yeah, I like the concept although i'm not convinced we can rely entirely on opt-in. Interested in hearing others thoughts though.

@catch it might - I'm generally OK with that, actually, if it's a slow move and not a drastic one.

@benjy one note mlhess brought up is that all branches with a stable release would get the thumbs up by default to continue the current process. I think it's reasonable to expect maintainers to edit this area since they are already doing that a bit.

catch wrote in #14:

Replying to #12, if we switch to a measure other than tagged stable releases for security team involvement, there'd need to be some indication on project pages as to whether security issues in a module should get reported in public or private (which is what 'supported by the security team' means in the first instance) if we change this.

In #12, I added #2457643: Only allow releases with security coverage to be recommended as a related issue to indicate how I want to indicate on the project page "whether security issues in a module should get reported in public or private". That change has already been approved by the TWG.

In addition, there is also #786702: Add information about releases covered by security advisories to project pages near download table, which is a broader discussion about visualization of project security coverage.

I.e. how to indicate security coverage on the project page is discussed elsewhere. However, it needs to be coordinated: If the gate for private security coverage is changed as a result of this issue, the visual indicator used must follow suit.

re #12:

I like the limit of installs being a gate, the RC/Stable is a good one as well but it doesn't always work, e.g. the Field Collection module has 140k installs yet avoids security processes by being in beta.

IMHO that is a problem that the RC/Stable limit will fix, not something it should tip-toe around. It will make maintainers more likely to create stable releases. Leaving 140k sites on a beta release is quite frankly irresponsible. And making it as awkward as possible by indicating on the project page that the release is potentially unsecure is a good step in the right direction.

For the same reason, I don't think it would be a good idea to let maintainers decide when the security team gets responsibility for a -dev or -beta release. dev and beta has meanings, they are not random tags for the maintainers to use.

Closing this as outdated.