As a follow-on from #2135189: Proposal to manage the Drupal core release cycle:

One of the main sticking points is the extra burden that this proposal would place on the Drupal security team, who already struggles with wrangling Drupal core security releases given only two active branches.

Quoting from greggles in #2135189-22: Proposal to manage the Drupal core release cycle:

I am -1 as it stands today.

Reiterating what I said a few times in #586146: [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x .

Any proposal that increases work for security team (especially for D6, but even for D7+) without addressing the workload of the security team is a non-starter from my perspective. It's not really the security team as a whole, but specifically people working on core security issues. If we say "The people in MAINTAINERS.txt need to fix security issues in their area" I'm open to that - the answer doesn't have to come from the security team.

As it stands, this proposal does increase work for the security team and doesn't seem to address that.

"The people in MAINTAINERS.txt need to fix security issues in their area" would be a very similar process to the one the security team uses for contrib modules right now, and it would help spread the load out among 60+ people, each of whom is arguably the best people to do the work. It'd also allow us to ferret out inactive maintainers to keep the doc up to date. It seems like a reasonable request to me. What say you?

Proposed solutions

Do not release 8.0.0 until all critical and highly critical security issues are fixed

This means that issues that are critical in the security.drupal.org tracker would be included as release blockers, and hence prevent a release or release candidate as well as public critical bugs/tasks.

Advantages; will ensure there is zero backlog of known serious security issues when 8.x is released.

Disadvantages: potentially a 6.x or 7.x port of a security issue could hold up 8.0.0, however this is also an advantage depending on your point of view.

Do not release 8.1.0 until all critical and highly critical security issues are fixed

Advantages:
- Instead of a one-time injection of core developer attention on the security queue in order to release Drupal 8.0.0, this would instead happen on a cycle at least once every 6 months, helping with general sustainability of core security releases.
- Working through the security backlog ahead of releasing a new feature release means the next release there are (hopefully) very few (and possibly no) issues to work on.
- Core developers in theory will be more motivated to help with security issues if they are blocking features they care about getting released.

Disadvantages:
- Puts the "every 6 month" release commitment heavily at risk, if current trends regarding open security issues continue.
- Holding up releases on 0 criticals (never mind security-related criticals) generally means never releasing (observe the first ~5 months of Drupal 7's release), and holds up other critical bug fixes from getting released and tested.

Comments

Crell’s picture

I'm dumb enough to be in MAINTAINERS.txt a couple of times.

"Be the person the security team harasses about core security issues" seems like a reasonable part of that amorphous umbrella. Whether that means actually making the fix, reviewing/vetting someone else's fix, or just coordinating someone else working on the fix likely will vary by issue.

So, a resigned +1 from me. :-)

plach’s picture

"The people in MAINTAINERS.txt need to fix security issues in their area"

From my perspective this makes perfect sense: security team members should just coordinate the fixing process, suggest solutions, and provide feedback. Subsystem maintainers should be in charge of writing patches.

Not sure whether people being both subsystem maintainers and security team members need "special casing", though.

Anyway +1 here

plach’s picture

Whether that means actually making the fix, reviewing/vetting someone else's fix, or just coordinating someone else working on the fix likely will vary by issue.

Yep, this makes sense too, but maintainers should be the ultimate responsibles for the fix to happen, IMHO.

klausi’s picture

This is already the status quo? We are already adding people from MAINTAINERS.txt to the core security issues in their area and hope for their input.

For contrib we just make modules unsupported if the maintainer(s) fail to respond, but we cannot make form API unsupported (which has security issues often), right?

So this is just about improving communication that those maintainers are responsible to fix security issues, and if they fail to respond remove them from MAINTAINERS.txt?

catch’s picture

For something like FAPI there are several maintainers, who are active on core (if not FAPI right now) this seems OK, I think it'd also be worth a list of core developers who work across lots of subsystems and are OK being added to security issues (without explicitly joining the security team) - since lots of people who work on things, aren't necessarily listed.

There are some places where this can fall down though, for example OpenID - there is no 8.x maintainer because there is no 8.x module, and it and several other modules were removed from core because the number of maintainers fluctuated between 0 and 1. As long as we're supporting 6.x or 7.x those modules are going to struggle to find people wanting to work on them (unless there's a flourishing 8.x contrib module that might have the same vulnerability).

webchick’s picture

Just spoke with the new security team lead Michael Hess. He made the strong recommendation, which I think there's quite a lot of merit to, of adding "Critical" and "Highly Critical" security issues to the list of release blockers before a D8 feature release.

Meaning, all 8.0.x highly critical/critical security issues must be fixed before 8.0.0 is released, all 8.0.x and 8.1.x highly critical/critical security issues must be fixed before 8.1.0 is released, etc. That includes D6/D7 backports of these fixes as well, where applicable, since we can't release a public security fix for only one affected version.

This would greatly help reduce the security team burden because a) we get guaranteed core developer focus on those issues at least every 6 months (hopefully well before that, since adding the appropriate people in MAINTAINERS.txt to security issues is going to become part of the security issue creation/validation process going forward :)), and b) then what we don't have is a situation where we just endlessly increase the number of versions/branches that important security issues must be backported to, causing endless delays in getting fixes prepared for all active branches.

I think this would be a really important concession, given that the security team concerns around semver were acknowledged but not really addressed in the original proposal.

Crell’s picture

There's a lot of sense to that. My concern would be if a particular issue doesn't get attention, or is way harder than we expect, or both, it could throw off the planned scheduled release cycle. If we're going to set an expectation of "you get a new .y release every 6 months", and then 8.2 doesn't come out for an extra 2 months, we're breaking the release cycle promise we're making *and* indicating "by the way, we know that the 8.1 release you're running has highly critical OMG security holes, but we're not telling you what." That's a great key to black hats to look into 8.1 because they know there's something really bad there that we're sitting on.

catch’s picture

I think that definitely works for 8.0.0.

If we get to the situation where the only critical issues are security ones, we can still fix 8.x critical issues in the public queue until there's a stable release. So we'd release an SA for 6 and/or 7 as each gets fixed, then the 8.x issue goes into the public queue as a critical bug. Whether we're in beta or RC at that point doesn't matter - although I assume we'd block the RC on known security issues as well. Anyone frustrated that the last issues are security ones can ask to be added to the private tracker for the remaining issues if they want to help.

@Crell I'm not sure the situation you describe with the 6-monthly feature releases is any different to other critical bugs (apart from the OMG undisclosed security holes waahhh issue, but that's a PR issue rather than a technical one).

If we have critical bugs, and we're going to not release feature releases with those in, then those could also mess with the schedule. Presumably Ubuntu and other projects on timed release cycles have the same problem. I'd definitely be concerned about this if we were starting it in the middle of the release cycle, but this means we'll have a clear-out of the current backlog of security issues prior to 8.0.0, so it'd only be brand new issues that affect the next release. That part could definitely use more discussion - I don't think we've discussed 8.1.x blockers at all really, but it feels like a more general thing to get resolved to me.

catch’s picture

Title:Figure out how to make core security releases sustainable» Block 8.0.0 and possibly minor versions on critical security issues.
Priority:Major» Critical

This needs to be sorted out before rc, so it's critical. Also re-titling to reflect the current proposal. Discussed this with Michael and others on a phone call today, and a couple of things came up - will try to update later.

catch’s picture

Issue summary:View changes

Updated the issue summary a bit. I think we should definitely block 8.0.0 on the critical/highly critical security report backlog, it'll be great to have a clean backlog for all branches when it comes out.

For 8.x feature releases it's definitely more complex, hoping we can discuss that more.

Currently, 7.x releases come out regardless of how many 7.x critical issues there are. This is because 7.x releases only come out when there are already committed bug fixes that need to be released (whether security or not). If each point release of 7.x was blocked on criticals, they'd be much further apart, or possibly never happen, which means bug fixes that are already done would be held up on ones that aren't.

With patch level releases (8.0.1) against Drupal 8, we'll definitely need to have the same policy. Since those are bug fix only, there's no point holding up genuine bug fixes, just because some other bug fixes aren't fixed yet (with the exception of regressions in the dev branch or something but those can be rolled back of course).

For minor releases (8.1.x), we need to discuss first whether we block those on critical issues at all, then whether we apply that to security issues is on top of that really. On the call today a few ideas came up:

1. Block the 8.1.0 tag/RC on critical issues (both public and security). This ensures zero backlog and possibly focuses efforts on getting them fixed, but could also mean we end up extremely wide off the mark in terms of a 6 month release schedule.

2. Don't commit any features to the 8.1.x branch while there are critical issues open. This would be a zero-threshold version of thresholds. We'd be starting with a clean slate with 8.0.0, but it doesn't actually ensure that critical issues get added slowly. i.e. say we release 8.0.0 with 40 undiscovered critical bugs, they could get found at any time regardless of what goes in after, then we're back to releasing with a critical issue backlog or not at all.

3. Block minor releases on critical issues that are more than x months old (and critical regressions from the previous minor release probably since it'd be silly to release with those). So say we picked three months. If a critical issue is opened after the cut-off point, it wouldn't block 8.1.x, but it would block 8.2.x. This means the youngest issue that could hold up a minor release would be three months old, and we'd have a maximum turnover time for critical bugs during the 8.x cycle of 9 months (or hold up feature releases on them).

Of those options, I like #3 the best although clearly the details of how that might work would need to be thrashed out a bit. The impact on security fixes would then be exactly the same as on other critical bugs.

Crell’s picture

#3 sounds promising. We may need to allow for certain issues to be exempted at the RM's discretion (eg, an issue that's critical, but doesn't affect the majority of people and is crazy-ass hard to figure out so after 9 months we're still not sure what to do with it), but I think something along those lines is probably the best compromise.

catch’s picture

Title:Block 8.0.0 and possibly minor versions on critical security issues.» [Policy, no patch] Block 8.0.0 and possibly minor versions on critical security issues.
Damien Tournoud’s picture

I'm not sure how this is different then what we are currently doing. But I guess it depends on what you call a "critical security issue", is that an issue that is public (ie. already released in other branches), or do you also count all the private security issues?

It boils down to:

  • If you are talking about public security issues, this doesn't seem different from what we are currently doing.
  • If you are talking about holding releases on private critical security issues, you are never going to release. Plus, who are the developers that are going to provide this "guaranteed core developer focus on those issues at least every 6 months"? And as a consequence who are we giving access to those issues?
catch’s picture

@Damien this is specifically talking about private security issues, but only those that are 'critical' or 'highly critical' (so not everything in the private tracker for core - agreed that'd mean a situation where there's never a release).

Plus, who are the developers that are going to provide this "guaranteed core developer focus on those issues at least every 6 months"? And as a consequence who are we giving access to those issues?

MIchael Hess has suggested elsewhere that those listed in MAINTAINERS.txt for components (and others) should be more or less automatically added to security issues against those components - i.e. try to involve more people who regularly work on core on individual security issues against core, not just those who explicitly volunteer for the security team.

Whether or not that means more attention to those issues in reality depends on people's priorities - however generally the last handful of issues that block a major release get worked on regardless of how unpleasant/annoying they are simply because they're the last handful.

For 8.0.0 I think it's 100% fine to block the release on actual critical security issues in the private tracker, and I've really got no qualms if that holds an RC/8.0.0 up for an extra month or two when all other critical bugs are fixed.

Every single major release of core has had several critical bugs discovered on the day or so after release (and before in some cases), and we already committed to doing a proper RC period (i.e. the final RC should have no critical bugs fixed before the 8.0.0 release). Adding security issues in the private tracker to this is another opportunity to release something solid instead of rushed out.

How this applies to 8.1.x etc. is still up for discussion, I liked the older/younger than x months + discretion idea since that gives us a known/finite amount of issues to deal with.

webchick’s picture

Issue summary:View changes

Fleshed out the issue summary more based on what I remember.

catch’s picture

Assigned:Unassigned» Dries
Status:Active» Needs review

Could use feedback from Dries.

Personally I think we should commit to this for 8.0.0 and there aren't really any disadvantages at all.

For 8.1.0, I'm not sure it's possible to make a definite rule until we get past 8.0.0 and see what kinds of security issues and how many critical issues in general are opened post-release. There's lots of edge cases: for example, a new issue against 6.x and/or 7.x that doesn't affect 8.x. Or a new 8.x issue that doesn't affect 6.x or 7.x.

If we agree to 8.0.0 now, we could always make this 'major' and continue discussing after release, then make a concrete decision on this issue a blocker for 8.1.0. @Michael not sure how you'd feel about deferring that bit from the point of view of the security team?

effulgentsia’s picture

Assigned:Dries» Unassigned

I discussed this with Dries and he's +1 to blocking 8.0.0 on fixing critical and higher security issues, just like catch.

For 8.y.0 (y>0), he wants to keep those on a predictable 6 month cycle as much as possible, so is -1 on holding those up on security issues that are already in released versions anyway.

However, for 8.1.0, he likes #10.2 (applied to security issues only): stopping new features from being committed until enough critical security issues are worked off. He's open to us incorporating a rolling time delay into that (ala 10.3) (e.g., only critical security issues more than X months old trigger a mini feature freeze).

He also wants us to treat whatever policy we come up with for 8.1.0 as an experiment. If it works well, continue it for 8.2.0, but if it introduces problems (e.g., people who want to work on features aren't willing/able to help on security fixes, and end up getting slowed down with no compensating gain to the project), then we adjust the policy until we find something that works.

catch’s picture

Status:Needs review» Reviewed & tested by the community

OK marking RTBC for the 8.0.0 blockage. Should be documented somewhere (security handbook? core issue priorities/release cycle docs?) and let's leave it in the queue for a few more days for last minute comments.

Don't think we'll properly be able to come up with a policy for 8.1.0 until 8.0.0 is out.

David_Rothstein’s picture

Blocking 8.0.0 on known security issues that affect Drupal 8 certainly makes sense, although I don't really understand why it's only certain priority levels? What this means is:

  1. If I discover a moderately-critical security issue (let's say a low-level-admin XSS issue) that only affects Drupal 8, I would file it as a critical bug in the public issue queue and it would block Drupal 8's release.
  2. If the same exact bug instead affects Drupal 7 and Drupal 8, I would file it privately with the security team, but because it's only moderately-critical it wouldn't block the Drupal 8 release?

That doesn't make sense. (Then again, our currently policy makes even less sense, so I guess it's progress to fix the workflow for the higher priority issues for starters.)

Also no one really addressed Crell's concern in #7 about telgraphing the existence of critical security issues to the world. This affects the 8.0.0 proposal as well. In general I think we should be more public about the number of open security team issues, but telling (or effectively telling) everyone that Drupal 8.0.0 is being held up on known high-priority security issues that also affect the stable D6 or D7 branches and that the security team is actively working to fix... that's going too far.

One way to address the above would be after a certain point (say the first Drupal 8 release candidate) simply encourage - but not require - people to report all Drupal 8 security issues privately. Then if the 8.0.0 release is held up for critical security issues, it won't be publicly known whether the issues are D8-only or not.

David_Rothstein’s picture

Also, this is an interesting proposal, but I don't see how it has much to do with the original purpose of this issue (original title: "Figure out how to make core security releases sustainable").

Once this is done, can we restore that title and continue brainstorming?

catch’s picture

If the same exact bug instead affects Drupal 7 and Drupal 8, I would file it privately with the security team, but because it's only moderately-critical it wouldn't block the Drupal 8 release?

We could block 8.x on lower priority issues in the private queue against 8.x. I do think it makes sense to not block 8.x on lower priority issues that only affect 6.x/7.x though.

Also no one really addressed Crell's concern in #7 about telgraphing the existence of critical security issues to the world. This affects the 8.0.0 proposal as well. In general I think we should be more public about the number of open security team issues, but telling (or effectively telling) everyone that Drupal 8.0.0 is being held up on known high-priority security issues that also affect the stable D6 or D7 branches and that the security team is actively working to fix... that's going too far.

I've addressed it, in that I think that clearing out the backlog for 6.x and 7.x, and having an 8.0.0 release with a clean slate is considerably more important than PR. Allowing security issues discovered post RC1 to be reported privately seems fine.

Also, this is an interesting proposal, but I don't see how it has much to do with the original purpose of this issue (original title: "Figure out how to make core security releases sustainable").

I'd rather a meta-issue than keeping this one open, but yes there's definitely lots of other issues than this. I think this is mostly a one-off, partial solution to #2136029: Decide if and how to extend D6 security support 3 months past an 8.0.0 release. Specifically to avoid the situation described in that issue where lots of 6.x security issues remain outstanding until 6.x is dropped, then get publicly disclosed within a month or so, because we're finally able to release the fixes for 7.x/8.x that have been held up on non-existent 6.x patches and testing. Doing similar for 8.1.x and 8.2.x would also mean that new critical issues with 6.x/7.x don't pile up either.

It doesn't at all address the maintenance issues with 6.x or whether anyone's realistically likely to keep supporting contrib at all. But at least with all critical issues resolved one argument for continuing to maintain support goes away (i.e. we don't have that backlog to get disclosed).

There's a couple of things that were discussed on a phone call that have not been mentioned here, specifically having a block on the d.o dashboard that shows 'my s.d.o issues' so people can see updates without having to visit the site separately - for component maintainers subscribed to specific core issues that'll make it easier to keep track of them.

greggles’s picture

The "my s.d.o issues" is at #2189621: Add a block for issues from security.drupal.org awaiting a review.

Anyone have notes from the call?

catch’s picture

Not that I know of, I tried to get people to comment here instead.

catch’s picture

Status:Reviewed & tested by the community» Fixed

Status:Fixed» Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.