- Drupal 8 is currently in beta: https://www.drupal.org/node/2350615
- The next possible milestone is supporting a beta to beta upgrade path, see: #2341575: [meta] Provide a beta to beta/rc upgrade path
Those issues discuss and document the patches that can be accepted, and any additional criteria, for those two milestones.
This issue is for discussing/documenting what happens once we get to zero critical issues remaining. As a reminder, once Drupal 8.0.0 is shipped, the intent is to put new minor releases of Drupal 8 out every 6 months (8.1.x, 8.2.x, etc.) which include new features other improvements that preserve backwards-compatibility.
Summary
Drupal 8 (like any software) will continue to have critical bugs found during the release candidate phase and after release. What is important for site owners is that the 8.0.0 release is stable enough that occasional new critical bugs can be resolved quickly, instead of adding to an extensive backlog of (as-yet undiscovered) issues that will cumulatively take many weeks to resolve. For core contributors, it is important that 8.0.0 marks the point at which issues which have been deferred to 8.0.1 can be worked on again, since this may include several hundred major bugs that could seriously affect sites in production.
To this end, the branch maintainers will monitor the rate and nature of incoming critical bugs weekly during the release candidate phase (beginning before the first release candidate). To allow for a regular release schedule with incremental improvements from the previous release, we may tag release candidates with open critical issues. We will schedule an official release date for 8.0.0 a minimum of three weeks in the future when we are confident that the rate of incoming bugs has slowed enough to ensure a stable, timely release.
- Release candidates
-
- Release windows for Drupal core are on the first and third Wednesdays of each month. For stable releases, the first Wednesday is a bugfix release window and the third Wednesday is a security release window.
- Once we reach zero critical issues, the first release candidate is tagged on the next of these release windows (whether bugfix or security) to synch up the Drupal 8 release schedule with that of Drupal 6 and 7. (If the next release window is less than 72 hours away, the following one will be used instead.)
- Further release candidates will be tagged on a fortnightly schedule during each release window.
- If a release candidate goes out with a known critical issue, this will be documented prominently in the release notes.
- During the release candidate phase, only critical issues (as well as documentation improvements, or major issues with very low chance of disruption/regression, on a case-by-case basis) will be committed. Everything else will be deferred to 8.0.1 or later.
- 8.0.0
-
- When the rate and nature of incoming critical issues has slowed to ensure a stable, timely release, we will schedule a release date for 8.0.0. The release date will be announced at least three weeks ahead of time, and will be set on one of the existing core release windows (on the first or third Wednesday of the month).
- Once the 8.0.0 release date is set, if there are not unexpected critical issues open, we'll allow code-style cleanups to be committed again so long as these have low chance of introducing regressions.
- Once 8.0.0 is tagged, patches deferred to 8.0.1 can be committed again.
- Patch-level releases will follow a monthly schedule corresponding to Drupal 7's release windows.
| Comment | File | Size | Author |
|---|---|---|---|
| #55 | d8_rc_phase_mar_17.png | 43.08 KB | xjm |
| #52 | d8_rc_phase_diagram.png | 44.5 KB | xjm |

Comments
Comment #1
xjmComment #2
catchGoing to go ahead and bump this to critical, it look quite a long time to get to a fully agreed and documented beta changes policy and it ended up being several weeks after beta 1 was cut, so this time we should make sure things are documented before we get there.
Suggested tl;dr
Comment #3
catchThe two main things I'm not sure of here:
Dries (I think) suggested the idea of picking a date for 8.0.0 release at the point where we'd normally just tag it, so that there's a chance to organise events etc. Since at that point we think we have the actual release ready this seems OK to me (i.e. there's no dilution of 'ready when it's ready', it's more 'released at a specified time after being deemed ready' which is extra time). However if we get a really, really horrible critical at the eleventh hour like a publicly disclosed security issue, what happens?
We need to decide exactly what class of issues will block 8.1.0 being released- critical issues introduced since 8.0.0 was tagged seems obvious - we ought to be able to roll back if nothing else.
Issues discovered after 8.0.0 was tagged but already in that version are less clear cut - although not opening 8.1.x with outstanding criticals should reduce the odds of that slightly.
Comment #4
catchComment #5
jhodgdonWow, great that this is being discussed now, and I think the issue summary and comment #2 both look great! Seems pretty clear and very rational.
No idea about the questions in #3. Good luck with those. :) But maybe you don't need to spell out or think of a response for every possible contingency... I think the policy is generally clear that 8.0.x should be released a short time after all the Critical issues are fixed, and that if possible it won't be released with critical problems, which is the main idea, right?
Comment #6
catchDidn't actually bump to critical in #2. I've also gone ahead and added the tl:dr to the issue summary.
Comment #7
gábor hojtsyWould be important to factor in some kind of typical work required up to the release.
Where does UI / string freeze comes in? Sounds like that would not be "code cleanup" so based on this policy both would need to happen before/at RC1. That makes sense because RC1 may become the release if there are no more criticals. Do we provide a window though for translators and UI fixers to have time before a release may hit? When should those earlier deadlines be cut? We don't exactly want to focus people on working on UI cleanup / UI text cleanup now, right? We'd love people solving critical issues swiftly all around. However if there is no space / process for fixing the UI then (those are not critical) then when is that going to fit in?
Comment #8
wim leersI'm not sure how broad this issue wants to go, but the IS talks about a 8.(N+1) version, but not about a 8.(N-1) version. i.e. backporting of bug fixes. But I suspect that's out of scope for this issue?
Comment #9
catchOnce 8.1.0 is released, there is zero support for 8.0.x at all, so no backporting needed from stable release to stable release.
However you're right that this doesn't mention backports from 8.1.x-dev to 8.0.x which of course will be necessary as soon as 8.1.x is open. Additionally since we won't support 8.0.x once 8.1.x is cut, we'll need some way to ensure that at least the critical issues go into 8.0.x promptly.
Comment #10
webchickIn most cases, won't these just be git cherry-pick?
We could say we don't commit any issues to 8.1.x without a patch that applies cleanly to 8.0.x as well.
Comment #11
catchYes to both, that might be enough.
Comment #12
catchI don't think there's another time for it, both of those would count as 'prioritized changes' during beta.
String/translation freeze is probably when the first release candidate is cut yes. Even if that ends up becoming the release, it should give time to actually do the translations since we won't be tagging super-fast, see this bit:
So that's at least ~10 days since the release candidate was tagged, plus at least ~10 days since we hit zero criticals before that (whether it was a beta or rc), works out about three weeks.
Then on top of that ~three weeks, we'll be setting a scheduled release window in the future for actual 8.0.0 release, which I imagine will be a decent chunk of time ahead (say another couple of weeks? more? This bit wasn't my suggestion and I don't know exactly how much time is needed for international party planning). Either way that leaves lots of time.
Comment #13
gábor hojtsyString/translation freeze is not really helped with the 10 days prior to the last RC, because localize.drupal.org will only get releases that are cut, not dev versions, because they change all the time (you don't want translators chase strings that may only last days in the code :). I think its important to make this clear then because those writing books who want them to get out at an early time as well as translators and even more so those who want to fix UI/strings will need to plan with this then.
Comment #14
catch@Gabor there'll be a 'final' beta cut each time (final ... each time ... hahaha) we hit 0 critical issues prior to the first release candidate, so that ought to help a bit?
Comment #15
dries commentedThanks for the writeup, catch. I think this is a great start to the discussion. I have a couple of concerns:
- I'm concerned that "no new criticals discovered for 10 days" for each RC, and therefore 20 days for 8.0.0, is a standard that can result in pushing a D8 release out much farther than it needs to be. I'm quite confident that critical issues will continue to be discovered even after 8.0.0 is released. Critical issues are still being discovered against D7, years after its release. Can we change this metric to only including really bad (definition of that TBD) criticals? As an example, I think #2181811: System info can become inconsistent if hook_system_info_alter() makes a module required and #2368975: ElementInfoManager::buildInfo() processes info data on every request are legitimately critical issues, meaning that I think it makes sense to hold up RC1 on stuff like that when that's known ahead of time. But I'm concerned if discovering new issues like that keeps pushing out our release, because there's probably no end to that process.
- I also don't see why we need a 10 day critical free window to go from RC1 to RC2. I think something more like 2 days would be sufficient. If we release RC1, then 3 criticals are discovered in the first 3 days, and we fix them a few days after that, why not release RC2 right away (or maybe after a 1 day commit freeze)? Wouldn't it be better to keep RCs flowing quickly, so that we get faster feedback from the sites out there that will be chasing them?
I don't think this makes sense. For D7, we had a 2 week window (Dec. 21, 2010 to Jan. 5, 2011) between announcement and release, but I heard from webchick that in hindsight this was too short, and that the people organizing the events would like more time, maybe as much as 2 months. Delaying a D8 release by 2 months after it's already fully ready seems foolish to me. I think it would be more practical to announce a date that's 2 months out once we reach something like RC2, and then release 8.0.0 on that date even if there are some newly discovered criticals that need to be pushed to 8.0.1.
Comment #16
catchYes, I'm sure it will, the idea here is that by definition there will be less criticals discovered after release if we fix more during RC. I very strongly feel we need both a decent stretch of release candidates (assuming people find critical issues in them at least), and to allow 'final RC blocker' patches time to bed in before we release the next release candidate. A release candidate should be an actual candidate for release, not something we cut every time we get to zero critical issues for 24 hours.
If we look at Drupal 6, a critical security issue #221072: Missing access_control on theme configuration was fixed February 13th, exactly one day before Drupal 6.0 came out (we actually had a fairly long RC period for Drupal 6 due to drag and drop, less so for 7.x).
If we look at Drupal 7, it released with a 'Drupal 7 release blocker' still open, that was overlooked because it was an issue that had to be fixed in Drupal 5/6 (#895014: All fields of a node type are lost on module disable). The same bug was reported in the Drupal 7 queue one day before Drupal 7.0 was out (Jan 6th) , except not marked as critical because the symptom unfortunately didn't make it obvious how serious the bug was (#1017672: D6 to D7 update process permanently deletes comment bodies and other data, and throws fatal SQL errors). I bumped that to critical on January 11th, but it took a further 5 months to get actually committed, while sites were haemorrhaging data meanwhile.
Also if we look at the Drupal 8 beta, critical follow-ups to the final beta blockers were posted within 2-3 days of the release, for example #2339219: [meta] Finalize URL generation API (naming, docs, deprecation) and #2351379: [meta] Define, then support exact use cases for link generation and storage - if we'd held the beta an extra couple of weeks (i.e. to allow the final beta blockers to settle in a bit), they might well have been discussed and fixed within those two weeks to get the beta out, now they're sitting there with the other 125 critical issues in the queue not getting a great deal of attention. There haven't really been any new beta blocking-ish issues since then, but these two are really affecting module porting and have no momentum.
So a longer release candidate period does the following;
I'd rather do that than there be release parties booked in advance for something that isn't remotely ready, which is exactly what happened with Drupal 7.
Comment #17
webchickWhile I certainly agree that we want to avoid those situations if we can, I think it can be dangerous to base current/future decision-making on those specific mistakes of the past, for two primary reasons:
1) We've introduced numerous process changes since then to avoid exactly those problems. Among them:
2) At this point, I'm of the strong belief that we do more harm to the overall Drupal project and ecosystem by holding Drupal 8 longer versus having it release with a few bugs, even if they're nasty bugs. Among my concerns:
In short, there's a lot riding on Drupal 8.0.0, and it impacts the entire community and ecosystem, not just developers and site builders. We obviously want to make sure we do our best to ensure that we don't put out a crappy product that blows up in peoples' faces. But at a certain point, we will actually do more harm to Drupal by holding it until every last problem is fixed versus getting it out there and continuing to improve it after. Remember that Drupal 8.0.0 is a .0 release, and it's a well known best practice to avoid .0 releases of software if you care about stability. It's perfectly ok (and expected!) if it is not 100% perfect. It never will be.
One final point: One of the factors that played into 7.1 being 5 months from 7.0 is I kept waiting to be back to zero criticals before cutting the tag. Going back down to zero critical issues in 7.x never actually happened. And even today, almost 4 years after D7's release, there are still at least 3 known critical issues in D7: https://www.drupal.org/project/issues/search/drupal?project_issue_follow... So the above proposal could at best delay Drupal 8 by weeks/months, and at worse literally lead to us never shipping Drupal 8. :\
I think I'm with Dries. Wait until ~RC2, pick a date. And do whatever we have to to work toward that date. We need to practice working toward dates anyway if we're planning to do 6-month release cycles, and far worse than Drupal 8.0.0 shipping with any kind of horrible bug you could imagine (there's always 8.0.1, 8.0.2...) would be for Drupal 8 to be hold for weeks/months longer than it otherwise would, for all of the above reasons.
Comment #18
catchIt's a lot more dangerous to skip over previous major core releases as a precedent.
I really don't think there's any real indication of health from lasting just 48 hours without a critical bug in the queue. We manage 48 hours without a new critical bug in the queue every week.
The beta to beta upgrade path, and a release candidate are as least if not more important for this than the actual 8.0.0 tag IMO. Especially if you're building a site as opposed to launching it, there's nothing wrong with release candidates at all.
At least for me, the strain got considerably worse once Drupal 7 was released than while it was at release candidate (and I was getting paid to work with Drupal 7 for about 18 months beforehand too). I'd be interested to hear from people who are actually fixing some of the harder Drupal 8 criticals on how they feel about this.
"Every last problem", really? Can we please drop the hyperbole and stick to what's actually in the proposal/discussion?
We're talking about issues that should be critical/release blocking (if they're not then they should be demoted to major), and going 20 days from the last one being fixed without a new one coming through before tagging. 20 days really isn't very long.
Yes this in large part because the release was rushed out and critical bugs got ignored for months afterwards (since it had been released, so they were no longer blocking any milestones except actual sites working and stuff), combined with the backport policy and focus moving to 8.x. Like you say at least some of these factors shouldn't be there this time.
Edited to fix bad HTML, catch--
Comment #19
Anonymous (not verified) commentedi agree with catch's proposal, and rejection of 'every last bug' hyperbole, which is the worst kind of straw argument. we're talking about ignoring literally hundreds of major bugs with this proposal.
i think we should hold the line longer, and be more certain that we won't release (and then ignore for a long period) a data-eating bug like we did with 7.x.
i don't think waiting 10 days between milestones will harm Drupal. if you try to put in a critical issue during one of those windows, you can be sure you'll be smacked down. the only way a bug will be allowed to remain a critical is if enough core devs agree it's really nasty, in which case The System Works, we caught a bad one before we potentially ate some production data.
also, 'harming Drupal' is not some objective thing we can measure - it has a different meanings for different people in the community. personally, i give no fucks about Drupal's install base growth, market share or 'beating' our competitors. rather, i think pushing things out quicker because of competitors, or worries about Drupal's market share, is harming Drupal.
Comment #20
dawehnerRelease process
I have to agree with catch that 48 hrs is not enough, given how people contribute/report issues. They often have
one day on the week on which they do concentrated work on core, bypassing them would be problematic. On the other
hand 10 days critical free might be too much as well, hard to decide but more later.
Maybe its hard to accept, but if we don't think about majors as well, they will be criticals
for the actual sites being affected. Having time before the rushing/release will help people to still
shape the release at the same time, see burnout syndrom later.
In general I wonder whether we really need a RC has to be critical free, for a while. I really like
that betas itself already have a predictable release schedule, so will have the 8.x.y releases,
but the RCs won't. Having a schedule would help people to plan updating their sites they work on.
How can we encourage people to build sites with D8
It was layed out, that people don't start to build sites with D8, partially because we don't
ensure update support, but from my point of view much more, because people don't
really know in which state D8 is at the moment, and based upon that can't even evaluate
whether they can built a client site with D8. In that context we can place in the idea of @catch and others,
to let shops built their site in D8. We should PUSH that, maybe with a blogpost somewhere.
Burnout syndrom
As we see more and more, people start to get burned out more and more, especially by things like the beta rush,
or in the future the RC rush. If you want to get a release out (no matter whether its 8.0-RC2, 8.0.0 or 8.1.0), please don't rush.
IMHO the normal daily work is not what burns people out, but rather the enforcements, both on schedule/issue level. Given that
having these 10 days before whatever part of the release process helps a LOT to reduce the stress level. At least in theory people can take something like a "break"
for a while. Especially once the release is out, contributions on core will slow down at least for a while. Contributions on other parts of the ecosystem, like contrib, might
go up though at the same time.
Comment #21
kattekrab commentedWe are all keen to see Drupal8 out there. The sooner the better!
But...
We need to provide time for people to test, replicate and report these issues, agree they're critical or not. And we need to be mindful that some people can only do this on weekends, and others only on weekdays, or in the evenings, or early mornings or whatever. We also need to beware the dreaded drag of timezones. Key people might need to align to chat through the issues. I think we need at least a week between each milestone of RC, final RC and the actual release.
I understand the frustration and desire to get D8 released. But I feel we'd be doing a disservice to our future selves to rush through the RC process.
Comment #22
webchickIt's fair to call out what I said as hyperbole. I didn't mean it that way, but my shorthand was too short. Apologies.
Here's a better way to say what I was trying to say:
Not all criticals are created equal. What I'd like to see us do in the event that a critical pops up during a 10-day window is discuss it as a group and decide whether or not it's worth restarting the clock for it.
I can respond to the rest more in detail tomorrow, but wanted to get that out quickly.
Comment #23
Anonymous (not verified) commentedre. #22 - webchick++
i think a group discussion about whether an issue is really a critical is a great idea. we have to carefully weigh up whether it's worth resetting the clock.
however, i don't think we need a new mechanism or 'this is critical, but not a critical critical' concept.
if a critical is created, we discuss it. if the majority of core committers think it's a For Real Critical, we fix it ASAP and reset the clock. if the majority of core committers think it can be demoted to major, we demote it and don't reset the clock.
Comment #24
gábor hojtsySo there is not much contesting those things on this issue, looks like the release date is more interesting. Should we consider this part an agreed policy then? Looking at what happened in Drupal 7, based on https://localize.drupal.org/node/640 and https://localize.drupal.org/node/712 it is hard to tell when/if there was a string freeze. We released Drupal 7 a year after. I'd like to keep translators up to date on what they should expect about translatability so they don't translate too soon but also they don't strt so late that they would need to rush.
Comment #25
catchYes I failed to mention this as an explicit goal, but this is exactly what burns me out (and apparently dawehner and beejeebus too), and exactly why I think a decent critical-free stretch will reduce burnout rather than make it worse.
Once we're at RC point, we'll be in near commit freeze (probably only critical issues and straightforward major bugs affecting actual sites). So really then people are only fixing very serious new bugs that are found, quite serious bugs that affect them/lots of sites but don't require disruptive fixes, or waiting around for new critical issues to be opened (and testing/translating/porting a lot hopefully).
If we don't add new critical issues at a massive rate, then this will actually be a calmer and happier time than the run up to release candidate - which is what you want when the consequences for introducing regressions are higher.
As soon as 8.0.0 is tagged (or even scheduled), people are going to start re-rolling and RTBC-ing issues we've punted to 8.0.1. If we're still rapidly adding new critical issues at that point, we get stuck between pushing those minor fixes back to 8.0.1 or 8.0.2 (breaks people's confidence and starts to risk the 8.1.x schedule/thaw too) or committing them despite open critical bugs which makes it harder to get the critical bug fixes done.
This is interesting and that's definitely the main weakness of the current proposal, but I couldn't think of a way to do that if release candidates are supposed to be actual candidates, that might be the thing to let slip!
The thing I really think we need here (and looks like beejeebus, dawehner and Kattekrab agree) is a solid period where we're critical free before the final decision is taken to tag 8.0.0. It's this that allows us to say we've fixed the 'known unknowns' in that the churn rate has gone down.
However, having that period at the end, doesn't mean we necessarily need to wait 10 days bug-free for every release candidate.
So... what if we put out release candidates on a fortnightly schedule - and we tag 'release candidates' with open critical issues, as long as they aren't regressions from the previous release candidate. I don't know if we can do rc1a, rc1b vs. rc2 on Drupal.org, but that would be the distinction internally at least.
That way we get to keep a predictable release schedule, some release candidates we'll know aren't going to be identical to the final tag, but they'll be better than the previous release candidate which is useful for module porters and site builders. We'll want make sure we avoid regressions in release candidates, but can handle that with a commit freeze of 48-72 hours prior to each one same as we do for betas.
To see how that could look in practice, let's say exactly the same number of bugs are found with the same distribution of timing, but compare the ~20 days and 72 hours/RC2 approaches. I'm also assuming that 8.0.0 tagging happens 4-6 weeks after the countdown starts.
Last beta -> 0 criticals.
Week 0: RC1 tagged with 0 open critical issues, 5 new critical issues found, 5 fixed (0)
Week 2: RC1a. 6 new critical issues found, 4 fixed. (2)
Week 4: RC1b. 9 critical issues found, 11 fixed within 10 days. (0)
Week 6: RC2. 0 critical issues found, 0 fixed. (0)
Week 8: RC 3. 'Final' release candidate. 8.0.0 countdown starts. 3 critical bugs found within 14 days (but at this point tough)
(...possibly keep with bi-weekly schedule here..)
Week 12-14, 8.0.0 is tagged, we do our best to do so with 0 criticals, but internally this is more like 8.0.1 for us - any new criticals found after RC3 were 'unknown unknowns' that we just didn't find despite the process, and no process can help with those.
With this scenario, if we hit zero again 3 days before RC2 is tagged, then 3 + 14 = 17 days critical free before RC3. If we hit zero again shortly after RC1b, then say 10 + 14 = 24 days.
That gives us a rough minimum/maximum of 17-24 days without a new critical issue being opened before the 8.0.0 tag decision is made, which works for me in terms of ~20, and we can stick to a release every two weeks while doing that.
I'd put the above scenario as medium case - we find 20 new, genuine, critical issues after we get to zero the first time, but they're mostly front loaded and fixed relatively promptly too. The 'real' release candidate period is two months (shorter than Drupal 6's iirc), then whatever time for party planning. Not fantastic but not bad.
For the RC2/72 hours case for scheduling 8.0.0, it'd look more like this:
Last beta -> 0 criticals
Week 0: RC1 tagged. 5 new critical issues found, 5 fixed. (0)
Week 2: RC2 tagged. 3 new critical issues found and fixed within 4 days, 72 hours after the fix is committed 8.0.0 release window is set, immediately tag RC3 (0)
Week 3: RC 3 (final) tagged. Drupal 8 countdown starts. 12 new critical issues found between week 3 and week 6 after this decision is taken (so again, tough at this point)
(possibly keep with bi-weekly schedule)
3 bugs are found during week 8-10 (also tough for these)
Week 7-9: 8.0.0 tagged with ? open critical issues
The release comes out faster, but there's 12 critical issues opened after the point where the 8.0.0 release window has been set when we'd otherwise not have set a date yet, so the time pressure on those is much higher, then we still get 3 added either just before or after 8.0.0 gets released on top of those.
The situation that Dries and webchick are concerned about is more like the following - for this I've kept the first two weeks the same then diverged with the bug distribution:
Week 0: RC1 tagged. 5 new critical issues found, 5 fixed. (0)
Week 2: RC1a. 6 new critical issues found, 4 fixed. (2)
Week 4: RC1b. 1 new critical issue found, 2 fixed. (1)
Week 6: RC1c. 1 critical issues found, 1 fixed (1)
Week 8: RC1d. 1 critical issues found, 1 fixed (1)
Week 10: RC1e. 1 critical issues found, 1 fixed (1)
Week 12: RC1e. 1 critical issues found, 1 fixed (1)
... goes on forever.
In this scenario, the long release candidate period hasn't helped us to find any more critical issues prior to tagging 8.0.0, but it's dragged out a lot longer. Making the decision 72 hours after RC1a got to 0 critical issues starts to look like the right decision/timing.
However, if this happens, it's going to be pretty obvious what's going on. Unless every new critical bug is exactly 14 days apart, we'd be getting very close to the minimum of 17 days quite regularly but it'd just happen to be across the two week cycle, preventing a clean release candidate but not indicating awful stability issues.
i.e. two release candidates with one critical bug each, that's 28 days with two critical bugs opened and three fixed. This is a 'lucky' situation to be in, and easier to bend the rules from that point - and we'll know within a month if that's happening or not because it gives us the time to actually see the trend and make a decision based on it.
The very worst case scenario for bug reports regardless of the model, is something like:
Week 0-2 - 1 issue reported
Week 2-3 - 3 issues reported
Week 3-4 - 25 issues reported.
The ~20 day window protects us against this kind of distribution, but the 72 hour one absolutely doesn't, and is considerably harder to backtrack from since we're locked in much earlier. Might have missed some other variation where things could differ dramatically, but I think those are fair scenarios to compare the two.
Comment #26
catchOn critical vs. major vs. really critical.
The only class of issue I can think of that is 'critical but not critical' is third-party library updates. Everything else I agree with #19/20 that in 99% of cases issues are critical or they're not, and the borderline ones we need to discuss anyway.
Until we get to release candidate, I feel that library updates are critical because not doing them beforehand opens us up to getting stuck on an unsupported version because we never quite get caught up once we're in release candidate and afterwards.
Once we get to release candidate, we've effectively said we'll release with whatever versions we're on (because otherwise we'd not have put out a release candidate). Delaying our final release at that point due to new releases elsewhere seem a bit odd unless there's specific reason we have to (Symfony security issue, massive jQuery performance improvement on mobile that requires an API change, or something).
We also need to discuss at some point whether to update libraries in patch level releases. Probably we won't unless it's necessary to fix a critical bug, so just general library updates could happen in minors - every six months is enough to keep up with most things.
So I think the change there is that:
Looking at Dries' examples I think both can be handled within the existing major/critical definitions, the performance issue is slightly more complex so discussing that first.
1. #2368975: ElementInfoManager::buildInfo() processes info data on every request
This is partly answered by #1744302-65: [meta] Resolve known performance regressions in Drupal 8 I think. While overall Drupal 8 performance is a critical bug, that comment gives us a framework to identify what threshold of individual issues should independently hold up the release. The ms numbers for individual issues on there are entirely conjectural - we might have to tighten the criteria or loosen it depending on what gets identified and how much we can fix, but the idea is that it identifies both the issues that really need focus now, as well as those ones which are important but puntable, and that we actually fix enough performance issues to close the meta-critical eventually.
Once we close #1744302: [meta] Resolve known performance regressions in Drupal 8, we're saying that overall performance is acceptable (or giving up ever fixing it...). At that point, the number of remaining performance issues that are undiscovered (whether actually fixed or not) and can't be fixed in a patch version should be a lot smaller. The most likely category is serious scaling issues with lots of modules or content since those are harder to find with stock core and could require schema updates or API changes.
So.. that's a case where out definition of major and critical might shift slightly, based on the overall performance health of the release, but a really serious performance issue that requires a schema or API change (out of memory or 30 second query on all sites with more than 5,000 nodes for example) I think both deserves to be critical and also to reset the timer - since there could be fallout from the schema or API change.
Newly identified performance issues with stock core are more bonuses than necessary to help us get under a threshold - because we'll have already met the general threshold based at this point. Really depends what we eventually decide is 'acceptable' to close the meta-critical.
2. #2181811: System info can become inconsistent if hook_system_info_alter() makes a module required
We actually have several critical issues at the moment which are adding hook_system_info_alter() implementations to make modules required to resolve critical CMI/content dependency issues, for example #1002164: The Book module can be uninstalled with nodes with a book node type still existing and #2348925-37: Uninstalling a filter plugin removes text formats - the latter issue is a security concern too.
We might end up getting rid of the hook_system_info_alter() implementations in core in favour of validating on module uninstall instead. If we do that, then #2181811: System info can become inconsistent if hook_system_info_alter() makes a module required really seems major (i.e. core would no longer rely on hook_system_info_alter() for this critical data-integrity checking, and would provide a mechanism for the most common use case to be avoided by contrib modules too.
If we keep adding hook_system_info_alter() to core for config dependency handling and don't switch away from it to validation before release, then that issue looks more and more critical and less of an edge case - since many core modules are going to rely on the behaviour and there's no alternative way to do it.
So how critical the hook_system_info_alter() issue is depends on context. Right now core relies on it for absolutely essential data integrity and security protection -> critical. If/once core doesn't rely on it for this -> not critical.
If that issue had been discovered as the last critical in RC1 instead of a few months ago, we'd definitely need time to figure out how critical it really is and to what extent the rest of core is relying on it. So depending on the answer to the above two questions, it would either be 'really critical' or 'major' - but that's a distinction that can be made based on the context of the bug within the overall state of the release.
Comment #27
catchUpdated the issue summary based on a bi-weekly release candidate schedule.
Comment #28
xjmComment #29
xjmFollowing #25 and #26, we discussed this a bit more with @Dries, @catch, @webchick, @alexpott, @effulgentsia, and myself. There are two things we need to take into account when we decide to release 8.0.0:
To try to address both these points, @catch and I drafted this for the release cycle page, based on those earlier discussions:
I'll incorporate this into the summary in a bit, but wanted to post this for everyone's review.
Comment #30
effulgentsia commentedLooks great to me. A couple questions:
Should this be prior RC instead of first RC?
Is that long enough? Sure, minimum means we can make it longer, but if the purpose of giving a time minimum in this doc at all is to satisfy concerns of the various people who need that information, does 3 weeks accomplish that?
Should this be incoming critical bugs?
Comment #31
catch1. I'm wondering what we'd do if we have a release candidate with a regression, then for some reason haven't fixed it by the next one. For example in Drupal 7 we had at least one issue where we wrote a bad upgrade path that corrupted data. We then had to ensure that the data was fixed in sites that had updated to that release, but additionally ensure that sites which were never on the bad release (say a fresh upgrade from Drupal 6 or didn't upgrade because of the bug) also got a clean upgrade path. That could easily take more than two weeks to fix. 'Prior RC' doesn't cover a situation like that unfortunately.
2. Minimum of three weeks, if we hold of the release of 8.0.0 only because of party planning, then that should be the shortest amount of time possible. It's a bit frustrating spending effort on trying to come up with a framework on this issue where we announce a release window rather than just tag - a lot of the discussion would be easier without that extra period at the end. Drupal.org should be release-ready when we get to release candidate IMO.
The announcement also marks the point where we officially set a date to drop 6.x support, but we'll have already been at RC for some amount of time by that point, and the three months has already been stated - so any extra time is extra.
Comment #32
Anonymous (not verified) commentedjust for the record - completely disagree with #29. it reads in a very contradictory way IMO.
"What is important for site owners is that the 8.0.0 release is stable enough that occasional new critical bugs can be resolved quickly, instead of adding to an extensive backlog of (as-yet undiscovered) issues that will cumulatively take many weeks to resolve."
there is no sure fire way to guarantee that we can reliably, quickly address yet-to-be-discovered critical bugs. however, releasing with zero critical bugs is the single biggest thing we could do to ensure we can quickly resolve further critical bugs, yet we seem to be explicitly rejecting that.
so given "even if we find criticals, we may just ship anyway because YOLO", i think that paragraph from #29 should say something like:
"What is important is that we ship Drupal 8.0.0 on a date, because $reasons. We favour a timely release over stability, as evidenced by our willingness to release even if we find critical bugs in the lead up to the release date."
much more directly reflects that we are explicitly making it more likely that criticals discovered after release will be less likely to be able to addressed quickly.
Comment #33
xjmThere is nothing in this that says we're going to deliberately release with critical bugs. We have to get to 0 critical bugs before we create a release candidate, so we won't even be discussing a date until that happens. It's subsequently uncovered bugs that we're trying to address.
Comment #34
Anonymous (not verified) commentedre #33 - great. so are we going to wait for 10 days after we hit 0?
Comment #35
xjm@beejeebus, we dropped the explicit, possibly endlessly looping 10-day clock thing in favor of paying attention to the rate of incoming critical issues overall. So if we're having 5 new critical issues per week around the first RC, we're certainly not picking a date until that settles down. On the other hand, if by some miracle we have only had 1-2 criticals incoming in the month before the first RC, and then 1 additional one between the first and second RCs, then we'd probably reasonably feel confident enough to set the date. And then starts the three (but probably more like 4-6) week countdown.
(Edited many times.) ;)
Comment #36
catch@beejeebus so for me the most important things are the following:
1. That there's a sufficiently long commit freeze prior to each release candidate being tagged to ensure that critical bug fixes haven't caused further regressions. 72 hours seems like an absolute minimum for that. This would have prevented the beta coming out when the link storage issue was found at DrupalCon Amsterdam, for example. Could definitely be longer too.
2. There's a decent stretch of time with a very low number of new critical bugs coming in before a decision is taken that the release is ready (ignoring the fact that this means setting a date in the future, let's assume we were going to actually tag 8.0.0 that week).
For #2 it's then a case of defining what 'a very low number of new critical bugs coming in' looks like. Here's three possible scenarios which for me all indicate a reasonable degree of stability:
a. We find seven critical bugs in the first week, fix them in the second. We then tag a new release candidate with zero issues, and manage three weeks without any new critical issues being found.
b. We find a new issue every two weeks for 14 weeks, but the total number open at any one time never goes above one or two, then we get back to zero again. All of the issues we find are straightforward to fix silly stuff, not horrible upgrade path hell or security.
c. We find one new issue, but it takes 8 weeks to fix it due to figuring out an API change that will affect contrib modules (or $reason). During that time, no new issues are found. #1333730: [Meta] PHP DOM (libxml2) misinterprets HTML5 could be an issue like that if we'd found it after release candidate instead of four years ago.
With the (a) scenario, I'd really want a completely dry three week period after we hit zero to avoid 'lucky week' syndrome.
With (b) and (c), assuming a trickle of issues like that, then tagging the final release candidate after 72 hours then making a final decision 72 hours after that seems OK - i.e. 6 days critical free rather than 20. We'd not have gone 10 or 20 days with zero open critical issues, but we could have gone eight weeks without finding any new critical issues.
Comment #37
effulgentsia commentedRe #31.1, are you saying something like:
- RC1: tagged with 0 known criticals.
- RC2: tagged with 0 known regressions relative to RC1.
- But, 2 days later we discover RC2 actually has a regression.
- 12 days later, HEAD has fixed some critical issues discovered with RC1 and RC2, but has not yet fixed the regression.
- Do we tag an RC3 or hold it up on fixing that regression?
So long as HEAD isn't any worse than RC2, I don't see why we shouldn't tag the RC3. If people are already on RC2, there's no reason for them not to update to RC3. If people stuck to RC1 because before they were about to update to RC2 they had already learned about the regression, then they can continue to stay on RC1 so long as RC3's release notes mention that the regression discovered in RC2 is not yet fixed.
Or, does tagging RC3 without having fixed the regression make it harder to then fix the regression, because now there's 2 RCs out in the wild with a regression relative to RC1. I'm not clear on why that would matter, but if the nature of the regression is that it does matter, then yes, in that case, it would make sense to hold up the RC3.
Meanwhile, I don't get what "no regression relative to RC1" accomplishes. Consider the above scenario but with each RC incremented by one. In other words, RC3 has a regression relative to RC2, but not to RC1. And then we need to make the call on whether to tag RC4 or not. RC1 is irrelevant at that point. So the most conservative thing would be to say "no known regressions relative to any RC", but I don't think we need to be that conservative with all regressions, only nasty ones that would complicate future upgrade paths.
Comment #38
catch@effulgentsia Let's assume we're talking about regressions that introduced new critical issues, not a bad interface string or something.
I think this would be confusing - as well as the fact that lots of people don't read release notes.
In a situation like this I'd want to unpublish RC2 (so people don't try to use it for new sites if it's an issue that affects new sites) then roll back the patch that introduced the regression before releasing RC3.
Comment #39
chx commentedThere are more thoughts swirling in my head , one of the issues (irrelevant which, people know) mostly just sparkles writing them down.
Now, let's talk of release. There are two routes ahead: PHP 5 and PHP 6. PHP 6 was poised to be a revolution, had books printed then stalled and never was released. PHP 5.0 was a horrible release, PHP 5.0.5 broke things so badly several people didn't touch PHP 5 until GoPHP years and years later made them. PHP 5.2 was a pleasant release.
I do not think the community can fix the mess created in any reasonable time frame. klausi had the right idea: pick a date, I recommend two weeks after DrupalCon Los Angeles and release Drupal 8.0.0. Deadline makes things happen. We know that. We can rally people much better around a big days and minutes countdown visible on every page than an obscure critical countdown marker that can not reach zero.
Comment #40
bojanz commented+1 for the "pick a date" idea.
My personal and non-objective feeling has been that the community (contributor, shop, client) enthusiasm has been severerely drained,
and the ever-shifting release date estimations are a large factor in that.
Comment #41
gábor hojtsyI created a Drupal 8 translation status page at https://localize.drupal.org/translate/drupal8 and posted this info:
It would be great to get confirmation that its correct, before I promote it more widely. I did not use the exact 10 day after the RC1 because that seems to be in discussion, so I put in a more relaxed "couple weeks", but we can of course update that or anything else.
Comment #42
gábor hojtsyGot confirmation from @catch over IRC that that text is correct. Posted https://localize.drupal.org/node/63758.
Comment #43
xjmJust a note, I think this understates the chance of coding standards fixes causing regressions. The patches tend to be easy to create but challenging to review, and things like renaming internal variables/protected properties or whatnot could actually easily break things. Now, I guess they're unlikely to introduce critical regressions, but nonetheless. I also think this should only take effect if we have testbot support for coding standards by that point, and we'd managed the issues in a controlled way (scoped cleanup for one rule, plus test for said rule). (Outside of docs that is. We can always make docs better.) It goes without saying, but I'll just say it for clarity: these also can't be any cleanups that introduce BC breaks (e.g. renaming classes or public methods).
To close this issue, I guess we need:
I've started doodling some diagrams.
Comment #44
xjmTrying to make it diagrammable...
Do we actually need to do that? Why not just start a 72-hour code freeze for said patch marination, and then tag the RC once it's marinated? Seems tagging a release with a life expectancy of 72h is overhead/confusion/mixed messaging.
Comment #45
xjmDoes our infrastructure support this?
Comment #46
xjmDraft diagram of the RC phase based only on the issue summary: https://docs.google.com/a/acquia.com/drawings/d/121Rx3G6l7deGSadOP5mVJZj...
I started to make an 8.0.0 -> 8.1.0 diagram but the updated summary doesn't look like what we discussed; it's got conditionals about 10 days? I think it's more important to keep a regular month patch release schedule and six-month minor release schedule, for predictability. Other software projects don't change their minor release schedules based on outstanding issues, do they? Barring something nuclear-level like a highly critical SA or data-destroying bug, I think we should keep to the strict release schedule. Critical issues should trigger feature freeze for minor versions though (which is not mentioned in the summary).
Comment #47
xjmI made the diagram based on the current summary, but the current summary is also more conservative than this from #29 (which had consensus at the time):
If we had only one critical bug in 2 weeks after we'd already gotten to 0 critical issues, and there had furthermore been only a low rate of incoming criticals leading up to the first RC, I would still be comfortable setting a release date.
Comment #48
catchI think it makes sense in a whack-a-mole situation, not if the rate of incoming criticals has slowed by the time we get there.
Comment #49
catchOr instead of that:
We currently have a monthly-ish schedule for betas. We could just make the next beta release an RC instead. Then people going release to release still have a predictable schedule.
Comment #50
xjmThat could make sense, but what if the last critical is fixed (say) 5 days after the previous beta? Would we stay in a full code freeze for 3+ weeks with no stable for folks to test? Maybe we could wait for the next "half beta" date if it's in the first half of the month? (In keeping with the 2-week RC schedule.)
Comment #51
xjmSo here is an alternate diagram, based on the earlier consensus from #29 and some related discussion @catch, @webchick, @effulgentsia, and I had over the past two weeks:
https://docs.google.com/a/acquia.com/drawings/d/1tELtDLgCIoLlMlKvFYaDYAM...
Changes from the previous diagram:
Comment #52
xjm(moving to previous comment)
Comment #53
xjmComment #54
effulgentsia commentedFWIW, #51 looks good to me.
Comment #55
xjmPer discussion with @webchick (as well as earlier discussion with @catch). Clarifying that it's not only the number of incoming criticals that affect when we set a release date, it's also their nature (i.e. what kinds of criticals they are). Some critical things are an easy hotfix; others have much bigger implications. So we will be taking that into account as well.
Comment #56
webchickOk, so if I'm reading #51 properly (using totally arbitrary and made up dates for the purpose of clarity) this is probably the "best-case scenario":
(BY THE WAY, DID I MENTION THESE ARE TOTALLY ARBITRARY AND MADE UP DATES BECAUSE THEY ARE! :P)
Conversely, in a worst-case scenario, we keep finding really nasty bugs and push release date out for months and months. Or, we find a horrible security problem the day before we plan to release, and have to release anyway. Or etc.
Some feedback:
In the end, I think on the whole I'm +1 here, but would love to thinker on it some more.
Comment #57
xjmSo the one mitigation to #56 is regarding:
I think the asterisk on the chart is important here for keeping people engaged -- committing non-disruptive majors, docs improvements, etc. Still, 8.0.0 to 8.0.1 will end up being a lot of RTBC catch-up.
Comment #58
xjmUpdated the summary based on the most recent proposal. The one part I didn't update is the part about 8.1.x, which we haven't discussed and I'm not sure about, but I also think that doesn't need to block the more urgent RC policy.
Comment #59
xjmComment #60
xjmI moved the discussion of 8.1.x to its own issue, since it shouldn't block us finalizing the RC phase info, but does need to be discussed in more detail in the coming months: #2455081: [policy, no patch] 8.1.x release schedule
Comment #61
xjmComment #62
alexpottI really like the predictable nature of the 2 week release candidates - this means that anyone building a site can easily plan when to move to the next RC. And releasing RC's with known and announced criticals is a good idea because it is likely that the previous RC will have the critical and therefore people can thinking about how mitigate its effects until the bugfix lands.
I'm also in favour of announcing a release date and sticking to it.
The plan looks good to me.
Comment #63
catchGetting in sync with the 7.x release windows prior to an RC is sensible - we can do either 5 or 3 weeks in between a couple of betas to sync up I guess. I'd want to keep monthly for the betas, just have it come out the same day as the 7.x window.
The only caveat to this is if we release the same day as the 7.x security window, that means we'll have public security issues in the 8.x queue and a full month before they're in the next beta. So I'd be tempted to match the bugfix release instead - then that gives us two weeks to get security issues fixed before the next beta after a security window, which for straightforward ones is enough time. Right now this doesn't matter much since we have plenty of publicly disclosed security issues, but it will matter more once the upgrade path is supported and when we're very close to RC.
There's definitely some risk here. A lot of critical issues start life as major bugs (because either people don't quite realise the impact when they open them, or they don't want to 'cry wolf' every time they find a bug, or the critical-ness is determined by the changes necessary to fix the issue for what might otherwise have got left as a major, or they turn out to fix some other critical issue that's been open for two years with no solution by making it disappear).
Also I think this ties in with the RTBC backlog - both issues point to us needing to triage the major queue before we hit release candidate, this would allow us to identify:
1. Issues that we'd accept during RC (RC target)
2. Issues that would be postponed from RC to 8.0.1.
3. Issues that would be postponed from RC to 8.1.x
4. Issues that would be postponed from RC to 9.x (API change and no obvious BC layer)
5. Hidden critical issues that should block the RC
6. Duplicates, won't fix, needs more info etc. etc.
If we do that, then there's less chance of newly found critical issues 'hiding' in the major queue. And it also makes it a lot clearer which issues can get committed during RC or an early patch version vs. being punted 6 months or 2+ years later if they're not done before RC. The earlier that is clear, then at least the RTBC backlog during release candidate might be a bit shorter than it otherwise would be, or have less frustrating issues in it.
Comment #64
xjmAgreed about the major triage. My hope is to rally some folks to start work on that around DrupalCon LA.
Comment #65
xjmComment #66
dries commentedWhile I was involved with offline discussions around this, I wanted to publicly state my support for this proposal and sign off on it. As long we're good at separating release blocking bugs from non-release blocking bugs, this approach should work well. The predictable release candidate schedule will be helpful for site builders and will give us confidence in the stability of 8.0.0. I'm also in favor of announcing a release date and sticking to it.
PS: Angie's example in #56 was very helpful. If you haven't read that, I recommend you do.
Comment #67
xjmI've updated the release cycle page to incorporate this information at:
https://www.drupal.org/core/release-cycle#rc
I also added a stub section about 8.0.0 and semver while I was at it.
Thanks everyone for all the feedback on this discussion!