Release branches for core patch releases/release candidates

afaik it's not possible to have branch-level testing on x.x.x-suffix branches.

We can test any Branch/Tag in the repo, and should be able to test any Hash or Ref we want.

We open an 8.0.0-release branch.

- all changes eligible and RTBC for 8.0.x get merged to 8.0.x

- all release blockers and RC-eligible changes are cherry-picked to 8.0.0-release

- all release candidates and 8.0.0 are tagged from the 8.0.0-release branch

The problem with this I think is that it's the wrong way around. The criticals and RC targets should not have to worry about chasing HEAD; part of the point of freezing commits is to ensure they go in as quickly as possible and without disrupting other stuff or being disrupted. I think it could only work if we commit to 8.0.0-release first and merge it into 8.0.x-dev, but then we still have to manage merge conflicts or revert patches that don't merge, and reopen those issues to be updated. Ah no, that doesn't work either, because the issues should still be filed against 8.0.x-dev.

I also don't want to commit ourselves (pun!) to committing lots of RTBC issues to 8.0.x-dev during the RC phase. To me, that also interferes with the goal of tight focus during RC.

Unless we explicitly added 8.0.0-release to the d.o version field. Then we could file RC targets and criticals against that branch, and specify that all other issues were filed against dev, I guess.

Still carries the other concerns though.

I don't think this is (currently) supported on Drupal.org. Project issue file test module tests against branches with release nodes; Project release & version control project release modules won't recognize the branch name as following the standard.

Following the pattern of using x in branch names, I think it would make sense to call the potential branch 8.0.0-x. The bit after the - is called "extra"; extra can be anything. (rc/beta/alpha/unstable are special extra components, they get put in that order when sorting.)

Reading the code, there might not actually be something technically stopping a 8.0.0-x release. It is worth trying out on staging, if that will work for core. (Or even on production with a contrib module, the equivalent would be 8.x-1.0-x.)

Is there a staging environment or whatever where we could test out #8? For example, a branch named "8.0.0."

Just to say this would indeed be really awesome from a release management POV. We're likely to end up with a 6+ page RTBC queue backlog (or worse, drive off all of the core contributors we have now) without something like this during RC.

staging should be ready, but it is under-tested. My recommendation for testing was 8.0.0-x, would that work for core? That is much more likely to work without code changes.

@drumm just helped me test this on staging, but unfortunately 8.0.0-x doesn't work. It won't show up in the selection to create a release against. :\

Line 111 here is the problem: http://cgit.drupalcode.org/drupalorg/tree/drupalorg_versioncontrol/plugi.... It specifically locks down to digit.digit."x"

Update: "8.00.x" passes the regex. EVIL GRIN. https://staging.devdrupal.org/node/2580084

One nice thing about this too is it allows for more granular test configuration; for example, we could choose to test PostgreSQL/SQLite/PHP7 on branch + commit for the "RC" branch instead of nightly, without driving up the testbot bill exponentially:

The branch tests show up as a separate heading:

Also confirmed that the "- 8.x issues -" filter picks issues filed against this branch up, e.g. https://staging.devdrupal.org/node/2454653. DrupalCI kicks off too (once you've configured the branch for tests at https://staging.devdrupal.org/node/3060/qa).

You do have to watch it a bit because the "8.00.x" and "8.0.x" options are pretty hard to tell apart in a quick glance at the selection list:

...but overall these seem like solvable problems.

So, basically, I think we can do this with the existing tools and no d.o changes. The question I guess is do we want to. It does introduce extra comimt/review overhead, and we risk forgetting to cherry-pick certain things.

(I would say yes, but curious what other committers think)

I cannot imagine successfully explaining the differences between 8.00.x and 8.0.x to contributors. I expect it'd be a mess. We'd only have to deal with it for a couple months I guess...

But I still don't get what the process would be. Something like:
1. RC targets and criticals get filed against 8.00.x, thereby disappearing from everyone's existing bookmarks.
2. When a patch is ready for 8.00.x, we commit it to that branch. We then try to cherry-pick them from 8.00.x to 8.0.x, reverting other 8.0.x work mercilessly if it conflicts.
3. Other issues proceed as normal against 8.0.x if they meet our criteria for patch releases, and are committed to 8.0.x normally, aside from the "merciless revert "thing.
4. Stuff that we have agreed is not eligible for the early Patrick leases gets explicitly postponed to 8.1.x.

Also, if we do do this, I propose that we do not start until after RC2.

I dunno, I don't find it that confusing. Either way the X means "this is variable." (8.0.0-rc1, 8.0.0-rc2 vs. 8.0.1, 8.0.2, etc.)

Personally, all of my bookmarks point to the general "- 8.x issues -" filter, and this branch name means they do show up under there. But either way, all bookmarks pointing to 8.0.x are going to break at some point soon, so that's not really a deal-breaker for me.

Otherwise, 1, 2, 3 I think is right.

4. I have no idea what a "Patrick leases" is (assume that's Siri's version of "patch releases"? :)), but otherwise that sounds right too. And then whenever we choose to start committing things to 8.1.x, we'd have potentially 3 places it could go (but most likely won't open that branch during RC). And then 2, 3 repeats, but for 8.1.x instead.

One thing I haven't seen explained here yet, is how we'll get those two branches back together again.

So, in a nutshell, this is what the current proposal is:

• 8.0.x only receives rc targets
• 8.0.0.x receives both rc targets and other patches

That means that the rc target committed to both 8.0.x and 8.0.0.x will have different commit hashes. That will make things quite hard to follow.

But… it sounds like eventually all commits will be cherry-picked back onto 8.0.x, in the same order. So basically 8.0.0.x is then our "super RTBC queue": RTBC patches that are already committed and can already be tested together with all other patches, and therefore are more reliable, and therefore "more RTBC" than other patches. Which, if true, makes 8.0.0.x a kind of "throwaway" branch, whose commit history (and commit hashes) don't matter.

If this interpretation is correct, then my concern about commit hashes being different doesn't matter.

Firstly: Ask the security team do whatever they say. :-)

While this helps to avoid paper-bag releases, it also means that we arbitrarily don't commit things at certain points during the month, and it results in an RTBC queue backlog which then has to be worked through.

This is a feature, not a bug. More like this, please. Commit freeze times are good.

Make it explicit, so that everyone understands that this is the process. During RC there are the two week cycles, but during normal release time this should be, say, a week a month.

This way everyone gets reviewed, which I think is the biggest opportunity to reduce wasted time and effort on everyone's part.

With patch releases, we may also want to let certain changes sit in trunk for more than 4 weeks before they go out into a patch release, that's simply not possible if we can only commit to a single branch then tag off it every four weeks.

Well, the reason this is needed is to minimize rerolls, right? Everyone knows the patch releases are a thing. So we only work on one release at a time, and any grand designs for the next minor release has to reroll.

That's the only process that makes any sense, given the patch-based workflow here. Otherwise you end up with the semi-ridiculous 8.00.000.0.1.01.x branch naming scheme.

Even then the release branch is disposable - once the release tag is cut it can be deleted.

If we go with a release node for testing, that makes any branch more permanent. Not that people should be using the packaged branch, but deleting the release would take it out from under them.

We might want to go more in the feature branches direction and add the UI to Drupal.org for testing effectively* arbitrary branches.

* In practice there might be rules for tested branch names, like a feature branch starts with an issue number, 2576141-rc-branch, to help keep the UI together, or not.

The regex is linked in #11. IMO 8.0.1-dev won't pass. You can play around on https://staging.devdrupal.org though if you'd like (but drumm needs to be around since some things need manual kicking).

I didn't see this earlier but this workflow using cherry-picks has caused the 8.0.x and 8.1.x to diverge. This has broken the tree model of the repository.

Everything that is committed to 8.0.x should also be part of 8.1.x, so in the repository's tree model the 8.0.x branch should be an ancestor of the 8.1.x branch. By using cherry-picks this is not the case, instead every commit to 8.0.x exists as a different SHA in 8.1.x, essentially duplicating these commits.

This breaks a whole bunch of git commands, low level as well as high level. We cannot trust the git log any more.

We discussed this issue on a call with the DA. It seems like we no longer need this, at least right now. We're getting what we need out of this by committing to 8.1.x and 8.0.x at the same time. Catch wasn't on the call though, so not marking "won't fix," but lowering priority.

#27 seems like it needs its own discussion if it doesn't have one already, since it's more about Git workflow than infra support for multiple branches.

@alexpott brought this up again last week or so as an idea to avoid having to backport 30-some patch-eligible issues after a minor release.

I think if we had an RC branch for a minor, we would ignore it in the issue UI. (We'd still need to temporarily add a release node for DrupalCI integration, unfortunately.) The workflow would be like what we do for security releases, except that the throwaway branch is maintained on d.o instead of just being local. So it would be something like:

• All issues are filed against 8.3.x
• In VCS and DrupalCI only we use 8.3.0-release, or whatever hack we use to work around the regex.
• Each time there is a critical, we commit it to 8.3.0-release, and then merge 8.3.0-release into 8.3.x and push both.
• If something in 8.3.x has a merge conflict, we revert the conflicting thing.
• When it comes time to release, we tag the tip of 8.3.0-release, then merge it into 8.3.x one last time and push the tag. After that, we disable the release node for 8.3.0-release so it goes away from the UI again.

Edit: I should mention that I don't actually have a problem with backporting 30-odd issues after release, but both catch and alexpott seemed to find it frustrating.

I guess that still doesn't avoid the problem of the fact that people will have sensibly rolled their criticals against 8.3.x, not 8.3.0-release. So we might inadvertently hurt ourselves there during RC.

With rc branches we probably have problems with automated test coverage.

As it turned out doing the cherry-picks on the days after 8.2.1 was not that bad. I guess the solution we followed for the 8.2.0 release period works and at least does not add much extra burden on anyone. The "patches to be ported" queue felt like a big mental burden and yet another queue to care about - but it is of a defined length and does have an end (unlike the rtbc queue). People's patches are in 8.3.x so their work will get done at defined point in the future - even if it does not land in 8.2.x.

For DrupalCI - not sure if this will solve it, but #2817783: Add per-issue branch testing is in my current sprint at the Drupal Association. (Sorry I didn’t get that issue filed earlier.) That plans to make branches like 2576141-8.3.x-release, {issue nid}-{any arbitrary text}, show the testing results on the issue page.

I think a holistic summary of the development workflow would be helpful here so we can get a better understanding of what we need to support.

Previously I had outlined some potential 'code targets' in the DrupalCI testing policy framework: https://www.drupal.org/drupalorg/docs/drupal-ci/drupalorg-testing-policy... and their labels, I'll use those labels in this comment.

I'll attempt to summarize what I *think* is happening:

There are two "Phases" to our dev cycle: 'standard', and 'prerelease'. My understanding is that if you pick an arbitrary point on this timeline (https://www.drupal.org/core/release-cycle-overview#minor), and draw a vertical line and cross two bars, its 'standard development phase', If you cross three, you are in 'prerelease phase'

During 'standard development phase', which we're in now, we've have three development branches, Major dev(9.x), Minor dev(8.3.x), and Patch dev(8.2.x). We're ignoring Major for the time being. During this phase all changes are filed against Minor dev(8.3.x) and are cherry picked back to the Patch dev(8.2.x) branch if it is a security or bugfix. For the sake of argument, we'll call this phase the "8.3 Standard Phase"

During the next *prerelease* development, we will have four eligible locations for changes: Major dev(9.x)Minor dev(8.4.x), Prerelease dev(8.3.x), Patch dev(8.2.x). For the sake of argument, we'll call this the "8.3 Prerelease Phase" (again we're ignoring Major).

This is where Im unclear as to the policy during this phase: do all changes get filed against Minor dev(8.4.x), and if they are security/bugfixes they get cherry picked back to Prerelease dev(8.3.x) and Patch dev(8.2.x)? What is the nature of the changes that need backporting after a release is cut? In this case are they changes that go immediately into Patch dev(8.3.x) for the "8.4 Standard Phase"?

If so, why weren't they eligible to be moved into Prerelease dev(8.3.x) during "8.3 Prerelease Phase" in the first place?

I think understanding that would make it easier for us to fix the options in the UI (especially if we build the workflow around labels, and not version numbers) and ensure testing is doing the right things.

As an aside, I *strongly* discourage any solution that involves hacking the version numbers. As soon as we introduce exceptions to our semver scheme, we'll have to handle those exceptions all over our codebase - everything else wont 'just work', it'd be a complicated thing that would introduce subtle bugs on the infra side. It will be much, much better for us to provide exactly what you need.

At the moment the criteria for what can go into a release candidate is stricter than the criteria for what can go into a patch release

Couple questions about that:

1. At what point does the criteria split? is it immediately when we go into beta of Prerelease phase? Or is it closer to the end of the Prerelease when certain bugs get pushed to the next point release? Is it when we go from beta->RC ?

2. Is the criteria for what can go in, and what gets pushed out, documented somewhere? Or can you briefly explain?