[policy, no patch] Update patch backporting workflow to be compatible with semantic versioning

So apart from the git history, there is a big trade-off if we switch from cherry-picking backwards to merging forwards, in the cases where there are merge conflicts.

So if we end up with a merge conflict, there's two ways to handle that:
- committer handles the merge conflict
- committer sets the issue to 'patch to be ported' and we use 7.x-style workflow with it.

When we commit to 8.1.x and cherry-pcik to 8.0.x, then if either of the above doesn't happen, then the very worst case is that the fix doesn't get into a stable release until 8.1.0. For all but the most critical patches, this is actually fine (it's definitely better than the other thousands of issues in the queue that aren't committed anywhere yet).

When we commit to 8.0.x and merge to 8.1.x, then on a merge conflict, we introduce technical debt into the next release, which then has to be resolved to prevent a regression. This can mean additional overhead for committers resolving merge conflicts themselves, or the possibility that issues sit at patch to be ported for months on end, and never make it into a future branch.

One possible way to avoid the latter situation, would be for issues where that can't be cleanly merged to 8.0.x, just commit them to 8.1.x in the first place, then use the old 7.x workflow with patch to be ported for those cases.

That would be a hybrid approach then - merges for anything straightforward, but manual backporting for anything that's not - but never committing something to 8.0.x and not 8.1.x for longer than a few minutes at a time.

I'm not sure why this is an issue. The cherry pick workflow isn't really "going against" the Git tree model. It's just a different use of the tree model than you expected.

                                                                                  8.2.x
                                          +-----+-------+--+--+--+------+-------+------>
                                          |
                                          |
                                          |                       8.1.x
        +---+----------+----+-----+----+--+-----+-------+--------+------>
        |
        |
        |                      8.0.x
+-------+---+----+-----+----------+---->

For the sake of discussion, let's say that the

+

signs that align vertically are the same commit. There's no reason that those need to have the same commit hash. The workflow that we have right now takes into account that some patches won't be merged back from 8.1.x to 8.0.x and some will. Fixes should go into the most recent version, and then step down through versions until the fix is no longer applicable or the version is no longer maintained. We have tooling to track this workflow (the issue queue), and actually *using* this workflow is very straightforward, and it absolves core committers of any responsibility for dealing with changes that need to happen when going from version n to version n-1 to make the change apply properly (whereas asking them to do a merge from 8.0.x into 8.1.x will require them to resolve the conflict themselves).

Merges of 8.0.x into 8.1.x could work, I guess, but it will also get hard to track what fixes have been merged into 8.1.x and what hasn't, especially when the way that you do something in 8.1.x and the way that you do something in 8.0.x are different in some way (due to code cleanup or whatever). You also introduce a bunch of extra merge commits, which makes the history harder to read (I wouldn't normally bring this up, but since one of the primary reasons for changing the workflow is to make the history more useful, then it should also be readable, no?). That also presumes that patches will go into version n-1 first, and then get forward ported to version n, which I think is not a good situation.

I don't see any reason why we need to treat the scenario of 8.0.x vs 8.1.x any differently than we treat 7.x vs 8.0.x. It's a different version of Drupal, and the backport policy/workflow that we've been using for years works just fine, and doesn't involve any kind of gymnastics with the git cli.

If I'm missing something or understanding the importance of changing this workflow, I'd love to hear more clarification on how/why.

To respond to your specific workflow complaint/suggestion:

* `git log 8.0.x..8.1.x`

The --cherry-mark and --cherry-pick flags are designed to make this work. `git log --cherry 8.0.x..8.1.x` should do exactly what you want.

* Apply to 8.0.x first and merge to 8.1.x

I really don't like this. It adds an extra merge commit potentially for every issue, which is not useful at all. It's also a much more simple conceptual model to tell new contributors "Work on the latest dev version first. Once the patch is committed there, it may be backported depending on disruption and BC and other factors" (as opposed to "work on the branch that has the most recent stable/public release, and then when the patch lands, it'll also be merged into version next if it applies properly, but if not, you'll have to update it for the next version too, and then once that's done, it might be backported to the previous version").

There's an extra consideration here, which is that there's no point in time where we support stable releases based off 8.0.x and 8.1.x at the same time.

8.0.x is essentially a release branch at the moment.

Once 8.1.0 is tagged, there will be no further commits to 8.0.x (except for something exceptional like a paper bag).

At that point, we only have 8.1.x and 7.x to worry about, and 8.1.x has the full history of the changes since 7.x.

The typical merge conflicts we will be dealing with will usually not be very hard since we should only be committing bugfixes.

That's not really the case, in patch releases we're also committing at least documentation and test coverage changes. Test coverage refactoring vs. additions for bugfixes vs. additions for features could very likely result in merge conflicts.

Maybe if we use the common terminology instead of our drupalisms you'll recognize it for what it is:

8.0.x: This is the current stable branch (the "master"). It only receives bugfixes ("hotfixes"), no new features. Every bugfix gets merged back to 8.1.x, creating a merge commit in 8.1.x that contains any conflict resolution.
8.1.x: This is the future release that is in development (develop). New features are committed here.

Um? develop is the default branch in Git flow. New crap gets merged into develop, and then eventually, develop gets merged into master for a release. Hotfixes are usually just another branch that gets merged into both develop and master.

What you are describing here is the classic CVS workflow that was popular 20 years ago. Back in those days we were dealing with code on a commit-per-commit level, and those version control system were indeed designed to deal with patches.

Our bad experience with git merge conflicts is mostly because of the endless patch rerolls that all of us have done over the years. This is a different scenario than merging two branches. Git is a lot better at merging branches than patches.

Unfortunately, we've changed our tool, but not our methodology (yet). We don't use a merge workflow for our stuff. It would be one thing if every single issue had a branch somewhere that core committers could merge into 8.0.x and 8.1.x (and completely predicated on whether or not that would even be an acceptable solution that the committers could get on board with), but we don't have that. We have patches, and until issue workspaces are a thing, that's what we have to work with. Pretending that's not the case is neither productive nor moving in the direction of fixing our workflow. It's the opposite:

Since git is distributed we can maybe "outsource" the very complex merges to the community, anyone with a d.o or github account and 10 minutes of spare time can clone the repository, do the merge and publish the result somewhere. We would need to run the tests somehow before merging this back into the canonical repo.

To me, this sounds like a massive f'ing headache for everyone involved.

I ask again: what is the problem with git log --cherry 8.0.x..8.1.x? We can argue all day (and honestly, probably agree) about the way that we should be doing things, but reality is getting in the way. We have to use the tools we have right now, and until new tools are available, I can't see how your proposed workflow is going to scale to a community the size of Drupal's.

We started using +x to automatically add cross-referencing between the commits to each branch, but I have a suspicion that might be breaking git cherry - since the commit messages are different.

Pretty sure git cherry works on the contents of the commit diff, rather than any metadata, but I could be wrong.

Also, if that breaks git cherry, maybe just stop doing that? :)

Well at least for me git cherry isn't successfully filtering out the cherry-picks. I don't have a patch to commit immediately to test the +x/commit message theory.

If I'm on 8.1.x, git cherry 8.0.x seems to output the right thing.

So the command that works for me is git log --cherry-pick 8.0.x...8.1.x. The command in #11 doesn't contain enough dots :)

Maybe we could less the merge commits if we only merge patch releases into the next minor releases when we tag? That would mean less merge commits and features get a longer period of stability to work against. The potential downsides of this are: that after every patch release we might have to spend time sorting out the conflicts; and we might get less testing of the new features going into the next minor.

The potential downsides of this are: that after every patch release we might have to spend time sorting out the conflicts; and we might get less testing of the new features going into the next minor.

To me these are both quite big downsides.

after every patch release we might have to spend time sorting out the conflicts; and we might get less testing of the new features going into the next minor.

I see these as pretty big disadvantages too. I don't think merging our release branches all the time is really a great solution. When we have issue work spaces, we can merge an issue branch into whatever release branch(es) it applies to and then this won't be a problem anymore, right? We'll have to switch to merging issue branches by necessity. Until then, though, merging 8.0.x into 8.1.x or vice versa adds complexity for releases and more potential for headaches for core committers.

Also, I still don't see much value in tooling that can figure out the difference between 8.0.x and 8.1.x. Outside of core development, are there any use cases where that's actually a requirement? For core dev work, I don't think it's a big deal to tell people to use alexpott's command in #18 if they really want to compare branches.

First of all, I agree it makes most sense to turn things around; we start by solving an issue for the oldest supported version, and "port it forward". The reasoning is mostly covered in the issue summary (while it is still supported, D7 could be handled as an exception, since the times where a fix would cleanly carry over - for the most part - to D8 and beyond would be few and far between). Any issue might potentially need to be resolved in three versions of Drupal; the current LTS version, the current minor version, and the next minor version. Or, possibly also the next major version? I am not sure what the plans say about continuing to develop minor releases while development on the next major release has started (I would expect that's not in the plans, because that would involve forward-porting features).

Basics

I think we need a policy where a resolution of an issue would consist of a "bundle" (workspace? sorry, I am not entirely clear on the terminology) of git branches that should be merged into the various project branches that need the fix. These should merge cleanly, i.e. resolving any merge-conflicts should be part of the resolution. Applying the fixes is (obviously) the first step in an automated testing procedure, which can simply fail if the fixes do not merge cleanly (it would fail anyway, because of the broken code due to conflict indicators, but we could probably make that a little more friendly).

How these branches are identified would be a subject of further debate. E.g. a specific naming scheme for tags/branches, or possibly a pull-request type workflow where a user posts a new pull request to an issue, where they could pick tags, branches and/or specific commits from their personal workspace for each of the supported project versions. Anyway, like I said, identifying the commits to merge is not the main point right now. Let's assume the pull request scenario for now, as it makes talking about this more practical.

Ideal situation

Ideally (and this is pretty much a pipe dream while D7 is still in the picture), an issue would be resolved for the oldest supported Drupal version first. If the developer is fairly confident that fix will just merge to all other required versions, he could post a pull request with just that specific commit as the fix for all versions. The automated test process would come along and try to apply the identified fix commits to the current project branches (LTS, stable minor release, next minor release). It would then run tests to ensure all is well.

Merge conflicts

When the fix for the oldest version does not apply cleanly, someone (either the original developer or someone else from the community) would take the original fix and merge in the next-oldest release that the fix needs to support. They would resolve the merge conflicts, at which point they now have a branch that will apply cleanly to that next-oldest version. There's probably a pretty good chance that branch will also apply to the next-oldest release after that, but if not, rinse and repeat.

Backport as fallback

In some cases (OK, while D7 is still around, in most cases), we will probably still have the same situation as we've had in the past, where chances of merging a change to an old release into the latest release is pretty much impossible. In that case, we can fall back to our old workflow, where we start solving the issue for the current stable version (and handle merge conflicts with the next minor release), and later backport it to the LTS version.

Stuff to sort out

There's some stuff to sort out here. For example, there are no assumptions baked in here about how exactly the fixes for the various supported versions come to be. They could be based off the same commits, with merging in the relevant project branches (preferable, I think), but there is nothing in the process that prevents each fix branch being a unique solution, or a cherry-picked version of the original fix. This is actually a good thing while we need to support D7 as well (but there might be situations in the future as well, where trying to merge in the fix for an older version only confuses the situation, because the target version is just too different). But we will need guidelines about the preferred way of working here. Also, obviously some tooling will need to be created to author and handle pull requests (but that is probably part of the other issue, #2488266: [META] Improve Git workflow on Drupal.org by implementing issue workspaces).

It has been almost 10 years since there was discussion here. Is there need for discussion or shall we continue with the current workflow?

If we don't receive additional information to help with the issue, it may be closed after three months.

Thanks!

The two changes since this was opened are we have the 11.x 'fake main' branch open, which always gets the first commit, then release branches for minor branches, and that we keep the previous major branch open longer.

The 11.x/main branch never changes as a target (except maybe once more when we change to actually use 'main'). This massively reduces churn in MR targets and makes long running issues (anything longer than four months) much less work to manage.

There are a _lot_ of commits that don't cleanly cherry pick even from e.g. 11.x to 11.1.x and this isn't because the bugs change lots of code but because other issues have - test refactors, OOP hook conversions, coding standards changes etc. and additionally there are a lot of considerations about which issues get backported or not - bug fixes with API changes or upgrade paths generally won't get backported at all.

Expecting people to figure which branch to target for the initial MR is unreasonable IMO, it is almost impossible to tell when starting work on a bugfix whether it will land in 10.4.x, 10.5.x, 11.1.x or 11.x and having to maintain multiple MRs or change the target around constantly would add loads of work.

So I think this should be won't fix.

@catch, thank you for a comprehensive comment.

Based on catch's comment, which I also agree with, I am closing this issue as recommended.

If there is more to do here, then either re-open the issue or open a new issue and reference this one. If the choice is to use this issue then add a comment change make sure to change the issue status to 'Active'.