[META] Improve Git workflow on Drupal.org by implementing issue workspaces

I was just at the DrupalCon session for this and I love the idea. When you start coding this, I'd love to help out. Really looking forward to the in-line editor making commits. That will be so nice for sprints for people that want to submit doc changes without having to learn git! I also like the idea of a per-user branch in each issue. It will make diff'ing everyone else's contributions from the command line so much easier. For now, if they don't attach an interdiff, I have to create a new branch for that person to track their changes in, do a git apply of their patch and for subsequent merges, I have to roll back to 8.0.x, apply the new patch, merge it back into the branch I have tracking their commits. I can see where the "shared branch" approach could be useful for big issues where a bunch of people are working on the same branch and "passing the baton", but I'm really for having my own namespace no one can muck up without me merging their changes in! It will also be very nice to be able to easily diff other branches, or cherry-pick commits into my branch when I only want part of their changes.

Not at DrupalCon, but followed Webchick's live tweet of it and love the sound of the proposed solution. I've patched a few modules, and worked with module's which required patches and neither scenario has ever been the easiest. This sounds like it would make contributing patches, especially trivial ones, super easy.

I also like #3's comments about being able to cherry pick commits from other branches instead of having to outright merge their changes, that is a good idea.

I would love to second disasms enthusiasm and his offer to help. I'm very excited to hack on this. As a former proponent of moving to something like GitLab, I am now a convert. To help evangelize this, I'd like to put down my reasons for my conversion.

The biggest problem and change that would be associated with migrating to a service and/or self-hosted VCS solution is the distributed collaboration. Not to be confused with distributed version control (This solution is NOT eschewing the benefites of ditributed version contol). The best way to illustrate this, and similar to the way it was illustrated at the DrupalCon Los Angeles session, is to go through a worklfow common on a service like GitHub.

Imagine that Joe has an issue with Project X. Joe has to find the project's page on Github and create an issue. Joe then forks the project, does his work, and submits a pull request. Nancy comes to the issue and reviews Joe's code. Nancy has some comments and decides she wants to make a change. Nancy now has to fork the project or Joe's fork and submit a PR to his branch or create her own PR to the project. Now, Joe comes along and appreciates her ideas, but has some concerns with one of her changes. Where should Joe make his comments? He can add a comment to his issue, which may or may not have Nancy's commit associated with it. He could go to her pull request and make a comment. He could also make an inline comment on one of her commits. As the issue gains traction, a third and fourth person might join the discussion, they may create their own forks and comments. One can no longer just read the single canonical issue to bring themselves up to speed. Whose fork is canonical? Whose work has been abandoned? The problem can be exponentially difficult to solve. This is the ugly side of distributed collaboration/deliberation, admittedly pessimistic, but nonetheless an outcome that is all too probable.

On Drupal.org, we instead have a patch-based workflow that has encouraged and enabled us to work on a single issue in an ordered, single thread. Joe comments, adds a patch, Nancy comments, adds a patch, and Joe responds in the issue queue and only in that issue queue. Any following collaborators can come to the thread, read it chronologically, and add their contributions to the last patch or last agreed upon patch.

In short, the beauty of the d.o issue queue is that it imposes a chronological order in the collaborative process. It allows us to deliberate together and helps us make sense of everyone's work as a whole. We only have so much brainpower. On the other hand, the beauty of git is that it handles the unordered and distributed nature of code commits. Unlike patches, commits don't easily become "stale." And when they do, it's much more trivial to refresh them. This is the brilliance of a distributed VCS - it helps us reason about something inherently chaotic. The issue queue is how we reason about the meta discussion around that same code.

This proposal promises the best of both worlds. We can clone and work independently, we can increase the velocity of our contributions, and we can preserve all of the benefits we get from a single deliberative issue queue.

Shout-out to @gabesullice for a very succinct description of the exact problem with "forks of forks." :)

I love the sound of this. It's great to see how active this community always is in innovating for the sake of said community. +5 stars and then some!

I came to Drupal and stayed because of the community, and the way Drupal.org works. This proposed plan sounds like it will fully embody the spirit of what "is Drupal" and our collaborative effort.

I pinged one of our system administrators at Gaggle. He hasn't had a time to use the feature of Git namespaces, unfortunately. He did say it looks to be a better implementation of the git alternates feature that aimed to cut down on the disk space issues of forking (quickest resource I could find, an article on implementing it in GitLab.) Just for some extra information :)

Well, this would be totally fantastic.

first of all thanks for the mention in the illustration, it least my non-productive patch got to be a good example for how the collaborative workflow can look like :P

i appreciate there has been a lot of research into options, moving to github etc and can see that we want to embrace collaborative workflows so couldn't find a ready to use solution yet. still i'm concerned as we are inventing our own solution again and my feeling is that this is not a drupal-specific problem at all.

are we the first ones to use the git namespaces feature to create issue workspaces?

i understand that the current patch-based workflow is really out of date and we need to move forward, but i think we should also make sure to share our learnings with others interested in the same concepts. ideally this problem would be solved similarly on github or other popular platforms.

We are definitely inventing our own solution for drupal.org. Typically its best to avoid re-inventing existing tools, but as far as I know, we will be the first ones to utilize git namespaces in this manner, and therefore we'd be the first to implement this style of workflow.

In researching all the various git daemons, self hosted solutions, and hosted services, none of them offered a concept whereby an issue can be worked on collaboratively in this manner. All work was assumed to happen on one branch in a forked copy of the canonical repo, which then becomes a pull request or similar - limiting transparency on work in progress, and fragmenting the interface, history and conversations.

I agree that as we build, test, and beta and ultimately launch this workflow, there would be definitely be value for other projects. Our collaborative workflow has helped us build extremely powerful software, and other projects could benefit from that process.

If this works like we think it might, and adds the value we believe it will, then I foresee it becoming something that could spread to others tools. Since we'll be doing everything in public, and all of our source will be available, we would definitely encourage adoption and collaborate with others. But I strongly doubt that this would be solved at github or other popular platforms without first demonstrating it's value to our community, which Is why we believe we have to make it happen here, on drupal.org first before promoting this solution 'upstream'.

Personally, I think GitHub is way more familiar that d.o. and way more user friendly.

Also, fork of forks and having multiple places for comments - this is not a big deal at all. Check out those issues on d.o. with tens of patches and hundreds of comments and try to find the proper answer or solution - if people keen to dig through that than they would probably be able to handle comments on fork of a fork! And this is actually a minority of cases. On the other hand, think of how easy it is for a module maintainer to merge pull requests comparing to applying patches and how easy it would be to resubmit a pull request comparing to re-submitting a patch!

I know that this has been discussed for a couple of years now in different issues on d.o.

With all previous discussions in mind, can we actually create a vote for all community members? That would be a TRUE community-elected decision and no one would feel like their opinion is left out. The vote would state simple options: leave as is, GitHub, namespaces. The explanation behind each option can link to this issue so that everyone would understand the reasoning behind this.

And one last thing. Many other projects successfully use GitHub and it has proven to be a valuable and reliable tool. So why does Drupal has to stay on this island?

Here is a link to a video of Workspaces presentation at DrupalCon LA 2015
https://www.youtube.com/watch?v=B392kwHg_Rs

thanks for the feedback mixologic. looking forward to see this implemented soon :)

i personally don't think voting on leave as is, GitHub, namespaces makes much sense because we don't have a process for voting on decisions on d.o implemented so far. this would just derail the issue. the current best practice is to find consensus or escalate decisions. at the current point i don't see any strong arguments agains implementing namespaces

@dasjo
>> we don't have a process for voting on decisions on d.o implemented so far.
Well, this is not the first time I'm reading this kind of a reply on d.o.

There is no reason to not implement such voting process (which is quite simple thing to do keeping in mind that Drupal Association already origanizes polls and votes for other things). Especially, for a such important thing as development workflow - the heart of any process development process.

So, I'd rather see this issue 'derailed' until properly voted, than "just do it" because there is no process to ask for community opinion.

But hey, I'm just a community member and a simple developer - so cares, right?

@gabesullice
Your points about fork of a fork a valid, but in reality would people actually fork a fork to submit their changes in majority of cases?

By looking at the patches on d.o. for contrib modules, the average re-iteration of a single patch would be around 3-4 changes. I think, those changes would likely be done as comments on the commit itself, rather than a fork.

Please keep in mind that other projects use GitHub and it does not scare their users. If a user is proficient enough to apply a patch it should not be a problem to find a related issue in the module's main repo (Github actually shows that a repo is a fork of the original repo).

Also, I'm not sure that introducing workspaces will make it easy to submit patches as it looks like that the whole process would have to be the same 13ish steps.

@alex.designworks you've raised valid points. I just wanted to say that workspaces won't keep the same 13 steps to generating a patch.

Issue works spaces act as a branch. All people working on that branch have their own branches. Work is done by merging and pushing. Then a comitter merges the issue branch. Thats modern fit workflow, at least in my experience.

I don't like GitHub. I hate trying to contribute to projects on GitHub. I've tried working on other communities and feel its harder.

Reading an issue queue with 200 comments is difficult. But its better than reading an issue, to see there is a PR for it. But the conversation happened in the issue AFTER the PR. So you need to read both conversations and compare the results.

FWIW, we did vote on Drupal.org features a year or so back, and the vote between "move to GitHub" and "make our tools better" was basically exactly a 50/50 split. Nothing has materially changed to sway various peoples' opinions since then, and the DrupalCon presentation does a good job I think of explaining why moving to GitHub, right now at least, is not an option.

#2261945: [META] Drupal.org ideation tool is worth a read for some of the challenges in ideation/voting, which include: voices of "insiders" vastly outweighing those of actual users, difficulty in broadcasting the opportunity to vote (for example, the DA's elections get about 1% participation), past votes being inflexible to changes that happen "on the ground" to change circumstances, ideas coming up with no one passionate about working on them, ideas that grow popular without fully considering the consequences, etc.

If you have other thoughts on how we can improve the ideation process for Drupal.org, we'd love to hear them, but please comment over on that issue, not here.

To further highlight the "fork of a fork" problem with GitHub, and why I'm excited about issue workspaces as an alternative, I can use a real-world example from Drupal 7: the Bartik theme. Unfortunately, most of these forks have now been deleted since then, so https://github.com/jensimmons/Bartik/network no longer shows the full story, but I'll try and recreate it here (please excuse my crappy photoshop skills; also any inaccuracies, since it's been 5 years :P).

GitHub was chosen as the collaboration hub for Bartik since a) Drupal.org was on CVS at the time [puke!] and b) the in-place editing tools allowed contribution by designers who had no idea what version control was. Makes sense.

So https://github.com/jensimmons/Bartik was started and quickly amassed a group of 10 or so people working on it: dmitrig01, JohnAlbin, Jacine, and others. And that worked really well when jensimmons was around to quickly pull in proposed changes from others, and the network graph for the project basically looked like this:

And life was good. There was a clear "canonical" place to find the "official," latest and greatest code. There was a single "project owner" that everyone knew to coordinate with. And that coordinator was able to be responsive and quickly merge in proposed changes.

Then, jensimmons got a job in Beijing, and went off the grid for awhile. Which is totally fine, and expected in a predominantly volunteer-driven project; paying bills is good! :) Work continued on Bartik in the meantime, but without central coordination, in dmitrig01's fork, in JohnAlbin's fork, in Jacine's fork, etc. Soon the network graph started looking like this:

And life was pretty awful. The "canonical" project had stalled out, and was effectively no longer accepting pull requests. So everyone worked on their own, some knowing to coordinate with others, some not. It was impossible to know where the "latest and greatest" code was to point people to for testing. And many contributors outside of the "inner circle" missed the message that work was now happening elsewhere, so they were starting from code in the official project from several weeks/months back.

I call out this example not to pick on jensimmons or the people who worked on Bartik. I love all of those people, and their effort was amazing and we managed to get it in core regardless of these challenges. But I think this real-world example highlights very plainly a number of realities that we have in the Drupal community:

- The vast majority of work on Drupal.org is still done by volunteers, or people with a very limited "attention span" (a client is paying me to care about this this week, next week it'll be something else)
- As a result, the person who starts the work is very often not also the person who finishes the work (at least in larger projects/issues).
- And in those larger projects/issues, there are nearly always at least 2-3 people involved due to peer review requirement. In core issues, it can be as many as dozens.
- Those people do not always know each other. In fact, often don't. We get many "drive-by" contributors or people who come to a contribution sprint for the first time, etc.
- It therefore needs to be stupidly easy to find the "latest and greatest" version of the code (and discussion!) to work from, so contributors don't end up wasting their time.

Issue workspaces solve a lot of these issues, plus lets the community "level up" their knowledge of Git so that if a move to GitHub is ever in our future (their issue management tools would definitely not support this currently), it would be a much easier migration than today.

Thanks for #23. That comment is so great, that I just had to bookmark it for future reference. Thank you :)

When Flag 8.x was on github, we had at least one time like #22. Most often we had, "this issue is stalled because your PR needs to be rerolls plskthxbai?" Forking a PR fork feels weird and is intimidating for newcomers because of the backend magic that powers it.

First off - thanks to everyone who worked on this proposal. Technically, it seems solid, and the workflow that it creates would be pretty cool.

It seems like a lot of people are getting hung up on why we aren't just moving to Github, and the two reasons I've heard are the PR issue mentioned above and that the issue management on Github is sub-par.

When we discussed moving to Github a couple years ago, I seem to recall a couple of Github staff members saying that they'd be open to talking with us if the path to moving to Github is not as clear-cut as we'd like. It seems like the issue being described above is something that could occur in *any* project, not just Drupal. Would it be totally out of the question to reach out to them and see if they'd be willing to help us solve that problem (presumably by adding functionality to Github)? If Github simply will not work, then sure - let's do our own thing (and if we're going to do our own thing, then the proposal sounds like it'll do exactly what we want). But I think having a major open source project migrate all of their repos to Github might be a big enough deal that Github might work some magic for us.

That said, I'm not sure the PR thing is really as big of a deal as people are making it. If a PR is stale or whatever, just open another PR. It's annoying and the PR list might get messy after a while, but cleaning that up is something we could automate pretty easily with a PR naming convention (if issue #1234 is closed, then close all of the PRs for issue #1234). It just seems strange that we're going through so much effort and expense to build and maintain our own Git hosting service when Github will do what we need it to if we change our workflow a bit (and it might even do exactly what we need it to if we take the time to ask for it)

re: #26

A third objection is that we would be moving our entire lifeblood to a closed source company where we cannot directly control anything that happens. I realize that this falls under the "not invented here" category, but this is the lifeblood of Drupal. Yes, Github may work with us and everything is peachy for a couple of years but that doesn't mean it will always be. They could turn into a Sourceforge and start plastering ads all around the site (unlikely but possible) or decide to start charging unreasonable amounts of money etc. At the end of the day, they have a financial interest to make money. When a better competitor comes along and steals away their paying customers they will do what they have to to keep the money train alive.

IMO, putting our entire lifeblood into Github is risky. Other people have expressed this as well over the past years of discussing this, not just myself.

There is a lot to like about the proposal.

On Drupal.org a comment will be generated on behalf of the user, that contains their code changes as well as their commit messages.

I think that inline diffs and line by line code review are part of an MVP. The experience of working on a pull request is vital, not a "maybe someday" feature. If this expands the scope too greatly, then I question whether we have the funding to build and maintain this system over the long haul.

I wonder if the proposal here for protected branches per user is adding a lot of un-needed complexity.

I also don't see any discussion of how/when branches would be rebased on core or the relevant project. I find it's much more comprehensible to do that than to try to merge from the upstream project.

In general, these feels like an overly complicated MVP. I'd think a workspace + 1 common branch (with contributors creating more as needed) per issue would be a real MVP.

donquixote had a good suggestion at the talk that we should really only be dealing with tags not branches in terms of what commits in the namespace get exposed in the UI.

A 1:1 mapping of tags to "patches" would actually make a lot of sense since tags are immutable and I can always start a local branch from a tag to continue work.

Initially at least you might require people pushing to git to make a tag to post a "patch" to the the issue, and an actual patch would be applied to a detached head and make a tag rather than trying to apply it to a branch.

Thanks, @pwolanin.
My complete suggestion, as it shaped during the discussion, is like this:

One git namespace per project per user, across issues.
The respective user is allowed to push to this namespace, operate with branches, do what they want.

One git namespace per project per issue, across users.
No users are allowed to push to this namespace. (*)
drupal.org is allowed to create tags on this namespace.

Tags on the issue namespace are associated with issue comments.
Whenever a user posts a comment, they can choose one (or more) commit id from their personal git namespace.
drupal.org will then create a tag in the issue namespace, based on this commit id, and named after the issue + comment number.
The core version or module release branch could also be represented in the tag name. E.g. "7.x-2488266-32". Maybe a custom suffix is allowed.
The issue namespace will only contain these tags associated with issue comments. It will not contain branches or unrelated things.

Tags in the issue namespace are displayed in the respective issue comment.
Alongside a tag there can be a patch download link. The patch would be auto-generated so it applies to the current drupal core or module version, or the last where there are no merge conflicts. The user could be given the option to have a patch based on a previous version instead.

Posting a patch is still possible, and does generate a tag.
Existing patches could be converted automatically into tags based on old commit ids where they still cleanly apply, based on the comment date.

People can add the issue namespace as a remote to their local git repository, and create new branches based on the tags therein.
Those branches only live locally and in the personal namespace, but not in the issue namespace.

Pull requests are gone.
Instead, people can mark their published tags with categories such as "proof-of-concept" or "pseudocode" or "preview" or "meant for merge".
Also they could tick a checkbox to decide if testbot should run on the tag.

--------------------

(*) Why not allow people to push on issue namespaces?
The idea here is to avoid poorly named tags and branches on these namespaces. E.g. someone might accidentally create a tag named after the wrong issue and push it. Or you get some tags associated with issue comments, and some orphan tags floating around associated with nothing. So you have a lot of distraction.

I really like the proposal. The namespaces with per-user repositories seem like a great solution.

The tags idea seems to make it more complex to build and use than is strictly necessary. I wouldn't mind having a comment per commit on the initial version. We can always rebase and clean up the history before making a push, so we do not litter the issue queue with needless WIP commits.

Not having to tag would better match the workflow we have now and make the transition easier. We can consider a push to be equivalent to uploading a patch in the current workflow. Now you only upload a patch if you want to publish the code you're working on, so in the future you would push to the branch.

Some people like to create lots of commits, so be it, this is also the case on Github and it doesn't bother anyone.

@pfrenssen:
A commit has to be in the history of either a branch or a tag, or else git will "forget" it.
This means, an issue comment referencing a commit id will easily get orphaned, if e.g. the branch the commit belonged to is being rebased.
The point of tags is that they are immutable, unlike branches.
And they are named, so we can let the tag name represent the issue comment number.
Commits just have their id, which is not meaningful at all.

pfrenssen and me had a discussion.
We agreed that it would be a powerful thing if git pushes generate issue comments, without manually posting a comment.
Maybe we could still automatically create a tag for these pushes, so that the history is not destroyed by garbage collection.

There is still the question whether you push to your personal namespace or the issue namespace.
My own concern was that issue namespaces would be filled with "junk" if we allow pushing to them. But maybe this is not as bad as I think.

The issue with junk commits could be alleviated if the core committers act as gate keepers for the git history. In addition to reviewing the code, they could assess the git history, and reject an issue if the history is considered "too dirty" and in need of a rebase.

Maybe initially we could keep the current approach of having one commit per issue, essentially squashing all commits and using the current commit message "Issue #12345 by abc: Fix xyz.". This would be less disruptive. Once everybody is getting used to the git workflow we can retain the git history.

RE #27, we did it with Symfony....

Symfony isn't closed source? If anything bad happens to Symfony, we just fork it.

Wouldn't automatically created tags create a lot of polution of the tag-space (even if it's only within a particular namespace)? What if for a commit, a patch is automatically created and only the commit hash is stored with the comment? If it is ever garbage-collected, the patch will still be there to catch the actual changes (btw, isn't the original commit hash preserved somewhere when a commit is cherry-picked or rebased?).

Wouldn't automatically created tags create a lot of polution of the tag-space

No, I don't think so.
Maybe we will have more tags than we currently have patches, because people will be more trigger-happy. But I don't see how this would "pollute" anything. git lola (google it, it's a nice shortcut) will show you the (non-linear) history of all the tags in the issue. git log or lol will show the linear history for a specific tag. you only see what you want to see.

Maybe a local repo will be polluted if it has tags for a number of issues. But there might be ways to further filter the log output.
And other than the log output, there is not that much to worry about, or is it? Maybe overall disk space?

What if for a commit, a patch is automatically created and only the commit hash is stored with the comment?

Then if you clone the namespace repository and look at the git history (e.g. with git lola), you will not see how the commit ids associate with issue comments.

If it is ever garbage-collected, the patch will still be there to catch the actual changes.

But patches are ugly, and they are not native citizens in a git workflow.
The point is to get away from patches, so we can checkout older commits (tags) posted to the issue, and compare different approaches.

I wouldn't worry about tags taking up disk space. I didn't know the pseudo-standard that is "git lola", I did know similar "tips & tricks" aliases to create a visual history on the command line. Graphical tools like SourceTree usually have this baked in (the biggest reason I'm actually a big proponent of graphical git clients :)). I do think we should consider other methods of inspecting tags than such visual histories, though. One might argue that tags are intended to easily find certain things without having to resort to inspecting the full history, so I think we should be careful when considering to auto-generate tags. The command line has its own methods of dealing with extremely long lists of "stuff" (e.g. grep), although I wouldn't be a fan of having to append a grep to my git tag calls all the time :) Because of the namespaces, the problem would probably [as you say] be limited to local repositories (and only show the tags for the issues the local user has worked on), which can be cleaned out.

Maybe a good idea to alleviate this problem somewhat, at least for those using graphical tools, is to use slashes in the tag name. The two graphical Git clients I've worked with, Tower and SourceTree (I've tested SourceTree just now, Tower I am 95% confident it will handle tags that way too), will render tags and branches with slashes in their name as folders. So, instead of 7.x-2488266-32 I would prefer something like "issue/7.x/2488266/32". Or maybe even "issue/7.x/2488266-meta-improve-git-workflow-on-drupal-org-by-implementing-issue-workspaces/32". That way, my tag list would only show a folder "issue" by default.

I wouldn't worry about tags taking up disk space.

To clarify:
- I am not at all worried about server disk space, only about the size of the repo when you download it / git clone / git fetch.
- The disk space is not used for the tag itself, but for the preserving of history behind these tags. E.g. if we rebase but keep the old tag around, the history of the old tag is preserved.
(So if in some strange way we want to preserve the history but don't want the tags, it will still use the same disk space.)

A lot of files will be identical before and after a rebase, and git is smart enough that in most cases we don't need to worry as much.

So again:
The history of these tags is exactly what we want to preserve, and what people want if they fetch from the namespace.
The tags give us exactly this.

We have to take a look at our current backport policy because it would be incompatible with issue workspaces. Currently bugs need to be fixed for the most recent version of Drupal and then backported to older versions. By using per-issue git branches this would mean that if a bug gets fixed for 8.1.x and then merged back to 8.0.x this would take the entire 8.1.x history with it.

This is being discussed in #2623118: [policy, no patch] Update patch backporting workflow to be compatible with semantic versioning.

Is this issue still on anyones plans? I still see more and more modules move to github. Generating and uploading patches is getting pretty outdated for most developers these days. Not to mention the extra work it causes for the maintainers.

This is very much something we still want to do here at the DA. We're (right now) doing some evaluation of some backend git tools that could replace some of our largely custom git backend and make moving in this direction a bit easier.

Sounds great :)

Can we have some transparency on when this might take place...

Fair question. This is actually postponed at the moment. We did not want to undertake this effort before Drupal 8 was released to prevent disruption of that process. The Drupal.org roadmap shows the current priorities for Drupal Association staff and the community initiatives we can actively support.

There are several initiatives ahead of implementing Git improvements, so it is more accurate to mark this postponed for now.

> This is actually postponed at the moment. We did not want to undertake this effort before Drupal 8 was released to prevent disruption of that process.

Is it time now to look again on this? It would make the world so much brighter...

@axel.rutz : a lot has changed since this was first proposed and where we're at now, but we have still been assessing our path forward on improving the development workflow for the drupal project. We'll have some updates soon.

You speak in enigmae, but i'll stay tuned!

@axel.rutz blog post posted today: https://www.drupal.org/drupalorg/blog/developer-tools-initiative-part-1-...

Lightly editing the issue summary to split out #3070177: Protected (per-user?) branches in issue workspace forks into something that can be done later. More updates for GitLab are needed.

Updating some of the issue summary to adjust the wording now that we have git.drupalcode.org running GitLab.

A lot of the work will end up in #2205815: Merge requests in Drupal.org issue queues? as well. This issue will be for creating the forks and branches. Then we need to handle opening merge requests from those branches.

We are all desperatly waiting for forks per issue and merge requests. I read all the related issues but did not found any documentation about the current progress / state since the last 7 months.

Maybe someone can give an update about the current state and the timeline / blockers?

@ayalon - Absolutely! Let me provide an update:

Firstly, here are the relevant child/related issues for issue forks/merge requests:

• #3071662: Decide on, and implement permissions for issue forks
• #3071674: Decide on force push rules for issue forks, implement if necessary
• #2205815: Merge requests in Drupal.org issue queues?

The first two issues above are focused on the per-issue forks themselves and setting the appropriate user permissions for creating and interacting with them. The last issue which is postponed on the first two will be for actually enabling merge requests.

This development site (the htaccess is: drupal/drupal) shows an example of the work in progress on forks:

• The fork button on issues: https://gitlab1-drupal.dev.devdrupal.org/project/drupalorg/issues/3085041
• The link to an existing fork: https://gitlab1-drupal.dev.devdrupal.org/project/drupalorg/issues/3086885
• What that fork looks like on GitLab: https://gitlab1.code-dev.devdrupal.org/issue/drupalorg-3086885

There are a few next steps and a few blockers

Next steps:

• Need to finalize who has access to which forks
• Need to finalize whether we allow force pushes on forks (we probably won't)
• We need to update the work in progress on forks to implement those permissions
• We need to update the work in progress on forks to implement the automated comments back to the issue with relevant activity
• We need to enable the merge request option on forks
• We need to implement the automated comments back to the issue with relevant merge request activity

Blockers:
There are a couple blockers right now - mostly having to do with the imminent release of Drupal 9, and the Drupal core team needing our support to make sure that Drupal.org's tooling is ready to support that. To quickly summarize:

• The blocker on the GitLab side with git object de-duplication has been resolved
• We are updating our packaging pipeline
• We are making updates to support contrib semver in time for D9 release (see child/related issues as well)
• We are updating how the api_tid (i.e: what core version things are compatible with) is used throughout the site, so people will be able to find D9 compatible things when that's released. (see child issues as well)

That work shouldn't necessarily take us all the way until the release day of D9 - so as we make progress we can hopefully turn back to this as our next highest priority.

@hestenet I honestly appreciate your precious time you took to summarize this promising status. Thank you and all others involved in making these milestones happen!

Fixed issue link

Any update on this?

The most-updated quick summary has been the issue forks & merge requests documentation at https://www.drupal.org/drupalorg/docs/gitlab-integration/issue-forks-mer.... (The easy to remember URL for this is https://git.drupalcode.org/issue, which links to that page.) That status is currently accurate:

Issue forks are in active development and being tested with some projects

The basics of issue forks, providing a shared Git repository to collaborate on an issue, are working. More features and fixes are in progress. #3152637: Opt-in to the Drupal.org Issue Forks and Merge Requests beta to get them enabled for your projects.

Follow progress at #2488266: [META] Improve Git workflow on Drupal.org by implementing issue workspaces and #2205815: Merge requests in Drupal.org issue queues?

DrupalCI testing for merge requests is coming soon

Merge requests are now enabled for issue forks. We’re working on followups and fixes, including testing with DrupalCI. For now, only patches will be tested as usual.

Follow progress at #3072048: Test merge requests made in GitLab with DrupalCI

Is the new issue fork UI described anywhere? I've seen issues using the new feature, but I'm unclear about the purpose of the various links.

In particular:

• Which links show me the entire set of differences vs. HEAD (equivalent to looking at the .patch file attachment)?
• Which links show me the changes between issue comments (equivalent to the interdiff file attachment)?

The documentation is at https://www.drupal.org/drupalorg/docs/gitlab-integration/issue-forks-mer...

Now that most of the information about issue forks is in the UI on issue pages, it is due for a visual refresh. Any documentation of the issue page UI, especially screenshots, will become outdated soon.

The icon will:
- If the branch has a merge request, link to the Changes tab of the merge request.
- If there is not a merge request, link to GitLab’ branch compare tool.

Issue forks are Git repositories, the interdiffs are the individual commits. Each commit ID is linked to that commit. The merge request page also has a Commits tab.

Here is a properly-embedded image of that compare icon, at least how it looks now:

There's quite a lot of UX WTFs in the Gitlab UI when in the Gitlab UI for an MR or Gitlab web IDE on an issue fork. They're mostly to do with options/text not lining up betwee the normal gitlab MR model and our D.O issue fork model. Shall I open an issue for each one and a meta to track them all, or would you prefer to keep it all in one new (or existing) issue?

Since GitLab is a separate product, we can only expect them to make UI changes if they align with their product vision. I’ve been striving to have the UI on Drupal.org line up with what GitLab calls things.

If these are merge-request-related, child issues of #2205815: Merge requests in Drupal.org issue queues? are best. For issue forks, before the merge request is opened, child issues of this issue work well. We can always move around issues if they aren’t in the best place.

I wrote it up my UX test as a single issue: #3174334: UX test of issue fork and MR . I'm not sure what's fixable and what's not, and it got pretty tangled, so I won't break it into child issues.

@jonathanshaw - thank you for doing that! UX feedback especially is really helpful in this phase of the beta.

I’m planning to make a significant change to access to issue forks to support #3072047: Handle merging of merge requests.

Currently, issue forks do not allow force pushes or other history rewriting. This ensures people do not accidentally overwrite someone else’s commits, and no work can be lost. However, this interferes with GitLab’s ability to rebase branches for merge requests, which does a force push. That in turn interferes with GitLab doing fast-forward merges, which will get linear commit history matching our patch workflow.

Allowing force pushes would let GitLab work the way it is designed. When this happens, our Git update hook can tag the previous branch HEAD, so the previous history won’t be lost. Force pushes could be over-used and disruptive, but they are a good tool when used judiciously.

Confirmed: I always work with rebased branches, which need force pushes. And manual selective squashing, so that merged branches retain a set limited of commits showing the significant steps taken building the merged feature, without having to keep all irrelevant details. Makes for more understandable merges.

+1 to rebases, merge commits just pollute a git history without reason

Instead of it better to educate people about rebase - it could be destructive but it makes you to pay more attension to what you're doing.

An important part of contribution for Drupal is providing credit. Hopefully people will not get into the habit of pre-squashing the history, or interactive rebasing, losing commits that deserve credit. The merge request notes will always have what people have pushed to the issue fork, but it might not be immediately clear if they pushed something that moved the issue forward or not. Or if interactive rebases do happen, the person pushing takes care to update commit messages.

The norm should be only do simple rebases keeping all the same commits. Only when needed to fix conflicts or bring in important upstream commits. GitLab will have the maintainer do one last rebase before fast-forward merging. We don’t need to mash the rebase button every core commit.

Hello,

I have read comments from comment 64 and re-read the documentation page https://www.drupal.org/drupalorg/docs/gitlab-integration/issue-forks-mer....

I have tried a little bit the "issue forks & merge requests" on #2856713: Authentication plugins and HTTP authentication and another issue on the CMIS project some time ago and here is my feedback if it may help.

Usage: at the testing moment, I found it not friendly to have to clone the project at each fork and so switching from issue to issue was non intuitive.

Now I see that commands on the issue and on the documentation page had been updated to clearly show how to add remote instead of having to clone again and again. I may have missed the info at this moment. But I am glad to see this improvement in documentation, UI, and happy to see that it is how I would have liked it to work.

Thanks!

Credits and Git history: on the issues I tested the workflow, when the code was ready to merge, I used the URL to get a patch from the MR, and then I made a commit from that to be able to control credit attribution and avoid the "fix stuff", "fix again", etc. commits of iteration on the issue.

I never used the squash feature in Gitlab UI (one day I should make a test project on gitlab.com to see what happen) so as I didn't want to risk wrong/unfair credit attribution and polluted git history, I prefered to manually do it like with previous workflow. And also using the "--allow-empty" option of "git commit" to git credit commits to more than one people.

From what I read in the last comments, squash can be interactive. I will need to test that. And +1 for rebase.

When working on a project I maintain, I still prefer making patches the old way instead of creating forks when I am alone on an issue.

This was my 2 cents, if this may help.

And thanks for the hard work on Gitlab integration!

From what I read in the last comments, squash can be interactive.

I was referring to git rebase --interactive, https://git-scm.com/docs/git-rebase#Documentation/git-rebase.txt---inter.... The only really interactive thing like this I’ve seen in GitLab is conflict resolution, https://docs.gitlab.com/ee/user/project/merge_requests/resolve_conflicts....

For the most part, GitLab’s UI does exactly what it says it will, immediately without confirmation. The merge button on a merge request will do exactly that. If you are doing a squash merge of multiple commits, there is a field to expand and set the commit message; it isn’t shown for merging a single-commit merge request. That specific limitation is not present in the API, so a merge button on the issue page will use the commit message template from the issue.

I have an idea of simplifying the transition from patch flow to merge request flow for existing issues with patches.

For issues that have the patch files attached, provide an option to create a fork with a branch that includes changes from the selected patch.

Currently I let composer (cweagans/composer-patches) apply the remote patches from d.o. Using issue forks I can't see how to link the patches in composer. Any thoughts on this? I saw the "download as diff" option in the merge request on gitlab, but I still need to create the patch and upload it to the issue before linking it.

There is a Lullabot article about A Patch-less Composer Workflow for Drupal Using Forks , would that be the new best practice?

You can still use patches. See proposal to improve UI to provide patch url for a MR: #2205815-29: Merge requests in Drupal.org issue queues?

From Reviewing and merging merge requests:

This will make a single squash commit with the branch’s changes, and the commit message you drafted.

It is possible to uncheck that box like on normal GitLab so that commits aren't squashed? As some folks mentioned earlier here, I'd like to preserve history, credit, etc.

Answering my own question:

Yes, I just found this at Issue forks & merge requests:

If you’d like the full commit history or a merge commit, you can merge from the merge request page, or push the merge using Git.

For issues that have the patch files attached, provide an option to create a fork with a branch that includes changes from the selected patch.

Something like this would be good to add. Doing this in a standard way would be useful. I hope that people stick to only making an issue fork when they have substantive work to do. Just opening an issue fork with existing work isn’t the best contribution.

On the other hand, the transition from patches to merge requests will be short in the long run. And this would be a bit of code to maintain indefinitely for the long tail of patches.

There’s been the idea of automatically making the issue fork commits for patches, we could do that as a bulk operation once merge requests are the common workflow on Drupal.org.

There is a Lullabot article about A Patch-less Composer Workflow for Drupal Using Forks , would that be the new best practice?

Yes, this looks like a potentially-good option for deploying patched project codebases to your site. GitLab’s merge request plain diffs are a moving target, new pushes to the branch will be in the patch. For deployment to a site, you want to know what you are deploying, and have control over when specific changes are pulled in. Keeping a copy of the patch in your own site repository would be another option. Both reduce dependance on 3rd parties when running your site build, which is also good.

It is possible to uncheck that box like on normal GitLab so that commits aren't squashed? As some folks mentioned earlier here, I'd like to preserve history, credit, etc.

Initially, we’re steering maintainers to keep the simple commit history, following how the old patch workflow is committed. If there is enough demand for different workflows, and GitLab/Git’s other options aren’t enough, we could look at making this a per-project option.

Should I create a child issue for this:

The links in the issue forks area on issues to "About merge requests" link to
https://www.drupal.org/drupalorg/docs/gitlab-integration/issue-forks-mer...

This page is useful, but it is kind of oriented towards learning about how they were implemented, rather than documentation about how to use them.

I think it would be better to link to the documentation page that now exists:
https://www.drupal.org/docs/develop/git/using-git-to-contribute-to-drupa...

It would also probably be a good idea if the page under drupalorg linked out to there for docs on how to use them... we shouldn't really have two, and I think the docs page under the Git guide is a better home for the user-oriented docs.

@jhodgdon - really good feedback, we should make those fixes.

For reference for others who might jump in, made a quick and dirty video demoing how this works: https://youtu.be/NIWCXE-aM6Y

https://www.drupal.org/drupalorg/docs/gitlab-integration/issue-forks-mer... was intended to be the documentation on how to use issue forks. As functionality was completed, there were some notes about the development status, but it is all practical information for contributors now. (I don’t see implementation notes on that page, documentation about implementation is usually doomed to become outdated.)

We should merge the pages and redirect /drupalorg/docs/gitlab-integration/issue-forks-merge-requests to /docs/develop/git/using-git-to-contribute-to-drupal/creating-issue-forks-and-merge-requests.

Once that’s done, the other place to update the link is https://git.drupalcode.org/issue. And this should help #3178260: Revise “For collaboration on [issue]” issue fork project desctiption as well.