This issue is a summary of research done by the Drupal Association staff, and based on various conversations, which happened in the Drupal community over the years. This was presented in more detail at DrupalCon Los Angeles.

Background

Current issue queue workflow while works, has its limitations:

  • Complex workflow with many steps
  • Trivial changes require complex workflow
  • Difficult for Learners
  • No visibility between issues
  • Patch Review can be challenging
  • Stale Patches

A better issue queue:

  • Uses git, instead of patches to submit, review and accept work (e.g. "Pull requests", "Forks")
  • Detects code conflicts and provides immediate notification
  • Integrates code review functionality
  • Allows inline editing for smaller fixes

We want to build that better issue queue, while meeting the following constraints:

  • Minimize disruption
  • Preserve collaboration
  • Keep Drupal.org home of the community
  • Limited resources

Possible solutions

  1. Do nothing :)
  2. Migrate to externally hosted solution (GitHub, BitBucket, Gitlab.com)
  3. Use self hosted solution (Gitlab/Phabricator, Diffy, FishEye / Crucible)
  4. Modernize Drupal.org's Issue queues

1. Do nothing

Is not really an option. We repeatedly hear from users, that they need better tools.

2. Move to a hosted service

Examples of hosted services: GitHub, Bitbucket, Gitlab.com, etc.

Moving to GitHub
GitHub came up in discussions again and again. While being a great site in itself, there is one fundamental difference, which makes the move to GitHub at the very least problematic, if not dangerous for our community.

The fundamental difference is in the data model or coding workflow.

GitHub's Data Model: Fragmented Social Coding

  • Code changes on GitHub are pull requests.
  • Both pull requests and issues have comments.
  • If someone else wants to work on the same code, they fork a pull request.
  • When you fork a pull request, you fork a conversation. It is now in multiple places.

Drupal.org's Data Model: Collaborative Social Coding

  • Code changes on Drupal.org are patches.
  • Patches live in comments on issues
  • If someone else wants to work on the same code, they create a new patch.
  • Conversation happens in one place.

GitHub would give us pull requests, code conflict detection and immediate notification, integrated code review functionality, inline editing for smaller fixes. However:

  • The move would be very disruptive, especially at this stage of Drupal 8 release cycle.
  • It breaks the model of collaborative coding, which is the basis of the Drupal community.
  • Drupal.org would no longer be the place where everyone goes for code.
  • Limited Resources - This solution is sometimes perceived as the most efficient and easy, however building integration between the current site and GitHub is a non trivial project, which won’t be much easier than some of the other options.

3. Move to self-hosted tool

Examples of self-hosted tools: Phabricator, Gerritt, Gitlab, Fisheye, Crucible, Diffy.

Self-hosted tools have a few advantages over moving to a hosted service:

  • Data models might not be as different, so we could potentially keep the collaborative model we currently have.
  • Drupal.org would still be the central place of everything Drupal.
  • Self hosted means we own the data.

However, switching to a new tool would be a massively disruptive change. Resource requirements are even more complex, because of the need to host the solution ourselves. We’d have additional maintenance and security burdens.

4. Modernize Drupal.org issue queue

This one is not a new idea as well. There were various plans and ideas how to modernize our issue queues before.
We have 4 years worth of ideas & suggestions to draw from.

We can switch to using git, instead of patches to submit, review, and accept work, code conflict detection and immediate notification, code review functionality, inline editing for smaller fixes.

And this options meets all of our constraints:
1. Minimal Disruption + Preserves Collaboration:
- we’ll have minimal disruption, because we are not making people learn anything new, we are improving what we have;
- our model of collaborative coding won’t change;
2. We’ll keep Drupal.org home of the community
3. Limited Resources: This will be a big undertaking, however right now we are positioned much better to make it a success than even 1 year ago. The specific solution we’d like to propose is not something completely new we’ll be building from scratch, essentially we’ll be using a new feature in Git, already built.

Proposed resolution

Implement issue workspaces on Drupal.org

Issue workspaces are a new way of working on issues that utilize some underutilized features of git, that allow contributors working on an issue to use the power of git to manage change.
It is very similar to “per-issue repositories” - like a per issue fork of the main repository (core, contrib etc).

The feature is Git namespaces that allows for logical separation in the git repository for references (heads, branches, tags), that are isolated from the rest of the repo, yet still share the same objects. Everything is there on the remote server, but everything appears to be different repositories to the client. This allows us to have cheap ‘forks’ of a project per issue, and easily merge those forks back in.

  • One namespace per issue
  • A per-user, per-issue protected branch in the namespace that only that user can push to.
  • A branch pointer to the “Latest” so users can easily pull in the most recent change into their own branch.
  • Any number of shared branches for those times when advanced users need to hash out several concepts.

How would the new Git-based workflow work

  • When a user wants to work on an issue, they will see instructions to clone the issue workspace e.g.:
    git clone username@git.drupal.org:project/drupal/issues/123456.git
  • User makes their changes
  • Instead of extracting and uploading a patch, they commit and push commits to their own branch in the issue workspace
  • On Drupal.org a comment will be generated on behalf of the user, that contains their code changes as well as their commit messages. Testbots will take the changeset that was pushed and run tests on it.
  • If user needs to add additional explanation of their work beyond the commit messages, they can add a comment.

Real life example from https://www.drupal.org/node/1740492

Legacy patch based workflow does not go away

  • Users can still upload patches, download and apply patches, and current tools like dredditor should still work, except no interdiff will be necessary.
  • All changes via the new git workflow will also be downloadable as a patch, so this workflow will not change.
  • When a user submits a patch, a commit will be created on that users personal branch so that other git using developers can work with those changes in git.

Additional features
As an additional feature, we would like to add the ability to inline edit work in progress changes in the issue, eliminating cumbersome workflows for simpler edits such as code style, documentation, typo fixing etc.
1. If there is already code changes in progress on an issue, either uploaded by a patch or via a git push to the workspace, then a user will see a button on the issue comment next to the change that will open a code editor (for example ace.js) that will allow the the user to make edits.
2. Once the edit is complete and saved, a commit is created on that users workspace branch and another comment is generated on the issue queue.

Code review in issue workspaces

  • As a result of preserving our legacy interface with patches, we can continue to leverage tools such as Dreditor without needing to implement code review and inline diffs as part of the MVP.
  • Additionally we can provide an option to compare any two changes in the issue, providing interdiff functionality.
  • Reviewing major changes does not mean a 300k patch. it can mean reviewing multiple small commits.

Merging changes upstream

  • Button on interface to “squash merge” all the commits into a single unit of work, for core probably, or even a straight merge.
  • A maintainer could merge in any branch into their production codebase without squashing too.

Velocity acceleration

  • Reveals issues conflicts using git
  • Reduces need for “disruptive patch windows”
  • Commits that break other changes notify immediately
  • Needs Review is meaningful.

History, previous discussions:

A comprehensive summary of Github vs Drupal vs Gitlab : https://groups.drupal.org/node/313068#comment-955593
Github/Gitlab/Drupal spreadsheet: http://bit.ly/1tPt3qS
Git Phase Next: The Promised Land. Slides:http://bit.ly/1qKecLI Presentation: http://bit.ly/1AGiEfI
https://www.drupal.org/community-initiatives/git/phase-3
Implementing per issue repos: https://groups.drupal.org/node/50438
https://www.drupal.org/node/2213153
https://www.drupal.org/project/issues/search?issue_tags=git%20phase%203
https://www.drupal.org/documentation/git/faq -’what about pull requests’

Members fund testing for the Drupal project. Drupal Association Learn more

Comments

tvn’s picture

Issue summary: View changes

adding details

tvn’s picture

Issue summary: View changes

adding more details

disasm’s picture

I was just at the DrupalCon session for this and I love the idea. When you start coding this, I'd love to help out. Really looking forward to the in-line editor making commits. That will be so nice for sprints for people that want to submit doc changes without having to learn git! I also like the idea of a per-user branch in each issue. It will make diff'ing everyone else's contributions from the command line so much easier. For now, if they don't attach an interdiff, I have to create a new branch for that person to track their changes in, do a git apply of their patch and for subsequent merges, I have to roll back to 8.0.x, apply the new patch, merge it back into the branch I have tracking their commits. I can see where the "shared branch" approach could be useful for big issues where a bunch of people are working on the same branch and "passing the baton", but I'm really for having my own namespace no one can muck up without me merging their changes in! It will also be very nice to be able to easily diff other branches, or cherry-pick commits into my branch when I only want part of their changes.

drakythe’s picture

Not at DrupalCon, but followed Webchick's live tweet of it and love the sound of the proposed solution. I've patched a few modules, and worked with module's which required patches and neither scenario has ever been the easiest. This sounds like it would make contributing patches, especially trivial ones, super easy.

I also like #3's comments about being able to cherry pick commits from other branches instead of having to outright merge their changes, that is a good idea.

gabesullice’s picture

I would love to second disasms enthusiasm and his offer to help. I'm very excited to hack on this. As a former proponent of moving to something like GitLab, I am now a convert. To help evangelize this, I'd like to put down my reasons for my conversion.

The biggest problem and change that would be associated with migrating to a service and/or self-hosted VCS solution is the distributed collaboration. Not to be confused with distributed version control (This solution is NOT eschewing the benefites of ditributed version contol). The best way to illustrate this, and similar to the way it was illustrated at the DrupalCon Los Angeles session, is to go through a worklfow common on a service like GitHub.

Imagine that Joe has an issue with Project X. Joe has to find the project's page on Github and create an issue. Joe then forks the project, does his work, and submits a pull request. Nancy comes to the issue and reviews Joe's code. Nancy has some comments and decides she wants to make a change. Nancy now has to fork the project or Joe's fork and submit a PR to his branch or create her own PR to the project. Now, Joe comes along and appreciates her ideas, but has some concerns with one of her changes. Where should Joe make his comments? He can add a comment to his issue, which may or may not have Nancy's commit associated with it. He could go to her pull request and make a comment. He could also make an inline comment on one of her commits. As the issue gains traction, a third and fourth person might join the discussion, they may create their own forks and comments. One can no longer just read the single canonical issue to bring themselves up to speed. Whose fork is canonical? Whose work has been abandoned? The problem can be exponentially difficult to solve. This is the ugly side of distributed collaboration/deliberation, admittedly pessimistic, but nonetheless an outcome that is all too probable.

On Drupal.org, we instead have a patch-based workflow that has encouraged and enabled us to work on a single issue in an ordered, single thread. Joe comments, adds a patch, Nancy comments, adds a patch, and Joe responds in the issue queue and only in that issue queue. Any following collaborators can come to the thread, read it chronologically, and add their contributions to the last patch or last agreed upon patch.

In short, the beauty of the d.o issue queue is that it imposes a chronological order in the collaborative process. It allows us to deliberate together and helps us make sense of everyone's work as a whole. We only have so much brainpower. On the other hand, the beauty of git is that it handles the unordered and distributed nature of code commits. Unlike patches, commits don't easily become "stale." And when they do, it's much more trivial to refresh them. This is the brilliance of a distributed VCS - it helps us reason about something inherently chaotic. The issue queue is how we reason about the meta discussion around that same code.

This proposal promises the best of both worlds. We can clone and work independently, we can increase the velocity of our contributions, and we can preserve all of the benefits we get from a single deliberative issue queue.

tvn’s picture

Issue summary: View changes

adding more details

webchick’s picture

Shout-out to @gabesullice for a very succinct description of the exact problem with "forks of forks." :)

tvn’s picture

Issue summary: View changes

More details for the session.

tvn’s picture

Issue summary: View changes
FileSize
216.38 KB

Adding workflow example image.

TorreyToomajanian’s picture

I love the sound of this. It's great to see how active this community always is in innovating for the sake of said community. +5 stars and then some!

mglaman’s picture

I came to Drupal and stayed because of the community, and the way Drupal.org works. This proposed plan sounds like it will fully embody the spirit of what "is Drupal" and our collaborative effort.

I pinged one of our system administrators at Gaggle. He hasn't had a time to use the feature of Git namespaces, unfortunately. He did say it looks to be a better implementation of the git alternates feature that aimed to cut down on the disk space issues of forking (quickest resource I could find, an article on implementing it in GitLab.) Just for some extra information :)

tvn’s picture

Title: [META] Implement issue workspaces on Drupal.org » [META] Improve Git workflow on Drupal.org by implementing issue workspaces

Updating title

Kazanir’s picture

Well, this would be totally fantastic.

dasjo’s picture

first of all thanks for the mention in the illustration, it least my non-productive patch got to be a good example for how the collaborative workflow can look like :P

i appreciate there has been a lot of research into options, moving to github etc and can see that we want to embrace collaborative workflows so couldn't find a ready to use solution yet. still i'm concerned as we are inventing our own solution again and my feeling is that this is not a drupal-specific problem at all.

are we the first ones to use the git namespaces feature to create issue workspaces?

i understand that the current patch-based workflow is really out of date and we need to move forward, but i think we should also make sure to share our learnings with others interested in the same concepts. ideally this problem would be solved similarly on github or other popular platforms.

Mixologic’s picture

We are definitely inventing our own solution for drupal.org. Typically its best to avoid re-inventing existing tools, but as far as I know, we will be the first ones to utilize git namespaces in this manner, and therefore we'd be the first to implement this style of workflow.

In researching all the various git daemons, self hosted solutions, and hosted services, none of them offered a concept whereby an issue can be worked on collaboratively in this manner. All work was assumed to happen on one branch in a forked copy of the canonical repo, which then becomes a pull request or similar - limiting transparency on work in progress, and fragmenting the interface, history and conversations.

I agree that as we build, test, and beta and ultimately launch this workflow, there would be definitely be value for other projects. Our collaborative workflow has helped us build extremely powerful software, and other projects could benefit from that process.

If this works like we think it might, and adds the value we believe it will, then I foresee it becoming something that could spread to others tools. Since we'll be doing everything in public, and all of our source will be available, we would definitely encourage adoption and collaborate with others. But I strongly doubt that this would be solved at github or other popular platforms without first demonstrating it's value to our community, which Is why we believe we have to make it happen here, on drupal.org first before promoting this solution 'upstream'.

alex.designworks’s picture

Personally, I think GitHub is way more familiar that d.o. and way more user friendly.

Also, fork of forks and having multiple places for comments - this is not a big deal at all. Check out those issues on d.o. with tens of patches and hundreds of comments and try to find the proper answer or solution - if people keen to dig through that than they would probably be able to handle comments on fork of a fork! And this is actually a minority of cases. On the other hand, think of how easy it is for a module maintainer to merge pull requests comparing to applying patches and how easy it would be to resubmit a pull request comparing to re-submitting a patch!

I know that this has been discussed for a couple of years now in different issues on d.o.

With all previous discussions in mind, can we actually create a vote for all community members? That would be a TRUE community-elected decision and no one would feel like their opinion is left out. The vote would state simple options: leave as is, GitHub, namespaces. The explanation behind each option can link to this issue so that everyone would understand the reasoning behind this.

And one last thing. Many other projects successfully use GitHub and it has proven to be a valuable and reliable tool. So why does Drupal has to stay on this island?

alex.designworks’s picture

Here is a link to a video of Workspaces presentation at DrupalCon LA 2015
https://www.youtube.com/watch?v=B392kwHg_Rs

dasjo’s picture

thanks for the feedback mixologic. looking forward to see this implemented soon :)

i personally don't think voting on leave as is, GitHub, namespaces makes much sense because we don't have a process for voting on decisions on d.o implemented so far. this would just derail the issue. the current best practice is to find consensus or escalate decisions. at the current point i don't see any strong arguments agains implementing namespaces

alex.designworks’s picture

@dasjo
>> we don't have a process for voting on decisions on d.o implemented so far.
Well, this is not the first time I'm reading this kind of a reply on d.o.

There is no reason to not implement such voting process (which is quite simple thing to do keeping in mind that Drupal Association already origanizes polls and votes for other things). Especially, for a such important thing as development workflow - the heart of any process development process.

So, I'd rather see this issue 'derailed' until properly voted, than "just do it" because there is no process to ask for community opinion.

But hey, I'm just a community member and a simple developer - so cares, right?

alex.designworks’s picture

@gabesullice
Your points about fork of a fork a valid, but in reality would people actually fork a fork to submit their changes in majority of cases?

By looking at the patches on d.o. for contrib modules, the average re-iteration of a single patch would be around 3-4 changes. I think, those changes would likely be done as comments on the commit itself, rather than a fork.

Please keep in mind that other projects use GitHub and it does not scare their users. If a user is proficient enough to apply a patch it should not be a problem to find a related issue in the module's main repo (Github actually shows that a repo is a fork of the original repo).

Also, I'm not sure that introducing workspaces will make it easy to submit patches as it looks like that the whole process would have to be the same 13ish steps.

mglaman’s picture

@alex.designworks you've raised valid points. I just wanted to say that workspaces won't keep the same 13 steps to generating a patch.

Issue works spaces act as a branch. All people working on that branch have their own branches. Work is done by merging and pushing. Then a comitter merges the issue branch. Thats modern fit workflow, at least in my experience.

I don't like GitHub. I hate trying to contribute to projects on GitHub. I've tried working on other communities and feel its harder.

Reading an issue queue with 200 comments is difficult. But its better than reading an issue, to see there is a PR for it. But the conversation happened in the issue AFTER the PR. So you need to read both conversations and compare the results.

webchick’s picture

FWIW, we did vote on Drupal.org features a year or so back, and the vote between "move to GitHub" and "make our tools better" was basically exactly a 50/50 split. Nothing has materially changed to sway various peoples' opinions since then, and the DrupalCon presentation does a good job I think of explaining why moving to GitHub, right now at least, is not an option.

#2261945: [META] Drupal.org ideation tool is worth a read for some of the challenges in ideation/voting, which include: voices of "insiders" vastly outweighing those of actual users, difficulty in broadcasting the opportunity to vote (for example, the DA's elections get about 1% participation), past votes being inflexible to changes that happen "on the ground" to change circumstances, ideas coming up with no one passionate about working on them, ideas that grow popular without fully considering the consequences, etc.

If you have other thoughts on how we can improve the ideation process for Drupal.org, we'd love to hear them, but please comment over on that issue, not here.

webchick’s picture

To further highlight the "fork of a fork" problem with GitHub, and why I'm excited about issue workspaces as an alternative, I can use a real-world example from Drupal 7: the Bartik theme. Unfortunately, most of these forks have now been deleted since then, so https://github.com/jensimmons/Bartik/network no longer shows the full story, but I'll try and recreate it here (please excuse my crappy photoshop skills; also any inaccuracies, since it's been 5 years :P).

GitHub was chosen as the collaboration hub for Bartik since a) Drupal.org was on CVS at the time [puke!] and b) the in-place editing tools allowed contribution by designers who had no idea what version control was. Makes sense.

So https://github.com/jensimmons/Bartik was started and quickly amassed a group of 10 or so people working on it: dmitrig01, JohnAlbin, Jacine, and others. And that worked really well when jensimmons was around to quickly pull in proposed changes from others, and the network graph for the project basically looked like this:

Several forks, each merging into mainline periodically.

And life was good. There was a clear "canonical" place to find the "official," latest and greatest code. There was a single "project owner" that everyone knew to coordinate with. And that coordinator was able to be responsive and quickly merge in proposed changes.

Then, jensimmons got a job in Beijing, and went off the grid for awhile. Which is totally fine, and expected in a predominantly volunteer-driven project; paying bills is good! :) Work continued on Bartik in the meantime, but without central coordination, in dmitrig01's fork, in JohnAlbin's fork, in Jacine's fork, etc. Soon the network graph started looking like this:

Several forks, each fragmented from the other.

And life was pretty awful. The "canonical" project had stalled out, and was effectively no longer accepting pull requests. So everyone worked on their own, some knowing to coordinate with others, some not. It was impossible to know where the "latest and greatest" code was to point people to for testing. And many contributors outside of the "inner circle" missed the message that work was now happening elsewhere, so they were starting from code in the official project from several weeks/months back.

I call out this example not to pick on jensimmons or the people who worked on Bartik. I love all of those people, and their effort was amazing and we managed to get it in core regardless of these challenges. But I think this real-world example highlights very plainly a number of realities that we have in the Drupal community:

- The vast majority of work on Drupal.org is still done by volunteers, or people with a very limited "attention span" (a client is paying me to care about this this week, next week it'll be something else)
- As a result, the person who starts the work is very often not also the person who finishes the work (at least in larger projects/issues).
- And in those larger projects/issues, there are nearly always at least 2-3 people involved due to peer review requirement. In core issues, it can be as many as dozens.
- Those people do not always know each other. In fact, often don't. We get many "drive-by" contributors or people who come to a contribution sprint for the first time, etc.
- It therefore needs to be stupidly easy to find the "latest and greatest" version of the code (and discussion!) to work from, so contributors don't end up wasting their time.

Issue workspaces solve a lot of these issues, plus lets the community "level up" their knowledge of Git so that if a move to GitHub is ever in our future (their issue management tools would definitely not support this currently), it would be a much easier migration than today.

Wim Leers’s picture

Thanks for #23. That comment is so great, that I just had to bookmark it for future reference. Thank you :)

socketwench’s picture

When Flag 8.x was on github, we had at least one time like #22. Most often we had, "this issue is stalled because your PR needs to be rerolls plskthxbai?" Forking a PR fork feels weird and is intimidating for newcomers because of the backend magic that powers it.

cweagans’s picture

First off - thanks to everyone who worked on this proposal. Technically, it seems solid, and the workflow that it creates would be pretty cool.

It seems like a lot of people are getting hung up on why we aren't just moving to Github, and the two reasons I've heard are the PR issue mentioned above and that the issue management on Github is sub-par.

When we discussed moving to Github a couple years ago, I seem to recall a couple of Github staff members saying that they'd be open to talking with us if the path to moving to Github is not as clear-cut as we'd like. It seems like the issue being described above is something that could occur in *any* project, not just Drupal. Would it be totally out of the question to reach out to them and see if they'd be willing to help us solve that problem (presumably by adding functionality to Github)? If Github simply will not work, then sure - let's do our own thing (and if we're going to do our own thing, then the proposal sounds like it'll do exactly what we want). But I think having a major open source project migrate all of their repos to Github might be a big enough deal that Github might work some magic for us.

That said, I'm not sure the PR thing is really as big of a deal as people are making it. If a PR is stale or whatever, just open another PR. It's annoying and the PR list might get messy after a while, but cleaning that up is something we could automate pretty easily with a PR naming convention (if issue #1234 is closed, then close all of the PRs for issue #1234). It just seems strange that we're going through so much effort and expense to build and maintain our own Git hosting service when Github will do what we need it to if we change our workflow a bit (and it might even do exactly what we need it to if we take the time to ask for it)

Elijah Lynn’s picture

re: #26

A third objection is that we would be moving our entire lifeblood to a closed source company where we cannot directly control anything that happens. I realize that this falls under the "not invented here" category, but this is the lifeblood of Drupal. Yes, Github may work with us and everything is peachy for a couple of years but that doesn't mean it will always be. They could turn into a Sourceforge and start plastering ads all around the site (unlikely but possible) or decide to start charging unreasonable amounts of money etc. At the end of the day, they have a financial interest to make money. When a better competitor comes along and steals away their paying customers they will do what they have to to keep the money train alive.

IMO, putting our entire lifeblood into Github is risky. Other people have expressed this as well over the past years of discussing this, not just myself.

tvn’s picture

Category: Feature request » Plan
moshe weitzman’s picture

There is a lot to like about the proposal.

On Drupal.org a comment will be generated on behalf of the user, that contains their code changes as well as their commit messages.

I think that inline diffs and line by line code review are part of an MVP. The experience of working on a pull request is vital, not a "maybe someday" feature. If this expands the scope too greatly, then I question whether we have the funding to build and maintain this system over the long haul.

pwolanin’s picture

I wonder if the proposal here for protected branches per user is adding a lot of un-needed complexity.

I also don't see any discussion of how/when branches would be rebased on core or the relevant project. I find it's much more comprehensible to do that than to try to merge from the upstream project.

In general, these feels like an overly complicated MVP. I'd think a workspace + 1 common branch (with contributors creating more as needed) per issue would be a real MVP.

pwolanin’s picture

donquixote had a good suggestion at the talk that we should really only be dealing with tags not branches in terms of what commits in the namespace get exposed in the UI.

A 1:1 mapping of tags to "patches" would actually make a lot of sense since tags are immutable and I can always start a local branch from a tag to continue work.

Initially at least you might require people pushing to git to make a tag to post a "patch" to the the issue, and an actual patch would be applied to a detached head and make a tag rather than trying to apply it to a branch.

donquixote’s picture

Thanks, @pwolanin.
My complete suggestion, as it shaped during the discussion, is like this:

One git namespace per project per user, across issues.
The respective user is allowed to push to this namespace, operate with branches, do what they want.

One git namespace per project per issue, across users.
No users are allowed to push to this namespace. (*)
drupal.org is allowed to create tags on this namespace.

Tags on the issue namespace are associated with issue comments.
Whenever a user posts a comment, they can choose one (or more) commit id from their personal git namespace.
drupal.org will then create a tag in the issue namespace, based on this commit id, and named after the issue + comment number.
The core version or module release branch could also be represented in the tag name. E.g. "7.x-2488266-32". Maybe a custom suffix is allowed.
The issue namespace will only contain these tags associated with issue comments. It will not contain branches or unrelated things.

Tags in the issue namespace are displayed in the respective issue comment.
Alongside a tag there can be a patch download link. The patch would be auto-generated so it applies to the current drupal core or module version, or the last where there are no merge conflicts. The user could be given the option to have a patch based on a previous version instead.

Posting a patch is still possible, and does generate a tag.
Existing patches could be converted automatically into tags based on old commit ids where they still cleanly apply, based on the comment date.

People can add the issue namespace as a remote to their local git repository, and create new branches based on the tags therein.
Those branches only live locally and in the personal namespace, but not in the issue namespace.

Pull requests are gone.
Instead, people can mark their published tags with categories such as "proof-of-concept" or "pseudocode" or "preview" or "meant for merge".
Also they could tick a checkbox to decide if testbot should run on the tag.

--------------------

(*) Why not allow people to push on issue namespaces?
The idea here is to avoid poorly named tags and branches on these namespaces. E.g. someone might accidentally create a tag named after the wrong issue and push it. Or you get some tags associated with issue comments, and some orphan tags floating around associated with nothing. So you have a lot of distraction.

pfrenssen’s picture

I really like the proposal. The namespaces with per-user repositories seem like a great solution.

The tags idea seems to make it more complex to build and use than is strictly necessary. I wouldn't mind having a comment per commit on the initial version. We can always rebase and clean up the history before making a push, so we do not litter the issue queue with needless WIP commits.

Not having to tag would better match the workflow we have now and make the transition easier. We can consider a push to be equivalent to uploading a patch in the current workflow. Now you only upload a patch if you want to publish the code you're working on, so in the future you would push to the branch.

Some people like to create lots of commits, so be it, this is also the case on Github and it doesn't bother anyone.

donquixote’s picture

@pfrenssen:
A commit has to be in the history of either a branch or a tag, or else git will "forget" it.
This means, an issue comment referencing a commit id will easily get orphaned, if e.g. the branch the commit belonged to is being rebased.
The point of tags is that they are immutable, unlike branches.
And they are named, so we can let the tag name represent the issue comment number.
Commits just have their id, which is not meaningful at all.

donquixote’s picture

pfrenssen and me had a discussion.
We agreed that it would be a powerful thing if git pushes generate issue comments, without manually posting a comment.
Maybe we could still automatically create a tag for these pushes, so that the history is not destroyed by garbage collection.

There is still the question whether you push to your personal namespace or the issue namespace.
My own concern was that issue namespaces would be filled with "junk" if we allow pushing to them. But maybe this is not as bad as I think.

pfrenssen’s picture

The issue with junk commits could be alleviated if the core committers act as gate keepers for the git history. In addition to reviewing the code, they could assess the git history, and reject an issue if the history is considered "too dirty" and in need of a rebase.

Maybe initially we could keep the current approach of having one commit per issue, essentially squashing all commits and using the current commit message "Issue #12345 by abc: Fix xyz.". This would be less disruptive. Once everybody is getting used to the git workflow we can retain the git history.

kylebrowning’s picture

RE #27, we did it with Symfony....

fuzzy76’s picture

Symfony isn't closed source? If anything bad happens to Symfony, we just fork it.

eelkeblok’s picture

Wouldn't automatically created tags create a lot of polution of the tag-space (even if it's only within a particular namespace)? What if for a commit, a patch is automatically created and only the commit hash is stored with the comment? If it is ever garbage-collected, the patch will still be there to catch the actual changes (btw, isn't the original commit hash preserved somewhere when a commit is cherry-picked or rebased?).

donquixote’s picture

Wouldn't automatically created tags create a lot of polution of the tag-space

No, I don't think so.
Maybe we will have more tags than we currently have patches, because people will be more trigger-happy. But I don't see how this would "pollute" anything. git lola (google it, it's a nice shortcut) will show you the (non-linear) history of all the tags in the issue. git log or lol will show the linear history for a specific tag. you only see what you want to see.

Maybe a local repo will be polluted if it has tags for a number of issues. But there might be ways to further filter the log output.
And other than the log output, there is not that much to worry about, or is it? Maybe overall disk space?

What if for a commit, a patch is automatically created and only the commit hash is stored with the comment?

Then if you clone the namespace repository and look at the git history (e.g. with git lola), you will not see how the commit ids associate with issue comments.

If it is ever garbage-collected, the patch will still be there to catch the actual changes.

But patches are ugly, and they are not native citizens in a git workflow.
The point is to get away from patches, so we can checkout older commits (tags) posted to the issue, and compare different approaches.

eelkeblok’s picture

I wouldn't worry about tags taking up disk space. I didn't know the pseudo-standard that is "git lola", I did know similar "tips & tricks" aliases to create a visual history on the command line. Graphical tools like SourceTree usually have this baked in (the biggest reason I'm actually a big proponent of graphical git clients :)). I do think we should consider other methods of inspecting tags than such visual histories, though. One might argue that tags are intended to easily find certain things without having to resort to inspecting the full history, so I think we should be careful when considering to auto-generate tags. The command line has its own methods of dealing with extremely long lists of "stuff" (e.g. grep), although I wouldn't be a fan of having to append a grep to my git tag calls all the time :) Because of the namespaces, the problem would probably [as you say] be limited to local repositories (and only show the tags for the issues the local user has worked on), which can be cleaned out.

Maybe a good idea to alleviate this problem somewhat, at least for those using graphical tools, is to use slashes in the tag name. The two graphical Git clients I've worked with, Tower and SourceTree (I've tested SourceTree just now, Tower I am 95% confident it will handle tags that way too), will render tags and branches with slashes in their name as folders. So, instead of 7.x-2488266-32 I would prefer something like "issue/7.x/2488266/32". Or maybe even "issue/7.x/2488266-meta-improve-git-workflow-on-drupal-org-by-implementing-issue-workspaces/32". That way, my tag list would only show a folder "issue" by default.

donquixote’s picture

I wouldn't worry about tags taking up disk space.

To clarify:
- I am not at all worried about server disk space, only about the size of the repo when you download it / git clone / git fetch.
- The disk space is not used for the tag itself, but for the preserving of history behind these tags. E.g. if we rebase but keep the old tag around, the history of the old tag is preserved.
(So if in some strange way we want to preserve the history but don't want the tags, it will still use the same disk space.)

A lot of files will be identical before and after a rebase, and git is smart enough that in most cases we don't need to worry as much.

So again:
The history of these tags is exactly what we want to preserve, and what people want if they fetch from the namespace.
The tags give us exactly this.

pfrenssen’s picture

We have to take a look at our current backport policy because it would be incompatible with issue workspaces. Currently bugs need to be fixed for the most recent version of Drupal and then backported to older versions. By using per-issue git branches this would mean that if a bug gets fixed for 8.1.x and then merged back to 8.0.x this would take the entire 8.1.x history with it.

This is being discussed in #2623118: [policy, no patch] Update patch backporting workflow to be compatible with semantic versioning.

fuzzy76’s picture

Is this issue still on anyones plans? I still see more and more modules move to github. Generating and uploading patches is getting pretty outdated for most developers these days. Not to mention the extra work it causes for the maintainers.

hestenet’s picture

This is very much something we still want to do here at the DA. We're (right now) doing some evaluation of some backend git tools that could replace some of our largely custom git backend and make moving in this direction a bit easier.

fuzzy76’s picture

Sounds great :)

kylebrowning’s picture

Can we have some transparency on when this might take place...

joshuami’s picture

Status: Active » Postponed

Fair question. This is actually postponed at the moment. We did not want to undertake this effort before Drupal 8 was released to prevent disruption of that process. The Drupal.org roadmap shows the current priorities for Drupal Association staff and the community initiatives we can actively support.

There are several initiatives ahead of implementing Git improvements, so it is more accurate to mark this postponed for now.