This issue is a summary of research done by the Drupal Association staff, and based on various conversations, which happened in the Drupal community over the years. This was presented in more detail at DrupalCon Los Angeles.
Current issue queue workflow while works, has its limitations:
- Complex workflow with many steps
- Trivial changes require complex workflow
- Difficult for Learners
- No visibility between issues
- Patch Review can be challenging
- Stale Patches
A better issue queue:
- Uses git, instead of patches to submit, review and accept work (e.g. "Pull requests", "Forks")
- Detects code conflicts and provides immediate notification
- Integrates code review functionality
- Allows inline editing for smaller fixes
We want to build that better issue queue, while meeting the following constraints:
- Minimize disruption
- Preserve collaboration
- Keep Drupal.org home of the community
- Limited resources
- Do nothing :)
- Migrate to externally hosted solution (GitHub, BitBucket, Gitlab.com)
- Use self hosted solution (Gitlab/Phabricator, Diffy, FishEye / Crucible)
- Modernize Drupal.org's Issue queues
1. Do nothing
Is not really an option. We repeatedly hear from users, that they need better tools.
2. Move to a hosted service
Examples of hosted services: GitHub, Bitbucket, Gitlab.com, etc.
Moving to GitHub
GitHub came up in discussions again and again. While being a great site in itself, there is one fundamental difference, which makes the move to GitHub at the very least problematic, if not dangerous for our community.
The fundamental difference is in the data model or coding workflow.
GitHub's Data Model: Fragmented Social Coding
- Code changes on GitHub are pull requests.
- Both pull requests and issues have comments.
- If someone else wants to work on the same code, they fork a pull request.
- When you fork a pull request, you fork a conversation. It is now in multiple places.
Drupal.org's Data Model: Collaborative Social Coding
- Code changes on Drupal.org are patches.
- Patches live in comments on issues
- If someone else wants to work on the same code, they create a new patch.
- Conversation happens in one place.
GitHub would give us pull requests, code conflict detection and immediate notification, integrated code review functionality, inline editing for smaller fixes. However:
- The move would be very disruptive, especially at this stage of Drupal 8 release cycle.
- It breaks the model of collaborative coding, which is the basis of the Drupal community.
- Drupal.org would no longer be the place where everyone goes for code.
- Limited Resources - This solution is sometimes perceived as the most efficient and easy, however building integration between the current site and GitHub is a non trivial project, which won’t be much easier than some of the other options.
3. Move to self-hosted tool
Examples of self-hosted tools: Phabricator, Gerritt, Gitlab, Fisheye, Crucible, Diffy.
Self-hosted tools have a few advantages over moving to a hosted service:
- Data models might not be as different, so we could potentially keep the collaborative model we currently have.
- Drupal.org would still be the central place of everything Drupal.
- Self hosted means we own the data.
However, switching to a new tool would be a massively disruptive change. Resource requirements are even more complex, because of the need to host the solution ourselves. We’d have additional maintenance and security burdens.
4. Modernize Drupal.org issue queue
This one is not a new idea as well. There were various plans and ideas how to modernize our issue queues before.
We have 4 years worth of ideas & suggestions to draw from.
We can switch to using git, instead of patches to submit, review, and accept work, code conflict detection and immediate notification, code review functionality, inline editing for smaller fixes.
And this options meets all of our constraints:
1. Minimal Disruption + Preserves Collaboration:
- we’ll have minimal disruption, because we are not making people learn anything new, we are improving what we have;
- our model of collaborative coding won’t change;
2. We’ll keep Drupal.org home of the community
3. Limited Resources: This will be a big undertaking, however right now we are positioned much better to make it a success than even 1 year ago. The specific solution we’d like to propose is not something completely new we’ll be building from scratch, essentially we’ll be using a new feature in Git, already built.
Implement issue workspaces on Drupal.org
Issue workspaces are a new way of working on issues that utilize some underutilized features of git, that allow contributors working on an issue to use the power of git to manage change.
It is very similar to “per-issue repositories” - like a per issue fork of the main repository (core, contrib etc).
The feature is Git namespaces that allows for logical separation in the git repository for references (heads, branches, tags), that are isolated from the rest of the repo, yet still share the same objects. Everything is there on the remote server, but everything appears to be different repositories to the client. This allows us to have cheap ‘forks’ of a project per issue, and easily merge those forks back in.
- One namespace per issue
- A per-user, per-issue protected branch in the namespace that only that user can push to.
- A branch pointer to the “Latest” so users can easily pull in the most recent change into their own branch.
- Any number of shared branches for those times when advanced users need to hash out several concepts.
How would the new Git-based workflow work
- When a user wants to work on an issue, they will see instructions to clone the issue workspace e.g.:
git clone email@example.com:project/drupal/issues/123456.git
- User makes their changes
- Instead of extracting and uploading a patch, they commit and push commits to their own branch in the issue workspace
- On Drupal.org a comment will be generated on behalf of the user, that contains their code changes as well as their commit messages. Testbots will take the changeset that was pushed and run tests on it.
- If user needs to add additional explanation of their work beyond the commit messages, they can add a comment.
Real life example from https://www.drupal.org/node/1740492
Legacy patch based workflow does not go away
- Users can still upload patches, download and apply patches, and current tools like dredditor should still work, except no interdiff will be necessary.
- All changes via the new git workflow will also be downloadable as a patch, so this workflow will not change.
- When a user submits a patch, a commit will be created on that users personal branch so that other git using developers can work with those changes in git.
As an additional feature, we would like to add the ability to inline edit work in progress changes in the issue, eliminating cumbersome workflows for simpler edits such as code style, documentation, typo fixing etc.
1. If there is already code changes in progress on an issue, either uploaded by a patch or via a git push to the workspace, then a user will see a button on the issue comment next to the change that will open a code editor (for example ace.js) that will allow the the user to make edits.
2. Once the edit is complete and saved, a commit is created on that users workspace branch and another comment is generated on the issue queue.
Code review in issue workspaces
- As a result of preserving our legacy interface with patches, we can continue to leverage tools such as Dreditor without needing to implement code review and inline diffs as part of the MVP.
- Additionally we can provide an option to compare any two changes in the issue, providing interdiff functionality.
- Reviewing major changes does not mean a 300k patch. it can mean reviewing multiple small commits.
Merging changes upstream
- Button on interface to “squash merge” all the commits into a single unit of work, for core probably, or even a straight merge.
- A maintainer could merge in any branch into their production codebase without squashing too.
- Reveals issues conflicts using git
- Reduces need for “disruptive patch windows”
- Commits that break other changes notify immediately
- Needs Review is meaningful.
History, previous discussions:
A comprehensive summary of Github vs Drupal vs Gitlab : https://groups.drupal.org/node/313068#comment-955593
Github/Gitlab/Drupal spreadsheet: http://bit.ly/1tPt3qS
Git Phase Next: The Promised Land. Slides:http://bit.ly/1qKecLI Presentation: http://bit.ly/1AGiEfI
Implementing per issue repos: https://groups.drupal.org/node/50438
https://www.drupal.org/documentation/git/faq -’what about pull requests’