Workflow iconIn October of last year the Technical Advisory Committee was formed to evaluate options for the developer tools we use on Drupal.org. The TAC consists of Angie Byron, Moshe Weitzman, and Steve Francia, acting as advisors to Megan Sanicki, the Executive Director of the Drupal Association.

The TAC's mandate is to recommend a direction for the future of our tools on Drupal.org. Megan will evaluate this recommendation, make a decision, and prioritize that work in the development roadmap of the Drupal Association engineering team.

What is the motivation behind looking at our developer tools now?

Close followers of the Drupal project will have noticed a trend in the last several months. From Dries' announcement of easy upgrades forever, to the revamp of the project application process, to the discussion about making tools for site builders— there is a unifying theme: broadening the reach of Drupal.

This is the same motivation that underlies this evaluation of our developer tools, and defines the goals and constraints of this initiative:

  • Adopt a developer workflow that will be familiar to the millions of developers outside our community
  • Preserve those unique elements of how we collaborate that have made the Drupal project so successful
  • If possible, leverage an expert partner who will help keeping our tooling up to date as open source collaboration tools continue to evolve

This means looking at a number of elements of the Drupal.org developer tool stack:

  • The underlying git service
  • How we tag and package releases
  • The contribution workflow (patch vs. pull request)
  • Project management workflows (the issue queues and tags)
  • CI integration
  • Maintainership
  • Project pages

If this looks like a tremendous undertaking - that's because it is. But there are some things we already know:

  • Drupal.org should continue to be the home of project pages
  • We should adopt a pull request workflow (and ideally we want to be able continue to accept patches as well, at least in the interim)
  • We should move contrib projects to semver, following core's lead
  • We want to preserve our familiar understanding of maintainership
  • We want to avoid forked code and forked conversation
  • We want to ensure the security team still has the tools they need to provide their service to the community

We also know that whatever decision is made, these changes cannot happen all at once. We'll need to take a progressive approach to the implementation, and focus on the parts of the stack that need to change together, so that we don't bite off more than we can chew.

What options are being considered?

At this time, the technical advisory committee is considering three options as they prepare to make their recommendation. The options are: GitLab, which offers both self-hosted and SaaS options; GitHub, which has recently been adding long-requested new features; or continuing to evolve our custom-built tooling, perhaps via issue workspaces.

GitLab

GitLab is the up-and-comer among Git hosts. GitLab can be self hosted using either their community or enterprise editions, or repositories can be hosted at GitLab.com. Though they recently stumbled, they have been notably open and transparent about their efforts to build a leading collaboration platform.

Gitlab is itself open-source, and has just released its 9.0 edition. GitLab has aggressively pursued the latest in development tools and workflow features, including project management tools, a ui for merge conflict resolution with in-line commenting and cherry-picking, docker registries for projects, integration with CI tools, and even Gitter, an IRC alternative for real-time collaboration.

GitHub

For quite some time, GitHub was the only real player in git repository hosting outside of rolling a custom solution (as we did for Drupal.org). Over the years it has become the home of many open source projects, and while most of those don't match the sheer scale of Drupal in terms of codebase size and number of contributors, there are certainly other major projects that have made their home there.

However, for all of its presence and longevity in the open source world, there are very few options for customizing its toolset for our needs, and we could no longer self-host our canonical repositories. The Drupal project would need to adapt to GitHub, rather than the other way around.
 

That said, in recent months, GitHub has been putting a strong focus on feature development, adding a number of new features including: automated licensing information,  protected branches, and review requests.

Custom tooling

We also must consider that the tools we have now are what built Drupal into what it is today. A great deal of work has gone into our existing developer tools over the years, and we have some unique workflows that would have to be given up if we switched over to a tooling partner. An idea like the issue workspaces proposal would allow us to achieve the goal of modernizing our tools, and likely do so in a way that is better tailored to those unique things about the Drupal workflow that we may want to preserve. However, doubling down on building our own tooling would come at a cost of continuing to be unfamiliar to the outside development community, and dependent on an internal team's ability to keep up with the featureset of these larger players.

Each of these three options would be a compromise between reaching outward and creating familiarity, and looking inward to preserve the Drupal specific workflows that have brought the project to where it is today.

What have we learned so far?

The TAC has conducted their own internal evaluation of the options as well as worked with Drupal Association staff in a two day exploratory session at the end of last year. The primary focus was to identify and triage gaps between the different toolsets in the following areas:

  • Migration effort
  • Project management
  • Code workflow
  • Project handling
  • Testing
  • Git Back-end/Packaging
  • Integrations beyond tools

This initial study also looked at the impact of each option on Drupal community values, and some key risks associated with each.

What comes next?

The next step for the TAC is to make their formal recommendation to the Executive Director of the Drupal Association. At that point, she will direct staff to validate the recommendation by prototyping the recommended solution. Once the recommendation has been validated, Megan will make a final decision and prioritize the work to fully implement this option, relative to other Drupal Association imperatives.

In the comments below, we would love to hear from the community:

What aspects of the way the Drupal community collaborates are the most important to you? What workflow elements do you feel need to be preserved in the transition to any new tooling option?

Comments

sylus’s picture

That being said, I would give a big thumps up to gitlab, if only for how the gitlab ci runner is brilliantly architected and can be leveraged in-house with gitlab itself as the orchestrator. The gitlab ci runner has many "executors" but the docker executor (dind) combined with a registry proxy cache for bringing infra up in near O(1) time has been quite the workflow improvement at least for me.

Promising future ahead!

jonathanshaw’s picture

Yeah, I've been hoping you'll choose Gitlab. It seems like it would have the most potential for experimenting and developing new community tools and ways of working.

benjifisher’s picture

If GitLab is an open-source project, then we have the option of starting with its codebase, developing our own improvements (such as issue workspaces), and contributing them back. This would still be an enormous job, but in the long run it might be easier to maintain than our custom solution.

Media Crumb’s picture

No question we should swap to gitlab.  Considering its current support level and the fact that we can custom fit it to our needs it feels like a no brainer.  

Chicago Dev and Designer. Owner of 2old2play.com and GameStooge video game communities.

Anonymous’s picture

+1 Self-hosted GitLab!

Thanks for the update and all of your work on this, it's exciting to think about how this will help massively ease development of Drupal for existing community members and make it much more easier to onboard new members!

I believe self-hosted GitLab is the best option - I've been using GitLab since last August and I find it has a lot of great functionality and whilst it may not look as polished in terms of design as GitHub, it's certainly a lot free-er which I think is an important issue when it comes to choosing tools to build Free/Libre Open Source Software projects like Drupal on. A couple of people I brought onto one project were wary at first as they hadn't used it, only GitHub - I said it's pretty much the same they said OK and had absolutely no problems using it straight away.

Issue workspaces are a great idea but AFAIK that's all they are at the moment, and I know how hard it is to turn ideas into reality! By using GitLab, as @benjifisher points out, we have the opportunity to develop customisations over time if we find we need to and benefit from the 'quick' start of a complete tried and tested environment which others outside the community may already be used to. Then introducing new, Drupal-specific features sounds to me better than having to onboard to a Drupal-specific environment to start.

Gitter++

I have a big bugbear with the current fragmentation of support in the community at the moment. Much of what made Drupal is the fact everyone collaborates mostly in one space, the unfortunate lack of investment available for tools has resulted in people having to go elsewhere to communicate. I remember trying to get marketing people to use g.d.o - they try to give it a go because it's important to them but soon it breaks down because it's just not usable to any great degree. I think it's important to use this opportunity to try and reign some of that in and bring people back to one space - at least one seamlessly connected even if not physically.

This brings me onto my final point - decentralisation. I've said for a long time the centralised nature of *.d.o is a great benefit but also a big issue, a central point of failure. Going forward, it would be great to be able to decentralise communications - for sure you can fork projects on GitLab but if you only have one place to communicate with others then you will end up growing more silos and central points of failure. I'm not sure how Gitter works but I have been watching and playing with matrix.org. I believe there's an opportunity here to lessen the load in terms of costs too as DA supporting organisations and others could provide hosting, in return of course they get visibility on things that directly help the community. There's some pretty good clients for these decentralised services so people can choose their favourite interface, recommended ones could be linked to. I'm using riot.im which is pretty nice and there's a #drupal channel in there where someone has linked the slack drupal channel in to the drupal-uk one. 

Decentralisation also offers the opportunity for there to be a choice of rules/CoC - I simply don't believe the current efforts of creating a community-wide CoC will work in the long run because that's essentially a one-world government and you're never going to please all the people all the time. It already doesn't work as I know many who have avoided any community interaction for many years - we have an opportunity here to change that and open up environments where others may feel more welcome and start joining. One can never have enough contributing community members especially with a project this size!

Thanks once again and I for one look forward to Drupal's new horizons!

xmacinfo’s picture

Gitlab is the clear choice for myself and where I am working.

Rolling up a self-hosted solutions is a no brainer and we even host multiple instances of Gitlab, where servers are secured and require the Git server being set in the same VLAN.

Furthermore, Gitlab welcomes contributions and incorporates them in a very quick pace.

Drupal and Gitlab are a great match (in heaven).

willhallonline’s picture

I would go GitLab all the way. The tooling is more comprehensive (GitLab CI is excellent, like an inbuilt Travis CI). I use GitLab.com personally and work has had GitLab for a couple of years. I've been really pleased with it (moved personal stuff from BitBucket). I have also been in contact with a number of different employees and found them all really responsive to demands and enquiries.

cferthorney’s picture

I almost feel a sheep for saying +1 to Gitlab, but I will anyway. I do like the idea of customising the community edition of Gitlab and contributing back (Although I am unsure how many would feel able to, from a code perspective at least, given Gitlab is written in Ruby).  I also remember someone saying that Gitlab had offered to work with the Drupal community surrounding moving to Gitlab (Though I cannot find the source for this belief at this time), which could mitigate any issues the community has with Gitlab's standard process.  I've migrated my company's and my personal development to Gitlab.com from Github.  Edited for clarification

David Thorne
Web Developer

wturrell’s picture

This is a modified version of something I emailed back in February and @hestenet kindly said he'd forward on but also asked me to post in the previous blog, which I never did (apologies).  So this isn't specifically in reply to anything anyone else has said (though for the record I've not used GitLab).

TL;DR baby / bathwater:

As relatively new user) I actually really rather like the current tools. Whilst the process is certainly "different" from, say, GitHub it didn't take me very long and I don't find many pain points.

I should note that I've been working nearly exclusively on issues in core, not contrib modules - but I happen to quite like the separation between patches individuals are working on and the main Drupal repository - it feels "safer", it's simpler to explain to people and and less disruptive to others if (when) you make a mistake.  Also, how would branching work on GitHub/Lab, given we may have more than one proposed solution for an issue? (c.f. comments re forks of forks on issue workspaces discussion).   

I've found downloading, applying and uploading patches and interdiffs pretty quick; crucially, creating a new local branch can be done in seconds, I'd hate a solution that required downloading a full repository every time. That said, we do have a bit of a problem with unnecessary 'busy work', where people have learnt how to reroll a patch etc. by rote and will blindly do that without taking the time to attempt to read or understand the state of the issue, which creates more work/noise for everyone else, so any improvements that would eliminate some of those routine jobs in the first place might help.  What really takes the time is mentally assimilating everything in an issue when working on a patch or reviewing it (if we could get everyone using the issue summary template..)  

Off-topic: the standard of reviews is all over the place and I think a review template might also help.  The most useful tips I've learnt are those I've found in ad-hoc, off-site blog posts and videos from core contributors.

Tags: we use them for everything and the documentation is incomplete - inline descriptions or the ability to hide/archive/merge old tags might help.

The only time the current tooling really gets in the way for me is when pages are slow to load - e.g. #665790 (Redesign the status report page) with it's ~500 comments, which also insists on loading on page 1.  Although it could be faster, overall I've been really impressed with the speed/uptime of drupal.org

I'd appreciate an "inline" view in the queue that let you expand an issue without leaving the index, so you could see it's most recent comment, with a pager to go back and forth.  

Another wish: an inline JS preview when you write a comment/update the issue summary to avoid waiting for a reload (like the Write/Preview tabs in GitHub).

I'd hate to lose the tight integration between the issue queues, dashboard, profile, contrib mentions, groups, jobs db, DA membership and so on.  I'm a big fan of metaphorically eating your own dog food – I'd like to think the tooling would be even more powerful if/when drupal.org is upgraded to D8. Some may have forgotten or not be aware of the existing benefits of dreditor, simplytest.me etc.

I wouldn't want to throw all that away just because we felt we needed a fancier UI to attract the (hypothetical?) "cool kids" – for me the test is will our processes be faster and have fewer mistakes. I'm not convinced anything we do can reduce the learning curve all that much; people are still going to need to familiarise themselves with all our policies, conventions and coding standards (and be good at git). 

I would be worried about the expense of a proprietary third-party solution..  To me, what little money the DA does have is better spent on engineering support, and unfortunately the DA was forced to reduce the number of staff in that area.

cburschka’s picture

  • How we tag and package releases

I'm not sure if this is in the scope of this discussion (or if it's already in progress elsewhere), but the activation of the composer endpoints has exposed the badly broken versioning that we still have for contrib.

It doesn't really matter what the git tags themselves are named - the 8.x-, 7.x- prefixes are a fine way to allow multiple core versions to co-exist. The same goes for the labels given to releases on the website.

But the composer release numbers are currently mapping 8.x-X.Y to X.Y.0. There needs to be a way to use actual semantic versioning and create an X.Y.Z patch release eg. by naming your tag 8.x-X.Y.Z, which was not possible the last time I tried. It has been discussed to map 8.x-X.Y to 8.X.Y instead, but that's even worse in my opinion. Semantic version numbers have three parts for a reason - BC-breaking changes, extensions, fixes. Contrib projects need all three, within a single core version.

Mixologic’s picture

 Tim commented above that we're working on this:

We should move contrib projects to semver, following core's lead

The reason composer is mapping the release numbers from 8.x-X.Y to X.Y.0. was to preserve the historical "Major API compatibility" tag (7.x/8.x etc), and simultaneously allow us to transition to semantic versioning behind the scenes by adding 8.x-X.Y.Z.  In an evergreen world of modules that *may possibly work* for 8.x or 9.x where we no longer have what is essentially completely new products, we might even see the eventual elimination of the "Major API compatibility" tag, which was more for humans in a different era.

There are several blockers to this being a possibility. The first being that drupal core itself cannot handle semver dependency specifications: https://www.drupal.org/node/2641658.

dsnopek’s picture

For the past 1.5 years, we've been using GitLab internally and I love it waaaay more than GitHub. Also, GitLab CI is double plus amazing!

The Panopoly project has been using Travis-CI for 3 or 4 years. Many distro's built on on Panopoly followed suite, including some of my consulting clients. It gets the job done, but it's painful to debug problems or hack the build.

Using GitLab CI (with the Docker executor) for last 1.5 years has been soooo unbelievably refreshing!! I'd almost say using GitLab is worth it for GitLab CI alone. ;-)

Anyway, I'd be 100% behind using GitLab! (Or even continuing to develop Drupal.org custom tools - just please no, not GitHub...)

colan’s picture

I'm glad the idea posted four years ago is finally gaining traction!

imadalin’s picture

I'm also confident in having Gitlab and adopt the common popular way to work and contribute to projects. Sometimes just to be able to do a really quick fix, forking and editing quick, doing a PullRequest that people can comment on it, etc. That workflow is totally more productive than we currently have.

Also, semantic versioning for contrib space should be next priority. But from the way that many people want to manage their projects, compared to how major projects are handled (Symfony, Zend Framework etc.), I don't expect promises from many to offer LTS for more major versions on a platform number, but that would be the professional way and might in time make a huge difference when evaluating modules that offer this vs modules that don't. That would solve partial even the problem of modules that duplicate functionality, which I expect to happen now that adding a contrib project on Drupal.org is open to anybody, as in time, modules that offer both LTS major versions among continuous new features in new versions will make a difference and stand out.

Would be the concept of LTS versions in contrib modules be a next thing to think about it and how to highlight projects that offer this?

Slovak’s picture

+1 for GitLab. We've been running a self-hosted version with an array of developers, clients, and deployment scenarios. In a number of project they even adopted for further internal use.

jurgenhaas’s picture

So much has already been said about Gitlab and I can only share my excitement to see this reaction. Gitlab is a perfect fit. Product is great and company's attitude is amazing.

I recall some concerns about lack of performance. But the latest release 9 has made a big difference and their discussions emphasise their commitment to ongoing improvements. 

Have you seen the module DRD? A must-have for Drupal administrators.

djalxs’s picture

Having used Gitlab, it is not the most resource-friendly option and can be quite slow, but the extensive feature set is a big plus.

I am noticing that nobody has mentioned Bitbucket in the post or comments? It's a great Git SaaS and has a lot of plugins and custom functionality that can be enabled.

They have a CI pipeline feature that is currently in beta phase but it works well.

I believe the Bitbucket option should definitely be on the table at this time as many people will already have account with them, they have unlimited private repositories and as previously mentioned, a whole multitude of extensibility options.

develCuy’s picture

Great move and feedback. Also, not a call for DA alone, since changing tooling will change how we contribute to Drupal for good or bad, so everyone involved in core and contrib should care about this move, specially leadership.

Preserve those unique elements of how we collaborate that have made the Drupal project so successful

Very important for sure, maybe a survey will help identify how ready the community is to adopt Gitlab. For sure many people knows about gitlab already but ignore how deeply and how far from tipping point we are as community. The way we collaborate can evolve for sure, not with better tooling alone but with the collaboration of everyone, starting from ourselves.

P.S: Go for Gitlab!! :D

--
develCuy
Drupal Full-stack hacker (D5, D6, D7 & D8)
See you at Drupal Picchu 2017, Oct 30th - Nov 3rd, Cusco - Perú

markusa’s picture

PR style contribution for contrib modules too!

generalconsensus’s picture

having used Gitlab in a medium size team capacity I fully support the transition, Gitlab is the logical successor for us to hopefully help increase contributions and increase the number of contributors

Jose Reyero’s picture

First I had no idea we needed to switch our development tools -yeah, I'm usually late to everythig, sorry. Now I've learnt it I have a few questions.

1. Since Drupal at its core is a software developing community, it's not only the software we develop but the process we use to develop it what makes us unique. Do we really want to "outsource" that process? Whatever solution may be customizable up to some point, so beyond that point we may need to update "how we work" because updating the tool may not be an option.

2. Yeah, Gitlab is cool, I've used it for some projects, I like it, but... Can someone point me at a real life example where Gitlab is being used for a project with a few thousand developers? (Note it being used by big companies doesn't mean these companies have 1000 developers working on the same project).

3. Gitlab is an open source software but AFAIK the project is owned by a Company. (I may be wrong though..?) I have nothing against companies just noting the difference here. And btw, unless we can run with the Community Edition (feature limited), are the others free open source software we can customize, share, etc, as much as we want? (Just asking)

And last but not least, it's not my intention trolling here just this is something I don't quite understand:

4. Why is the DA deciding on the tool we are using? I thought they should be committed to support whatever tool *we/you* developers decide to use.

Wouldn't all developers voting -define "developers" as you want- make more sense?

markusa’s picture

I think if you want to have thousands of collaborators, Gitlab/Github is definitely the way to go.  I recently become co-maintainer of CiviCRM LTS, and the CiviCRM community has used Github for quite a few years.  I couldn't imagine juggling patch files, that is just medieval. Whatever happens the Github/Gitlab PR model is superior.  We have 100s of contributors, and management is quite "easy" with Github.  PR's can be commented on, specific blocks of code can be commented on, and merging/resolving conflicts can generally be done through the interface.  Being able to comment on PRs separately from the main issue is a huge boon, as you may have multiple PRs from multiple contributors going toward resolution of the issue, and each comment stream is kept separate.  Its much easier to organize the development conversation.  CiviCRM has Github hooked in with a Jenkins based testing infrastructure for quality control and to reduce regressions. CiviCRM uses JIRA for issue queue management, and simply putting the issue number in the PR, makes the PR show on the JIRA issue page.  I'm sure that something similar could be worked out for Drupal. The company I work for uses Gitlab for internal development and its basically the same thing, you're own private GitHub.  I sure do hope Drupal moves to the PR (pull request) style model, I think you'll see an increase in contribution.  If it becomes possible to offer this for contrib modules hosted on D.O as well, that would just be out of sight awesome.  A lot of contrib module project nowadays are developed on Github collaboratively, and then just pushed to Drupal when its ready, because managing the development is so much easier. By the looks of things so far in the comments here, its "Yay +1 for Gitlab/Github"....So Go DA! please do Github/Gitlab

Jose Reyero’s picture

Thanks, this is interesting though it doesn't really address the "thousands of contributors for a single project" concern.

About JIRA, I'm sorry you have to use that thing -me too for some projects... :-(_

develCuy’s picture

The DA pays the bills and payroll for d.o infrastructure and redesigns. They already paid a few million USD so far. Also, they forecast reduced income from Drupalcon which is just enough for DA's budget. In short the DA is forced to migrate current tooling into something more cost-efficient without hurting or even improving current development process. Not an easy task...

--
develCuy
Drupal Full-stack hacker (D5, D6, D7 & D8)
See you at Drupal Picchu 2017, Oct 30th - Nov 3rd, Cusco - Perú

Jose Reyero’s picture

The DA pays the bills...

Well, I don't thing you got it right, but anyway, since I understand you don't speak in the name of the DA, nothing else to say.

Thanks for replying and posting your opinion though.

develCuy’s picture

Relax boy and realize that Drupal.org infrastructure has a big budget, maybe you ignore it but it is real. If developers come up with X tool that the DA can't afford to pay, forget about it. That simple. Now look:

According to the official 2014 Leadership Plan and Budget, the Drupal Association spent US$ 1.4 million on Drupal.org alone on 2014, representing 23.3% of expenses (6 million). Now make numbers yourself and realize if all that money would be spent better if we could migrate most of drupal.org from custom in-house stuff to efficient dedicated services.

More info in my original post: Is Drupal the right tool for drupal.org and the project itself?.

You are welcome ;)

--
develCuy
Drupal Full-stack hacker (D5, D6, D7 & D8)
See you at Drupal Picchu 2017, Oct 30th - Nov 3rd, Cusco - Perú

connorshea’s picture

(Community Advocate at GitLab, obviously biased but I wanted to answer)

I can't answer all of these, but:

2. VLC uses GitLab (https://code.videolan.org/videolan), though that's not 1000s of developers as far as I know. GitLab CE is developed on GitLab.com, which is operated as an open source project (https://gitlab.com/gitlab-org/gitlab-ce), though that may not count based on your criteria since we're technically a company. They aren't all active, but we do have over 1000 developers who have contributed to the project over the years.

3. It is indeed run by a company, and we do reserve the right to reject feature requests/changes, but as it's MIT Licensed you can always fork it and we generally try to incorporate features if they're requested enough. The project is written in Ruby on Rails (with some Go components), so the Drupal community may not be able to contribute as easily to it, I suppose that's important to note.

You can always look through our issue tracker (https://gitlab.com/gitlab-org/gitlab-ce/issues) or handbook (https://about.gitlab.com/handbook/) to see how we do things. We also have our monthly Retrospectives/Kickoffs open to the public, you can join them if you want or watch them later on YouTube :) We try really hard to be open and transparent about as much as we can.

Jose Reyero’s picture

Thanks, yes, these are some answers.

About 2. yes, I was just looking for examples of 'projects with many developers', for which it doesn't make a difference whether they're run by a company or not. I just wanted to make the point that "it being used by huge companies" doesn't mean "it is being used for huge projects inside these companies". Actually I don't think any of these companies have many projects, if they have any, the size of Drupal, just looking at the number of developers.

Again, I'm currently using Gitlab for some projects and I think it is a nice tool. Still I am concerned about, not really Gitlab, but the MR-based workflow which everybody seems to love -me included-, but I've just seen it working for reduced groups with centralized decision making. So I'm mostly wondering about advantages / disadvantages of using our  current issue tracker which requires the big group that may participate in an issue to focus on a single patch vs having a ton of different MRs scattered all over the place.

connorshea’s picture

(I'm a Community Advocate at GitLab, this isn't a vote for obvious reasons)

Awesome to see all the support here :D If there are any questions you'd like to ask I'd be happy to answer them. And if you do end up going with GitLab I can see about organizing a call with some other team members on the project to discuss migrating :)

Elijah Lynn’s picture

Welcome to the Drupal community Connor!

The biggest hurdle we as a community are going to have is developing a solution that can live in parallel and/or compliment our patch based workflow. I don't know if GitLab has any ideas about that but one of the caveats to switching to a normal Merge Request workflow is that if a user abandons a MR then we have to fork it into a new MR and lose the context. One solution would be to somehow "share" a Merge Request and I understand that that would traditionally involve giving a user access to that source branch. That doesn't seem feasible but maybe your team has had this hurdle already and has some thoughts?

We gain a lot of value collaborating on code and with a MR you don't really collaborate that much.

tldr; We need a feature called Collaborative Merge Requests.

-----------------------------------------------
The Future is Open!

connorshea’s picture

I've shared this with the Product team and they seem to like the idea, feel free to open an issue in CE to discuss it more :) https://gitlab.com/gitlab-org/gitlab-ce/issues

We have one that could help with this, though I'm not sure how inconvenient it'd be to deal with all these different remotes and branches when contributing via someone else's fork: https://gitlab.com/gitlab-org/gitlab-ce/issues/22292

Elijah Lynn’s picture

Good to hear. I will think it over more before I open an issue. 

-----------------------------------------------
The Future is Open!

Elijah Lynn’s picture

FYI, I am not the lead on this initiative and have been mostly watching from the sidelines. This is not exactly what we need but it is one piece of the larger initiative, IMO. 

-----------------------------------------------
The Future is Open!

Elijah Lynn’s picture

I think this could really work well. Especially since they just acquired Gitter and will be open sourcing Gitter in the coming months as well. I think it might help with the IRC vs Slack debate around Drupal.

Imagine every module just having a quick link to launch a chat, and that chat would be part of a larger, inclusive, coherent chat?!

Having GitLab will increase driveby contributions, a lot. I do still want a way we can all work on a patch file, sometimes Merge Requests (GitLab calls them the more logical, merge request vs Github's Pull Requests) get abandoned and all the discussion and work has to be forked into a new MR without the context. Not sure how this will work but since it is open I think we can find a way. 

I think the Github boat has sailed and I don't see a strong case for going to Bitbucket since it is a closed tool as well. 

I am really excited about this and look forward to seeing this evolve!

My advice is to keep the momentum going. This thread is on fire right now and we need to keep that energy alive and grow it even more. 

-----------------------------------------------
The Future is Open!

Fabianx’s picture

I am also +1 for GitLab - as long as we keep the patch based workflow, as well.

I think we in general have two collaboration models:

- One main developer + PRs by single people (classic GitHub style), mostly true for contrib

- 30 people working on one issue and refining patches over time, mostly true for core

We also have to think about how we distinguish architecture from implementation.

In general sometimes issues are less about the patches itself, but more about the approach and architecture and there issues are sometimes superior.

I also found that within GH it is almost impossible to sometimes find old information.

So I like to have a history of 'patches', which would translate to a history of PRs, e.g. you not only commit to your branch, but you also actively promote / show / present a certain tag / commit hash of that branch, so that later persons can see the discussions within the original context. I feel preserving that context is what makes patches the most valuable. (That part is of the idea of issue workspaces, but is key IMHO.)

That also helps with security. In theory it is possible with push based workflows to push a commit at the exact time before the merge / pull happens - though I am hopeful that both GitHub and GitLab have mitigations for it.

Or less malicious: Last minute changes that the committer did not review, yet that slip in.

TL;DR:

- +1 to GitLab with customizations

- Keep patch workflow somehow

- Ensure that history of presented PRs is preserved together with the context of the relevant discussion

- Find a way to discuss architecture outside of one specific PR (unified discussion, history)

- Treat PRs more like "patches", which resemble a discrete change set at a certain time

Elijah Lynn’s picture

Well said! The Issue Workspaces idea was/is really appealing when I heard about it some time ago. 

-----------------------------------------------
The Future is Open!

connorshea’s picture

>That also helps with security. In theory it is possible with push based workflows to push a commit at the exact time before the merge / pull happens - though I am hopeful that both GitHub and GitLab have mitigations for it.

That's preventable with GitLab by requiring that the CI builds pass before merging the MR. When a new commit is pushed, the MR has to be re-tested or it can't be merged. If someone hits "Merge when jobs succeed" and the author pushes a new commit, the merge is cancelled.

As for the patch workflow, I'm not sure I understand what you mean by that, could you explain (or link to an example)?

pwolanin’s picture

Agreeing with the general sentiments here.

I find the pull-request workflow anti-collaborative and frustrating.

I know webchick has often remarked on this point - that we can get a "swarm" of different people to work on an issue in a series of patches and come to a better net solution.

I don't care a lot about the underlying git hosting solution.  Would be great if we had some easy way to allow simple online edits, but lets not lose the most important collaborative elements.

---
Work: BioRAFT

markconroy’s picture

We've used GitLab for some of our sites and I really like it. The CI is great there (as long as someone else sets it up for me!!!). Also, the attitude of the GitLab folks seems to align quite well with the attitudes of Drupal folks in terms of being open source and willing to iterate.

Also, just to note Matrix.org have offered to help us get set up on their platform. See here:

https://twitter.com/matrixdotorg/status/853282565528182788

Elijah Lynn’s picture

Matrix = An open network for secure, decentralized communication (also open source, https://github.com/matrix-org/)

-----------------------------------------------
The Future is Open!

Elijah Lynn’s picture

I think the key to a chat solution is a low amount of friction.

IRC = High Friction (medium with IRC Cloud but only if user already has IRC Cloud account, plus $5/user to make it like it should be)

Slack = Medium Friction (the whole invite thing is awkward) + closed source, no open source community free plans, 10K message limit

Gitter = Low Friction (the gitter button on a Github page is incredibly low friction, time to interaction is seconds). Check out https://github.com/hechoendrupal/drupal-console for an example. Click here to launch Drupal Console Gitter chat

Matrix = ? (curious to see how we could implement as easily as a Gitter button on a page)
Also something I just found is https://github.com/matrix-org/matrix-appservice-gitter which seems like a bridge for those already on Gitter and moving to Matrix. Not sure we would need it if we start out with Gitter or Matrix. 

-----------------------------------------------
The Future is Open!

jurgenhaas’s picture

Not to forget Mattermost (https://about.mattermost.com). It's very low friction because it is highly integrated with GitLab and from UX point of view it's similar to Slack. However, it's free and OSS and if you wanted it simple to self-host as well.

Have you seen the module DRD? A must-have for Drupal administrators.

jonathanshaw’s picture

The issue summary in the "Issue Workspaces" issue says it well:

GitHub's Data Model: Fragmented Social Coding

  • Code changes on GitHub are pull requests.
  • Both pull requests and issues have comments.
  • If someone else wants to work on the same code, they fork a pull request.
  • When you fork a pull request, you fork a conversation. It is now in multiple places.

Drupal.org's Data Model: Collaborative Social Coding

  • Code changes on Drupal.org are patches.
  • Patches live in comments on issues
  • If someone else wants to work on the same code, they create a new patch.
  • Conversation happens in one place.

As I understand it,  it seems like we would have 2 choices with Gitlab:

1) Have multiple PRs (well, 'Merge requests' in gitlab speak) per issue, and police a convention that architectural comments are made on the issue and detailed code comments are made on the PR. This has the advantage of making the issue easier to read. But in practice it could well be a nightmare: the conversations would get fragmented, and also making a quick fix to someone else's PR would involve starting your own based on it, which is probably more complex than would be ideal.

2) Have everyone be members of the project with some level of commit access,and have 1 branch per issue (normally), so issue collaborators are all working on a common branch.

#2 seems like the only way that's likely to be acceptable for Drupal. Is it viable with Gitlab?

dsnopek’s picture

GitLab has the idea of "protected branches" which only people of a certain role can push to.

So, I could see this working that any branch that was used for releases (ex. release/* <-- you can use wildcards!) was protected and only the maintainers could push to them, and then anyone with an account on the GitLab instance could create other branches for particular issues.

This could lead to people creating all sort of random branches, but that no worse than someone creating all sorts of random patches :-)

Elijah Lynn’s picture

I was thinking of #2 initially but the issue with that is that there isn't freedom to "try something new". Drupal currently does well by having many different patches in flux at any given time. Sometimes #34 works better than #61 & #67 etc. 

Some variation of #1 seems like the path forward to keep that model alive. 

-----------------------------------------------
The Future is Open!

colan’s picture

2) Have everyone be members of the project with some level of commit access,and have 1 branch per issue (normally), so issue collaborators are all working on a common branch.

To prevent too many branches from being created, how about allowing folks to create only one branch per issue (as suggested above), named after it (something like issue-123456), which either gets auto-created or gets created with a button push?  Everyone with an account could push to it.

Either that, or allow lots of branches.  Then maintainers can delete without too much work by hitting the Delete Branch button from the UI.

catch’s picture

There are certain things it's easy to do with an issue/patch workflow, that github makes very difficult. I haven't used gitlab though.

1. Quickly compare competing proposals on the same issue.

Sometimes this can be as simple as the same person having two different suggestions and posting them both for review.

2. Taking over an issue from someone else.

We have multi-year issues with tens of participants. However, even two week issues may have a patch posted by one person, improved by a second person, RTBCed by a third and committed by a 4th, without the first person necessarily returning to the issue at all.

On github that second patch requires forking the pull request and then closing both eventually.

3. Reviewing discussion.

The inline nature of code review on github allows for minor changes to be made and forgotten. However if you have a conceptual issue with part of a patch, minor changes can remove those. This has changed a bit with the 'start a review' feature github rolled out recently but it's still not entirely resolved for me. Also discussing sections of a patch becomes hard, discussing two sections of a patch etc.

4. Test-only patches. On github, travis will test the most recent commit, but on Drupal.org we like to see red for a test-only patch and green for the full patch.

This isn't an exhaustive list but these are (mostly social) aspects of the way core development works that while they aren't perfect I do sometimes miss on github for custom projects - and custom projects have about 1/100th the number of people working on them.