Technical Advisory Committee formed to modernize developer tools

By megansanicki on 18 October 2016

Workflow icon At DrupalCon Dublin, I spoke about The Association’s commitment to help Drupal thrive by improving the contribution and adoption journeys through our two main community assets, DrupalCon and Drupal.org. You can see the video here.

One area I touch on was my experience as a new code contributor. Contributing my patch was a challenging, but joyous experience and I want more people to have that feeling—and I want to make it as easy as possible for others to contribute, too. It’s critical for the health of the project.

At the heart of the Drupal contributor community are our custom development tools, including the issue tracker, Git repositories, packaging, updates server, and automated testing. We believe there are many aspects of Drupal’s development workflow that have been essential to our project's success, and our current tooling reflects and reinforces our community values of self-empowerment, collaboration, and respect, which we seek to continue to uphold.

It’s time to modernize these developer tools. To support the Association with this objective The Drupal Association created a Technical Advisory Committee (TAC). The TAC consists of community members Angie Byron, Moshe Weitzman, and Steve Francia, who is also our newest Drupal Association board member. The TAC acts in an advisory role and reports to me.

Building off of the work the community has already done, the TAC is exploring opportunities to improve the tools we use to collaborate on Drupal.org. The crux of this exploration is determining whether we should continue to rely on and invest in our self-built tools, or whether we should partner with an organization that specializes in open source tooling.

Our hope is that we will be able to bring significant improvements to our contribution experience faster by partnering with an organization willing to learn from our community and adapt their tools to those things we do uniquely well. Such a partnership would benefit both the Drupal community—with the support of their ongoing development—and potentially the broader open source community—by allowing our partner to bring other projects those aspects of our code collaboration workflow.

The TAC will use a collaborative process, working with staff and community to make a final recommendation. The TAC has already begun the process and has some very positive exploratory conversations. The TAC and staff will be communicating their progress with the community in upcoming blog posts.

Comments

Glad to hear it!

cyb_tachyon commented 18 October 2016 at 21:20

As far as I'm aware, here are the people I know are actively working on better Drupal project build tools (local development side of things). You may already be in contact with them (I think most are at Acquia), but I wanted to make sure somebody had at least reached out to them at this point.

https://github.com/joestewart
https://github.com/grasmash
https://github.com/danepowell
https://github.com/geerlingguy

Best of luck and thanks for all you do!

Specifically, tools on Drupal

drumm

he/him

NY, US

commented 19 October 2016 at 15:35

Specifically, tools on Drupal.org like the issue queue, Git, releases, etc. We don’t have plans to attempt standardizing the fast-moving world of local development infrastructure.

Gitlab

attiks commented 19 October 2016 at 07:19

This discussion might be of interest as well, https://groups.drupal.org/node/313158.

It's a bit dated since gitlab improved a lot in the mean time.

Attiks - Print and web design

hallelujah, hallelujah!!

ricardobeltranl commented 19 October 2016 at 13:09

I am glad to hear this initiative, please bring in the feelings of people all around drupal world, even if they do not use Drupal.org.
If you listened to enzo's keynote last Drupal Con you should recognize him as a longterm advocate for this effort.
This initiative should go beyond development tools and include internationalizations efforts to Drupal.org, our tools are deprecated and are kicking away a new generation of developers.

I wish you the best and I'm looking forward seeing the work of this new committee transformed into more and new people coming to Drupal.

This sounds good, especially

wizonesolutions

he/him

English

Norway

commented 19 October 2016 at 13:36

This sounds good, especially right now when the Association is struggling financially and pretty much can't invest in our tools much anyway. A partner organization probably could. There's probably some tweaking involved in making the deal sweet for all sides (such that they partner with us long-term), but that's what the TAC is for.

Support my open-source work: Patreon | Ko-Fi | FillPDF Service

There will be some cost

drumm

he/him

NY, US

commented 19 October 2016 at 15:39

There will be some cost regardless, integrating with or migrating to a 3rd party is also work that would take time. The Association is increasing financial stability again; this committee will help see what needs to be budgeted.

Great answer for the big question!

DevElCuy commented 19 October 2016 at 16:07

I love to hear this news from you Megan! This is not a new topic but Dublin helped things focus on the change we need.

This is my post recommending different OpenSource toolting for each of the areas you mention:

https://steemit.com/drupal/@develcuy/is-drupal-the-right-tool-for-drupal...

I hope the TAC takes some time to look at it.

Also, would like to know what will be the communication means with the TAC, since I'm pretty interested on participating actively in this process, as part of the broader community outside US & Europe.

--
develCuy
Hiring the next generation front-end devs, contact Dilygent

Gracias

moshe weitzman commented 20 October 2016 at 12:42

Hola develCuy. Gracias por ese blog. We definitely read it and really like the ideas there. We are reading al the tips posted here and on Planet.

Its going to take a lot of knowledge, cooperation, trust, and acceptance from the Community to get this project done. I know we can do it, and end up in a far better place. As for communication, expect a blog post in the coming weeks from the TAC. We will outline there how best to chime in with ideas, caveats, etc. In the meanwhile, feel free to send a message via the Contact tab to any of the 3 people listed in Megan's post.

Reviewboard / Phabricator /

droplet commented 20 October 2016 at 07:48

Reviewboard / Phabricator / Gitlab

Not very familiar with these

GiorgosK

Chios, Greece 🇬🇷

commented 27 January 2017 at 09:37

Not very familiar with these tools but there was another thread about phabricator integration

https://groups.drupal.org/node/313218

------
GiorgosK
Web Development

Github integration

chriscalip commented 20 October 2016 at 23:28

It would be nice to have usable and holistic github integration

Willy Winger here to complain again․

Liam McDermott commented 22 October 2016 at 05:41

Sadly I suspect this means forcing everyone to use a proprietary 3rd party service to contribute to Drupal, which means handing over control of substantial amounts of project infrastructure to a for-profit service that could: go down at any time(as we have seen today), just decide they don't want us on their service (this has happened to other projects), or stop being the code platform du jour.

It is also impossible to contribute changes to a proprietary service, there is the network effect of forcing Drupal users to use it, which is also an effective endorsement and advertisement for that service.

As we saw on The Drupal.org Complexity it doesn't matter what criticisms are raised, the people raising those criticisms will be called names (like 'dinosaurs', 'afraid of change'); replies avoid the point, patronisingly dismiss them, or make grandiose yet utterly unproven claims: Loads of new developers contributing drive-by one line patches! Everyone else in the Universe uses GitHub (so we should all jump off that cliff too!)

Change is good and much needed, but the strengths of what we have should be properties of a new system, or we won't get anything better we'll just get different positives and negatives. Some strengths of the existing system that Github (for example) lacks:

There's no cognitive barrier (caused by a different site/look and feel/functionality) between the user's area and the developer's area that will put off new contributors who started as users, i.e. let's not pander to new contributors coming from GitHub at the expense of those coming from Drupal.org
What we have is tightly integrated, a set of different services on different domains and different designs will cause confusion for any new contributor (as they work out the purpose and orthogonality of those services with each other and Drupal.org).
Free software, this should be non-negotiable.
Full-featured issue tracker that fits Drupal exactly (though it could do with some UX work).
A tailored Version control tab, among others, that provides instructions on how to contribute to any module.

Gitlab is probably the only 3rd party software that provides the functionality many developers expect while also being Free, sadly there's still the difficulty of erecting a barrier for contributors.

Perhaps Gitlab could be themed like Drupal.org (and with single sign-on) the barrier for non-developers wouldn't be so high as to stop them contributing. There are big hurdles to this however: Gitlab currently lacks the ability to be themed and even if it could, it would be a heck of a lot of work.

Also important is the question: what is Drupal's audience? Is it developers who already use Github (Enterprise devs?), business owners/hobbyists running their own sites, both? The former is the contributor from Github, while the latter is the contributor from Drupal.org.

Edit: I failed to comprehend what was being said:

whether we should partner with an organization that specializes in open source tooling.

Basically says retooling will not use a proprietary service, so that part of my post is wrong, sorry.

Our hope is that we will be able to bring significant improvements to our contribution experience faster by partnering with an organization willing to learn from our community and adapt their tools to those things we do uniquely well. Such a partnership would benefit both the Drupal community—with the support of their ongoing development—and potentially the broader open source community—by allowing our partner to bring other projects those aspects of our code collaboration workflow.

Since they have offered multiple times, I'm going to wildly speculate this means Gitlab. On balance (particularly since they seem willing to work with Drupal and make changes to Gitlab) this seems like the best balance of providing a service developers expect, with pull requests and all that, while not alienating normal users from Drupal.org and providing full featured issue queues. This is fantastic news.

It only says that they are

klausi

he/him||they/them

German

🇦🇹 Vienna

commented 22 October 2016 at 07:54

It only says that they are searching for an organization specializing in open source tooling. Github is such an organization that provides tooling for many open source projects.

Gitlab: while the feature set is really nice and the system powering it is open source it is also quite a bit slower than Github. I evaluated gitlab.com (the free cloud version) for my company and it seemed that it just couldn't handle the traffic it receives. Slow git pushes/pulls, slow page responses in the UI, very slow responses to actions in the UI (rebase branch takes minutes for more complex projects). The only response you get from Gitlab regarding that is that you can host a Gitlab instance yourself or use https://githost.io/#pricing .

So at this point our choice was to use Github because it is much more popular and has proven to properly scale on github.com.

Remember when we discussed the migration from CVS to Git or Bazaar in 2010? IMO it ultimately came down to deciding for the more popular product which was Git and which was the correct decision in hindsight. Github is the most popular code collaboration platform right now, so this is important to keep in mind even if it is proprietary.

There is also bit of history to the Github discussion already, see https://groups.drupal.org/node/313068

Gitlab speed

attiks commented 22 October 2016 at 08:19

While gitlab.com might be slow, self hosted instances are fast even on modest hardware. The main benefit is that we can control it and if we want to go the fancy way we can use the geo extension and run it distributed.

To be honest no idea how any of them will act when we add 30.000 repos and 100.000+ users.

The main benefit of gitlab is that they move fast and are completely open to any feedback, GitHub evolves quite slowly.

Attiks - Print and web design

Yeah. A tool with online

droplet commented 25 October 2016 at 17:55

Yeah. A tool with online editor also important. For example some reviewing work:

https://www.drupal.org/node/2782915#comment-11747633

I believe the developers will fix them directly if there's an online editor

Open to change

ricardobeltranl commented 23 October 2016 at 17:06

Liam,
You have very strong arguments, I respect and agree broadly with your point of view. However, IMHO it's urgent to open this discussion with a commitment to change and analyze whether it makes sense to reinvent the wheel all the time and support our inventions forever.

This should not only include what tools should we use and why. Hence, the discussion is about what DA aims for and where it should spend time and money.

Slack, is a good example of what you will never accept according to your rules, even if I agree with your point of view, the pertinent questions is whether the developers will accept to use what we say or they will just go away and use whatever they prefer to support any other CMS with a more flexible community.

I think we should consider relaxing some rules and select with more wisdom which battles do we want to fight.

If we keep forcing having Drupal.org as it is, we cannot use our strengths to beat our competitors in Web Content Management market, which is by far more relevant to me than using IRC or Slack rooms to support drupal discussions.

PS: I will never call you a dinosaur or something similar. We should stick to our arguments and do whatever is the best for the Drupal Community ;-)

IRC, Slack and others

fgm

French

Paris, France

commented 23 October 2016 at 19:04

Much as I like to use Slack, experience in large communities, even though smaller than Drupal, shows it not to be actually a viable choice.

https://medium.freecodecamp.com/so-yeah-we-tried-slack-and-we-deeply-reg...

Gitter anyone ? DrupalConsole uses it already, but at a much small scale.

Mattermost

attiks commented 12 December 2016 at 08:23

I've been using various alternatives, basically they are all similar, but since gitlab is still an option it might be worth considering to use Mattermost which is part of their Omnibus install, it does not look as slick as Slack/Flowdock but is 100% open source so it can be customized.

Attiks - Print and web design

Slack over IRC

Chris Charlton

English

Los Angeles

commented 20 December 2016 at 23:07

I used to use IRC, for more than Drupal and it's very dated. An LA Drupal member told me they're seeing the largest number of chat users in the Drupal Slack channels than IRC ever seemed to have, which is better for those onboarding Drupal.

Just sharing a respected member's observation.

_{Chris Charlton}
_{Manager LA Drupal}	_{Drupal Author Published since 2007}

How can we help?

ponies

Chico, CA

commented 25 October 2016 at 16:59

How can we help?

Enhanced issue and module metrics could help

Chris Charlton

English

Los Angeles

commented 14 November 2016 at 19:03

I think evolving a project's metrics will help Drupal.org and project maintainers. Download stats are only one dimension of a project; adding additional detailed metrics would help with a project's narrative, direction, and support needs. Let me explain.

If a module maintainer can see that after cutting a new release, the issue queue volume has increased for that 7.x/8.x/9.x update then that's a signal something is awry. A feature like this could encourage release candidates of project updates to make sure stuff doesn't blow up before cutting a new stable release. And on the other side, it could also expose which projects are lingering in a alpha/beta/rc state too long, which is a big problem D7 has (for life!). I can see a better metric/score for project consumers would help them pick between redundant projects.

By providing easy to consume metrics we can look at our add-on ecosystem in various ways. Imagine we can have on-the-fly conversations about the metrics of a project release/branch, unit test fail/pass ratio (of open tickets), issue queue volume (trends by date, and release, MoM/YoY), dev vs stable release usage, etc.

Also, don't be opposed to throwing in fun, trivial, non-value metrics for contributors to enjoy. Because life needs a smile now and then.

_{Chris Charlton}
_{Manager LA Drupal}	_{Drupal Author Published since 2007}

How about opendata?

DevElCuy commented 14 November 2016 at 20:05

I'm seeing other projects letting the community to build their own statistics, just by letting everyone access raw statistics data. That is the equivalent to open sourcing code, people creates new stuff and sooner or later some of that stuff gets into the core. d.o would get better metrics just by opening data and watching how the community innovates with it.

--
develCuy
Hiring the next generation front-end devs, contact Dilygent

Bare metal

attiks commented 12 December 2016 at 08:21

Gitlab is planning to move to bare metal, for various reasons but mainly I/O and latency, they wrote a very extensive blog post about their approach, https://about.gitlab.com/2016/12/11/proposed-server-purchase-for-gitlab-... and this also explains why their current gitlab.com platform is slow

Attiks - Print and web design