Within weeks of introducing the contribution credit system on Drupal.org we realized we had created something powerful. Like all open source projects, Drupal has a behind-the-scenes economy of contribution in which individuals, organizations, and end users work together to maintain the software as a public good. That behind-the-scenes economy was brought to the fore when we chose to rank the Drupal Marketplace by issue credits. For the first time, Drupal.org gave businesses a direct financial incentive to contribute code.  

Being good stewards of these incentives is a sobering responsibility, but also a great opportunity. We can use this system to recognize the selfless effort of our community volunteers, to reward the organizations that sponsor their employees' time to give back to the project, and to connect end-users with the organizations that are the biggest contributors.

But as we often say in this community—contribution is more than code. It is the time provided by dedicated volunteers; the talent of community organizers, documentation maintainers, and developers; and the treasure provided by organizations that sponsor Drupal events and fund the operations and infrastructure that maintain the project.

What are we changing?

We’re updating the ranking algorithm for Drupal.org’s Marketplace of service providers and list of all organizations in the Drupal ecosystem. We've expanded on the issue credit system to create a more generic contribution credit system which lets us recognize more types of contribution. Each type of contribution is now weighted to give the organization an overall amount of contribution credit. We've built this system so that we can continuously evolve the incentives it creates by adjusting the weight given to each type of contribution as the project's needs change. To prevent gaming, we will not be publishing the exact weights or total contribution score, but those weights have been reviewed by the Association Board and Community Working Group.

We've carefully chosen a few new types of contribution to factor into the ranking. These were selected because they create incentives to reach specific goals: encouraging organizations to sponsor development of Drupal, gathering more Drupal 8 success stories that can be used to promote Drupal adoption, and recognizing the financial contributions that promote the fiscal health of the Drupal association.

We now calculate the following 4 types of contribution into overall contribution credit:

What about other types of contribution?

Of course, these new factors still don't include all types of contribution. This iteration aims to add measurable factors that reward the behavior of organizations that are good Drupal citizens, and incentivize some of the most important contributions that have a big impact in moving the project forward. But there are other factors we'd like to include in the future! We're keeping track of these additional kinds of contribution, such as sponsoring local user groups, organizing training days, writing documentation, and more, in this issue: #2649100: Improve contribution statistics on user and organization profiles.

There are two factors in particular that we are not yet including that we'd like to address.

The first is project application reviews. These reviews are a critical part of the lifecycle of a new project on Drupal.org, but because we are making the Project Application Revamp a key priority for the first part of 2017, this was not our focus in this initial update. We may revisit this factor as the Project Application Revamp initiative gets underway.

The second is camp organization. We know that there are many individuals and organizations who invest heavily in Drupal Camps, and this has been a critical part of the project's success. However, at this time our data about the individuals and organizations who participate in camp organization is purely self-reported, and therefore too vulnerable to manipulation to include in the algorithm at this time. In the future we hope we can find a responsible way to measure and credit this kind of contribution.

We’ll continue to look for other good factors to add, and do our best to weigh them fairly.

How often will the algorithm change? Who governs these changes?

As this is our first major change to the marketplace ranking system since the launch of issue credits, we may need to make some small adjustments in the first weeks following the launch. However, we know that too frequent changes to the incentive structure will be frustrating for the individuals and organizations who are contributing to the project. Therefore, after the initial tuning we intend to update the marketplace ranking system on a roughly 6 month cycle.

While the primary responsibility to manage the contribution credit system is ours, we have committed to vetting these and future changes with members of the Drupal Association Board and Community Working Group.

Comments

kreynen’s picture

These seem like logical improvements to the organization ranking, but I would really like to see the DA put as much (if not more) focus on the communities of contributors who operate in sectors like higher education than the organizations that sell services to these sectors.  I understand that the groups who sell Drupal related services are important to the DA and the Drupal project, but when I read this post, Megan's post about the industry verticals and still have to navigate to the full list of organizations from the sidebar of Marketplace, it's hard not to feel like the DA values me more as a someone who can influence purchasing decisions at my organization for vendors of Drupal than it values my organization's contributors to the Drupal project.

hestenet’s picture

Thanks for your forthright feedback, Kreynen. I think you're absolutely right that the contributions of non-service providers (and of individuals for that matter) are equally important. 

It is part of our particular mandate for this year to focus on promoting Drupal adoption, and that is certainly part of that common thread you've pointed out running through some of our recent work.  

But we certainly don't want to leave end-user organizations, community groups, or individual contributors out in the cold. 

One of the challenges with finding ways to recognize those contributors is understanding what kind of recognition they want to see. For service providers it's very easy for us to find logical ways to use the marketplace as a tool for recognizing and incentivizing contribution. The path for recognition and incentivization for these other types of contributor is less clear. We would certainly not want to attach a rank to individuals. 

If you don't mind my asking - as someone who works in higher-ed yourself - what kind of recognition do you think higher-ed institutions would appreciate? Would a good first step simply be making that organization contributors view more easily accessible beyond the marketplace?

--
Tim L
Drupal Association - Director, Engineering

mikeker’s picture

Sorry, I know this post was almost a month ago but I just got around to reading it.

We would certainly not want to attach a rank to individuals

I respectfully disagree. As a freelancer, I would love to be ranked along with companies. I would like to tell clients that even through they are hiring a single person, they are getting someone who ranks (wherever I rank) among all these other companies and freelancers. Currently, the only way for my name to appear in the marketplace is for me to create a company and assign my work to that company, at which point I've lost all my credit up to now.

I opened https://www.drupal.org/node/2843472 as a suggestion of how to resolve this. (Ironically, I think that issue was opened the day this post was published... I swear I wasn't trolling!  :)

Thank you, all, for starting and continuing this discussion.

- Mike

hestenet’s picture

Thanks for your feedback, Mike. As an individual user who is a freelance service providers, I'd actually encourage you to go ahead and set up an organization profile on the marketplace as a freelancer. There are a few other freelancers who are doing the same. 

This makes it an opt-in process - you can choose to be treated the same as any other organization offering services, just an organization of 1. 

I do understand your concern about losing your previous credit, but you can actually go in an edit previous issue comments and add the organization attribution. (Though I admit that will be a bit labor intensive)
 

--
Tim L
Drupal Association - Director, Engineering

kreynen’s picture

I know these issues get lost as progress is made, but I made the case for what I wanted to see back in https://www.drupal.org/node/2580729.  Because we still can't sort organizations that use Drupal by sector, we wrote something to find the the higher ed organizations in the full list and started posting the Top 10 Contributing Higher Education Organizations.  You may have noticed several new higher ed organization nodes have been created since that was published.  I plan to blog about updates to that list each month and highlight some of the contributions.   As an organization, CU is actively trying to top Waterloo for the #1 spot.  

hestenet’s picture

Ah yes, thanks for reminding me of that issue! That would be a good improvement to make. It's also great to see what you were able to do just using the data on your own. Seeing what orgs and individuals do on their own with this data helps provide a model that we can look to. 

--
Tim L
Drupal Association - Director, Engineering

markconroy’s picture

This looks very interesting, especially for members of actively contributing Drupal organisations/companies.

Just an aside - if someone notices an organisation on the marketplace that may be incorrectly self-categorised (for example, they say they have offices in a country but they don't, and may be trying to game the system), is there a flagging/reporting mechanism?

darrenwh’s picture

I would like to second this, there are many companies showing up in the UK that don't have physical office here, would like to see this addressed as this would be classed as gaming. 

ressa’s picture

I just had a look at the Danish Marketplace section, and I see at least four companies (out of thirteen), who don't seem to have a Danish office.


Try Drupal 8 at simplytest.me
tvn’s picture

You can report such cases in the Webmasters issue queue.

colan’s picture

@hestenet: How about including project releases?  Some of us spend a fair amount of time cutting releases when nobody else does so we're not all stuck with development versions.

hestenet’s picture

This is a great idea. It's the kind of activity we could track as part of our individual contributor contribution credits (though we would not be able to expand it to organizations at this time, as the act cutting releases has no associated attribution to an organization).

I've added the suggestion to #2649100: Improve contribution statistics on user and organization profiles

--
Tim L
Drupal Association - Director, Engineering

xjm’s picture

My organization funds me to help manage core releases, for anywhere from an hour or two (bugfix releases) to 20+ hours (security and minor releases) in a given month. Currently this investment is credited less than fixing a typo in a code comment. It would be great to see it recognized.

Related question: Where are we at on crediting contributors to security advisories? I believe work was being by done on crediting those essential contributions but it's not mentioned here.

drumm’s picture

That was getting close, #2461167: Create Security announcement content type, but still needs some work.

davidhernandez’s picture

Credit for anything related to camps would be great; sponsoring, organizing, and speaking, Though it may be terribly difficult to get accurate information on volunteers, sponsorship and speaking should be easier to verify. Maybe you can start with drupalcon.

One question. Are there plans to use the ranking for any other lists? I believe this is currently used only for the service providers page? What about the other ones like the training page. I think that is still just alphabetical.

hestenet’s picture

That's a good idea, David. We do have more information about volunteers, speakers, and sponsors for DrupalCon. Camps are so important though, and they really deserve their own attention. One idea we've floated is using Drupal.org as an identity provider, so that camp organizers could let users log in via D.O and we could collect some organizer info that way - but that would be a very large undertaking. 

Good point about using the ranking algorithm for other listings - I've opened: #2843435: Consider using marketplace rank for other listings (e.g: training, hosting etc) to think that through.

--
Tim L
Drupal Association - Director, Engineering

davidhernandez’s picture

One idea we've floated is using Drupal.org as an identity provider

That certainly seems a bit difficult for a first pass for sure. Also, speaking as an organizer, most of our volunteer work and signups occur completely unrelated to our website. Speakers though should all be listed. Maybe there is a way for organizers to identify them (they aren't always associated with an account) and have it feed in to drupal.org.

hestenet’s picture

Also, speaking as an organizer, most of our volunteer work and signups occur completely unrelated to our website.

Yeah - that's the real part that makes this tricky. It seems like an at least partly manual process could be inevitable, and that could be very hard to moderate. 

--
Tim L
Drupal Association - Director, Engineering

Wim Leers’s picture

I think what could work is:

  1. create a "Drupal Events" project
  2. create one issue per event, owned by one of the organizers
  3. the organizer credits every volunteer for the event
  4. d.o doesn't count these as issue credits, but as "volunteering for an event"

This reuses existing infrastructure, and only requires the organizer to be trustworthy (which seems like a fair assumption).

Thoughts?

jhodgdon’s picture

I think this is an excellent idea. A few details:

  • We would need to make sure event organizers know about this project and procedure
  • If I am not mistaken, only Maintainers of a project can use the checkboxes and other mechanisms that grant issue credit. So for this idea to work, every event organizer would need to be added as a Maintainer of the Drupal Events project. That could be a bit cumbersome.

Given that, it might actually make sense to modify this idea slightly and create a new *type* of project called an Event (parallel to Module, Theme, Distribution, etc. that already exist), rather than one single big project for all events. Then there could be one instance of this type of project for each recurring event, owned by an organizer (for instance, there could be one called Pacific Northwest Drupal Summits). Then within the issue queue, each specific event (such as the 2017 PNWDS in Vancouver) could have an issue for organizing it, and the volunteers for that specific event would be credited there. The lead organizer for each year's event then could be added as Maintainer to the main project, which isn't so difficult to take care of because the project owner probably knows them personally (if it's even someone different).

davidhernandez’s picture

That's an idea that might be heading in the right direction. There may be issues with determining who controls this for specific events, and it might be tedious to credit large numbers of people (if iit's including speakers,) but it's probably a lot better than trying to create a connection with the camp website. This would also make it a lot more transparent as anyone involved with the event can see who is getting credit.

colan’s picture

Events already exist on groups.drupal.org. See https://groups.drupal.org/toronto for an example.

We just need to add the additional credit option (perhaps not right away), and start using the data.

C_Logemann’s picture

--
There is a module for that!
My company: Nodegard GmbH

jhodgdon’s picture

What about documentation and support? I don't see anything in the current or planned credit system that addresses people who maintain documentation guides on drupal.org, edit or create documentation pages, or voluntarily provide support and guidance to other members of the community (via forums or StackExchange or IRC, which of course would be hard to track).

hestenet’s picture

Absolutely - both documentation and support activities are very important. We've captured several of these items as things that we'd like to track and provide more credit for in #2649100: Improve contribution statistics on user and organization profiles 

There are two problems we need to solve for here: 
 - One is the general question of: What does it mean to recognize individual contributions? Right now, the primary way we do that is through listing those contributions on the user profile. (Such as we do for documentation edits, for example) But what other forms could that recognition take?  As I said above, I don't think a 'ranking' of individual users would be within the spirit of the community. 
 - The second is: There are also organizations that contribute to these kinds of tasks as well - but we don't have a way to attribute these kinds of site activities to organizations - can we come up with an elegant way to do that? 

These are harder problems, but still very much important to us. 

--
Tim L
Drupal Association - Director, Engineering

jhodgdon’s picture

I'm glad these are on your radar -- figured they were. :)

Regarding the organizations and rankings...

The Drupal Core project has at various times published lists of who contributed to fixing the most issues. I agree with you that these types of rankings are not necessarily within the spirit of the community, but I just point this out to say that we are already doing that, to some extent, in some contexts. It's definitely a way of thanking those people who contributed a lot of time to the project, to publish a list of the top contributors in a certain time frame or on a certain version.

But the context here is about giving organizations credit for time they devote to helping the Drupal project, and displaying them on the Marketplace in order by the amount of contributing they do. For that to work for issue fixing, the mechanism is basically that when a person adds a comment (or patch) to an issue, they can credit the organization that funded that work, and then when a maintainer marks the issue "fixed", they can check off which of the individuals made a substantive contribution. From that, we can figure out (a) on the user profile, a count of how many issues each person contributed to fixing and (b) a count of how many issues each organization contributed to fixing. I don't see a technological reason why we couldn't have similar mechanisms for (1) making a revision of a Documentation page (we are already counting those revisions, as you said, on the user profile -- all we'd need is a way to credit organizations when someone makes a revision) or (2) acting as a Maintainer for a page or section.

Speaking of which, maybe a person who acts as Maintainer for a Project (module/theme) should also get some credit -- was that already mentioned above? Can't remember.

Support is much more difficult... I don't even know what the official line is on how individuals/organizations in the community can provide free voluntary support to other members of the community. It used to be forums and IRC, but I don't know what the current line is... StackExchange? I think there is a way there for individuals to get recognized for providing answers, and especially answers that people "like". Not sure... IRC would obviously be more difficult (read: impossible?) to track.

kreynen’s picture

The idea of weighting rank based on involvement in hosting meetups and camps has been discussed within higher ed organizations involved in EDUDU, but the devil is in the details.

There are several camps hosted universities and colleges.  The problem is that not all of these are equal effort/value.  Some schools charge the full rate they change any other event.  Other offer the Drupal community a discount because the school benefits.  Some university IT groups pay the hosting fee out of their budget.  Other schools host for free.

How to do you track this level of detail?  How what value do you give it?

As @jhodgdon has pointed out, commits are not all equal either.  One big difference is commits happen so often giving "unearned" credit isn't that big of a deal.  I will often give a new contributor credit for a patch even if I had to rewrite 90% of it to encourage future contributions.  

Camps only happen once a year, but many require a lot more effort than most commits.  We are only only giving credit for commits in the last 90 days (I think?). How long do you factor in credit for hosting or organizing a camp?

As @jhodgdon has also pointed out, patch -> commit process includes a method to provide feedback about who deserves credit.  When documentation is managed with commits, it also has this.  

PA reviews currently do not.

The number of truly unhelpful reviews I've read from new Drupal.org users whose only motivation to do a review was the bonus required to avoid PAR's 1+ year backlog is maddening.  I can't imagine how the Drupal project could possibly do more harm to itself than giving even more of a value the garbage reviews many developers experience as their first interactions with the Drupal community.

I understand that the Project Application Revamp is on the DA's roadmap and that credit could be rolled into that project, but to really do it right you are going to need a Stack Overflow-esque point system.  I would really hate to see fixing PAR get delayed yet again because this makes it even more complicated.

All of this said, I was skeptical of the commit credit tracking and organization credit when it was announced and you've done a phenomenal job with that.  I'll try to remain optimistic that you'll find out elegant ways to factor in these other contributions as well.

vilepickle’s picture

I've been burned by project application reviews 3 times now. I'm following the revamp but the level of attention that is getting doesn't seem to be significant enough according to the activity I've witnessed. Everybody that's "in" on the Drupal contrib club already don't care about it IMO. I'm a supporter of Drupal and code for it professionally and I'm still not able to contribute to the actual project in a meaningful way with full projects. This is something that is really hurting contributions to Drupal for many people.

ressa’s picture

We are only only giving credit for commits in the last 90 days (I think?).

It seems like this has been increased, first to half a year, and recently to a full year:

Credited on 4 issues fixed in the past 1 year


Try Drupal 8 at simplytest.me
drumm’s picture

Organization pages are limited to 3 months, user pages were extended to 1 year. For people, we really should have all time, once #2042697: Add historical issue credits to Drupal.org user profile is done. 3 months is a bit short for people, taking a few months off of contributing happens. Keeping organizations with a rolling window helps keep the list current and can help keep momentum.

ressa’s picture

Thanks for clarifying @drumm. +1 on extending personal credit to forever.


Try Drupal 8 at simplytest.me
GiorgosK’s picture

There is nothing done recognizing small individual contributors that write bug reports or help out in the forums or write or review patches and hence move an issue forward with constructive comments ... or in general if there was a way to distinguish the help givers as opposed to the help getters it would be nice ... correct me if there is something like this already in place ... but I was thinkin for rating system for each question and answer or general post (perhaps something like stackoverflow)

xjm’s picture

In core issues, we credit people who file helpful bug reports, who provide reviews, and who post constructive comments equally with people who author patches. We've had this as a practice for a year or two (it wasn't always). Hope that is encouraging!

xjm’s picture

Contrib is of course encouraged to follow the same model. :) It's up to each project maintainer whom to credit, but often practices we use for core also get used in contrib as the community becomes familiar with them.

nedjo’s picture

I've made these suggestions offline previously, but I'll repeat here.

Re these goals:

encouraging organizations to sponsor development of Drupal, gathering more Drupal 8 success stories that can be used to promote Drupal adoption, and recognizing the financial contributions that promote the fiscal health of the Drupal association.

Seemingly missing is the (obvious) question: what's the best way to meet the needs of organizations looking for service providers? Which is, presumably, the main reason we have marketplace listings in the first place.

Specifically: is the average Drupal user best served by the companies that have contributed the most? That idea sounds self-evidently wrong. Companies that have contributed the most will tend to be large and costly. So, from the point of view of Drupal users rather than service providers, the current system might serve large corporations well, since they are indeed looking to connect with top level service providers. But for everyone else it would seem to be more of a barrier than a help.

So an improvement might be: use the statistics gathered to class actively contributing companies into groupings, from low to high contributions, in, say, five classes. Don't use these to order results. Instead, provide a search facet, defaulted to, say, the second lowest class, then order results alphabetically. By default, users browsing the marketplace would see alphebetized listings of the companies that meet a threshold of current contribution.

This would provide an incentive to actively contribute. It would also allows users to filter for the highest contribution levels, if they wish. It might avoid many of the problems with the current system.

Jaypan’s picture

And once again forum contributions are completely ignored.


Checkout my Japan podcasts.
kreynen’s picture

Where does it end? 

Jaypan’s picture

The forums are on Drupal.org. Many people, myself included, learned how to program with Drupal based primarily on forum posts. Googling issues regularly ends up with forum results.

I don't have a direct answer as to where it ends, but wherever it ends, it shouldn't be before counting forum contributions.


Checkout my Japan podcasts.
GiorgosK’s picture

My question above was also about forum contributions so I think its a valid concern from JAYPAN...

hestenet’s picture

Recognizing support contributions in some way is certainly important. As this thread shows, there are still quite a few more kinds of contributions. 

Calling back to the conversation above with @jhodgdon - I'd like to better understand what recognition of individual contributors should look like (models other than a ranking). 

--
Tim L
Drupal Association - Director, Engineering

jhodgdon’s picture

For years, whenever anyone mentions the Forums on drupal.org, many people have rolled their eyes, because they're very limited and hard to use compared to something like Stack Exchange. For that reason, I think many of the support questions/answers have moved off the drupal.org forums and onto Drupal Answers on Stack Exchange.

IRC is also an important support channel for the project.

So, my recommendation would be that before we try to figure out how to *recognize* support contributions, we first need to figure out what the "official" support channels for the Drupal project should be. That conversation has come up numerous times in the 9+ years I have been involved in the project (along the lines of "can we possibly replace the Forums with something better?), but has never gone anywhere.

And then once we figure out what the official channels for support are, we can figure out how to integrate a measure of contributions on those channels (if possible -- probably isn't with IRC for instance) into the Marketplace as well as user profiles. But to me, asking that Forum contributions be recognized is premature if the forums are basically defunct... and if they aren't defunct, we should be thinking about how to add features to them to make them more like Stack Exchange (like voting for answers, etc.).

Jaypan’s picture

That conversation has come up numerous times in the 9+ years I have been involved in the project (along the lines of "can we possibly replace the Forums with something better?), but has never gone anywhere.

Not through lack of attempt: https://www.drupal.org/node/2536122

I put in a significant effort/push to get the forums upgraded last year, but the Drupal Organization was not willing to play ball. The conversation ended up being shut down altogether by the organization. Every impression has been given by the organization that they just don't care about the forums, and the lack of any credit given for forum support shows that they don't consider as a support channel with value.


Checkout my Japan podcasts.
David_Rothstein’s picture

The official support channels are listed at https://www.drupal.org/support, and Forums are certainly on it. It would be bizarre if they weren't, since they're hosted on drupal.org :)

Contributions to the forums would also be a lot easier to recognize than contributions not on drupal.org (obviously), since they're already in the drupal.org database.

kreynen’s picture

eh... I'm not sure I can buy into it being easier to integrate Drupal's neglected forums than leveraging the point logic used to rate both questions and answers that has been developed over the years by StackExchange and easily available via https://api.stackexchange.com/.  Would it be better if we had those features on Drupal.org?  Sure, but we don't.  

There are also an awful lot of conversations about official initiatives only happening on Slack for it not to be considered an official channel.

davidhernandez’s picture

I'm not sure why Slack isn't listed there, but it is listed in the Getting Involved guide. https://www.drupal.org/node/2798167

I'm assuming it is because the /support page can only be edited by certain people. We should be free to consider the main Slack channel at least semi official since DA staff have ownership access to that Slack team.

kreynen’s picture

A well written blog post about a module can take more time than actually writing the code.  The Planet feed gets some of the data about the number of posts in the feed.  The feed could be linked to an org, but without some custom Drupal meta tagging similar to the commit message structure, I'm not sure how you would credit an individual with having written the blog.

Samuel Lavoie’s picture

Rating seems to work fine, take a look at the system used by MOZ Q&A with an interesting voting system, and comment voting, etc. or the very well know Quora Q&A site with multiple rewards system for individual contributors

Jaypan’s picture

A rating system would work great, the problem is that there is no one to build it. The Drupal organization has no interest in doing so, and they don't create an environment conducive to volunteers doing so for them.


Checkout my Japan podcasts.
davidhernandez’s picture

Drupal 8 case studies

Is this part accurate? When looking through the marketplace it looks like the case study counter for every org is including their total, not just D8 case studies. Is there a difference between what is calculated and what is displayed or is the calculation not limiting to 8?

drumm’s picture

They are two different numbers. The display on organization pages and listings, for all stats, are straightforward counts. The ranking calculation has additional factors, like only counting D8 case studies.

Pages