There's been a lot of discussion about rating projects, so I took a quick stab at adding support to Project.module. It's built on top of VotingAPI, which may be a philosophical issue for some, but it sped things up for me. If VotingAPI isn't installed, the voting options just won't show up.

Voting can be turned on and off on a project-by-project basis in the same fashion as the issue tracker. If multiple categories have been entered into the Issue Tracker 'categories' field, they'll also show up as separate criteria to rate the project on.

Initial results of the patched moduel slapped onto one of my sites can be seen at http://www.talkonomy.com/node/2. The theming of the voting widget and the voting results could use some work, but it's functional and gives us some direction. Any thoughts, feedback, improvements, etc are welcome.

CommentFileSizeAuthor
#58 reviews.png86.5 KBeaton
#1 voting.zip1.87 KBeaton
project_voting.patch8.77 KBeaton
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

eaton’s picture

FileSize
1.87 KB

The theming function for displaying vote results also points to the images in the 'voting' directory. Attached is a zipped version of the dir/images.

nedjo’s picture

Jeff, thanks for doing this. Nothing like a concrete example after lots of talk.

Want to take a crack at adding a browsing option, so that we can browse by rating (as well as category, date, and name)? This is done through the 'project_sort_methods' hook. The most straightforward way is to modify project_project_sort_methods() to add your method. (The alternative is to implement a function in your own module.)

Begin by declaring your new method:


    case 'methods':
 -     return array('category' => 'project', 'date' => 'project', 'name' => 'project');
+     return array('category' => 'project', 'date' => 'project', 'name' => 'project', 'rating' => 'project');

For case 'sql_settings' :, put the pieces (joins, fields to select, sort order, etc.) you'll need to add to the sql select statement. This works a bit like views.module (which lamentably I hadn't yet looked at when I wrote this into project.module).

You'll also want to put


    // Does this method page results?
    case 'pager':
      switch ($method) {
        case 'category':
        case 'date':
+     case 'rating':
          return TRUE;
        case 'name':
          return FALSE;
      }

And that should do it. Email me if you're digging into this and have questions.

eaton’s picture

Yep, that's next on my agenda. It doesn't look like it will take too much doing. Projects will only be browsable by the 'overall' rating, as each project might have different sub-criteria.

moshe weitzman’s picture

subscribe - i am toying with this. just for fun, for now. subscribe.

dww’s picture

Version: x.y.z » 5.x-1.x-dev

Bump: http://drupal.org/node/77976 is now marked duplicate with this -- it contains some discussion that might be useful. There's a lot of interest in this these days, so it'd be nice to come up with something...

jeanhr’s picture

Subscribing and highly interested in seeing this come to reality soon.

See also related: http://drupal.org/node/166471

NancyDru’s picture

As much as I would love to see something to improve the quality of some modules, I also see comments about rating that very much trouble me. "I want to download the 10 most highly used modules." ... "I want to use the 10 most highly rated modules." ... etc. Those statements are totally ludicrous! And yet I see them frequently as reasons to have a rating system or download count. These are people who want to choose a solution and then go looking for a problem - the tail wagging the dog.

People need to choose a module that solves a problem they are having or a function they need.

Yes, there are many modules with exceeding poor documentation (I hope none of mine are considered as such). And there are poorly written modules out there (obviously not mine). So how are we to encourage the developers to improve their product?

First, we need to make sure users are aware that they can - and should - post feature and support requests to improve the documentation and functionality. When some of us who are familiar with Drupal coding choose someone else's module, we should be welcome to offer positive suggestions on how to improve them.

Second, perhaps all projects ought to come with a pre-built queue that is for user comments on the module.

These two ideas allow a developer to fix some of the problems and actually look the better for doing it. A rating system will leave (despite the ability to change a vote) bad ratings around even long after the problems that caused them have been addressed and solved. Many people who give poor ratings will never come back, hence they will never change the rating.

eaton’s picture

Nancy, I agree. Any rating system for drupal modules would need to take into account the nature of the system. Possibly, an 'overall' rating and a per-branch rating. All ratings would be cast on a per-release basis, while the 'overall' would be averaged over releases. That would give modules a chance to 'make up' for a bad release.

That's just one solution, though. I think gathering useful statistics about issue queue traffic, developer activity, and so on are all just as critical.

e.r.n.i.e’s picture

Title: Project.module: Add user ratings for projects » Leading new users to discover the power of drupal

I've posted this already on the thread about drupal 6 beta 2 and was pointed here because it was off topic there:

Whether a module is highly rated or frequently downloaded is no reason why you need it.

I won't pick a module just because it's one of the TOP TEN - but sure I would take the time to look at it and maybe play around with it (on a local test site, of course).

That's one way of learning drupal! And not "the tail wagging the dog". ;-)

There are hundreds of modules for drupal 5 and I think knowing which of them are the "most wanted" will be a good way to discover the power of drupal. As a new user I'm just learning what can be done with drupal, but I find it rather hard (and time-consuming) to dig through the overwhelming crowd of modules to discover wich possibilities lie hidden in there.

And then there are modules which (seem) to have similiar features. Ratings and download rates might be misleading sometimes, but they can't be worse than the current alphabetical order. At least the rates give you a hint which module to try first.

To guide new users through the "modules jungle" there should be:

  • Number of downloads/per week or month
  • Ratings with a site-wide fixed set of criteria. I would suggest:
    1. The usefullness of the features/functionality added by this module
    2. Stability (Number of bugs, how fast are they fixed)
    3. Quality of the code
    4. Quality of the documentation!
    5. Ease of setup and administration
  • A calculated score for the activity of this modules development like on sourceforge

All ratings should be bound to one version, otherwise they won't make much sense.

e.r.n.i.e’s picture

Title: Leading new users to discover the power of drupal » Project.module: Add user ratings for projects

Ooops - I changed the issue title inadvertently. :-O

NancyDru’s picture

One technique that I haven't seen mentioned elsewhere is that the advanced search can be limited to check project (module) pages.

dww’s picture

NancyDru’s picture

Until that is implemented, one may still limit one's search the way I suggested.

jeanhr’s picture

Ernie,

That is a good proposal as far as I am concerned (a good first step that could be done) and there are interesting similarities to my proposal which goes further. Please check it out here: http://drupal.org/node/166471

Nancyw,

I like the idea of having a queue for user comments on the module. Let's say recent comments.

You are right in not encouraging a rating system just like a voting system. I don't think the users would like just that.
This needs more consistency in addressing the real issues like letting the users qualify things in some more details.

An idea, would be to rate the different points on a shorter span of time, like the last year (or 3 months) only, instead of since the very beginning. This would reflect more fully on the "actual" work being done by the "active" developers.

In any case, from a user's standpoint, something should be done about a ratings system fairly soon.
Others do it and I think it is really needed for Drupal.

And it would further encourage "the" developers that are doing an already great job.

Like Ernie also said, to guide new users through the "modules jungle". It is a real jungle and it would be a whole lot better to guide the new users to those quality modules instead of the contrary.

You don't know until you tried... but maybe hundreds have tried before you. Let's get their opinions...

In the end, this would certainly increase (and somewhat insure) the overall satisfaction in using Drupal.

My humble opinion.

NancyDru’s picture

Indeed, I'd love to see something that rates the quality of certain aspects of a module. I'd love to think I'd rate fairly well. But I am still quite concerned about "the tail wagging the dog."

"I have not failed, I have simply found 10,000 ways which do not work!"
— Thomas A. Edison

jeanhr’s picture

Nancy,

Of course, anyone, implementing some quality measurements in their systems for the first time, is always quite concerned.
For some reasons, I have seen this reaction over and over in the industry... It's quite normal.
I've implemented such things... and it has to be done smoothly and the goals have to be well explained and documented for everyone.

But after the initial fears, everything settles in 99% of cases.
And remember that adjustments can and will be made on a regular basis.

If you, Dries, and a group of good people including developers and users oversee the implementation, why would "the tail wag the dog"...?

Going ahead intelligently is much better than not taking the appropriate steps to improve quality on the basis of some fears.

NancyDru’s picture

There's one indicator of "quality" that I always look for, but I know from various forum posts that others don't, or don't understand. That is the issue queues.

Perhaps the system could create and display something like this:

This display shows how long issues have remained for this contribution.

Status                    Average           Longest
----------               -------------    --------------- 
active                   1 week, 2days    8 weeks, 1 day
active, needs more info  3 days           8 weeks, 2 days
patch needs review       3 weeks, 4 days  16 weeks, 3 days
fixed                    4 weeks, 5 days  32 weeks, 1 day
won't fix                1 week, 4 days   2 weeks, 5 days
closed                   6 weeks, 6 days  34 weeks, 3 days


Note that "closed" status happens automatically two weeks after an issue has been marked with one of the resolution statuses ("fixed," "by design," etc.) and all activity ceases. This can "inflate" the average times and should be taken into account.

If these figures could also show "three months ago" that would be even better.

A few other things I'd like to see are: 1) in indicator (check box) for whether the module has been checked with the Coder module; 2) an indicator showing whether a translation template is available.

dww’s picture

@nancyw: see http://drupal.org/node/79550 for automated quality metrics and statistics abou thte issue queue. also, please check out the existing "statistics" pages for the issue queues for each project, for example, start here: http://drupal.org/project/issues/project then click on "statistics": http://drupal.org/project/issues/statistics/project. pretty similar to what you're talking about.

Seems I need to make another wiki on g.d.o about "all the issues about project quality stuff" so people don't splinter the conversation. this issue is about *user ratings*. if that's not what you're interested in discussing, please look elsewhere. thanks.

jeanhr’s picture

NancyDru’s picture

Thank you, Derek. I was unaware of the other discussion. While I knew about some (not all) of the statistics, I doubt seriously that 1% of Drupallers do, and even fewer ever check them.

If you wish, you may have my post deleted from this thread.

catch’s picture

Subscribing to this. Just quickly I think a multiple axis vote would just get abused (10,10,10,10,10 etc.), unless it was hard to do that.

Also I like the idea of votes being aggregated per release and across releases - some kind of degrading of votes across time is very, very important.

Caleb G2’s picture

Thank you dww for pointing me to this thread. This is a great initiative! Also, thanks to Eaton for getting it started and rolling a patch for it.

That said....gah!!!!!!!!

This poor issue has been languishing for too long, and has seen its fair share of sidetracking. Please let's get it back on track asap. :-)

Let's invigorate this issue and reach the successful, and implemented, conclusion that it so richly deserves:

  • (In the spirit of 'agile' development) - Let's eschew to many abstract/meta discussions about what would make the 'perfect' rating system:
         A) almost *anything* would be better than what we have now
         B) there will never be a 'perfect' rating system
         C) it's not like things can't change once *something* is committed. This has been stalled long enough.
  • I can only assume that the original patch Eaton supplied does not apply anymore as the project.module has recently undergone radical changes. Can someone (more able than myself) help roll a new one with similar functionality? I'd be glad to help test, but do not feel qualified to do such an important thing that will be released on D.O. myself.
  • For anyone feeling the dying urge to dissect or discuss the myriad possibilities for a rating system - please consider queuing your requests/comments for after an initial commit of the rating system. Eaton is semi-singlehandedly (fully?) responsible for the votingapi which powers most Drupal voting modules, so I'm sure everyone can quite agree that is credentials enough for rolling a patch to enable some rating functionality in regards to the project.module.
Amazon’s picture

See http://drupal.org/node/192410

Voting is known to have two problems. Cummulative low quality voting leads to low quality results. "I went to Drupal and the most popular modules were crap". Second, voting can be gamed. We've see voting and even reviews gamed on Joomla and Amazon's system.

But community feedback has it's place and I think we can evolve solutions for public review.

Please read: http://pivots.cms.si.umich.edu/sites/pivots.cms.si.umich.edu/files/pivot... and http://pivots.cms.si.umich.edu/sites/pivots.cms.si.umich.edu/files/Poste...

We will get the pivots up on scratch and see what people think.

jeanhr’s picture

@caleb: Thanks for your post. Your comments are encouraging and are right to the point.

@amazon: I read your pivots doc and it is another very interesting road.

@all... Please note... I am most probably just an average user... but I love Drupal like many.
I am not a qualified team member of Drupal at all, but still as a serious day-to-day user/admin/integrator implementing the system for about every new customer, I just feel the urge to voice my concerns which might not have been fully addressed. Just hoping, in the end, some of my inputs can be useful to make Drupal even better. And I would love to participate and/or test things out if you ever needed. Do not hesitate to get an opinion from just an average user like me. Cheers.

NancyDru’s picture

@jeanhr - anyone who is a a serious day-to-day user/admin/integrator implementing the system is a more than a qualified team member of Drupal.

All it takes is a willingness to help.

jeanhr’s picture

Willingness there is a lot... so do not hesitate to "use" me ;-)
Yours truly.

Caleb G2’s picture

Hm. So, I guess effectively this thread is dead in favor of the issue that Amazon posted and is actively working on (see his comment above for more)?

All further discussions there?

dww’s picture

Status: Needs work » Postponed

No, this thread isn't necessarily dead, but Amazon's comment #23 shows some of the potential pitfalls this thread faces. If you're interested in the pivot stuff, go to that other thread, but don't hijack it to say "I just want to add user ratings and reviews to projects!". ;)

Probably what we need is a whole friggin' group on g.d.o about this topic, and a lot of brainstorming, mockups, proposals, etc. (@Amazon: would the existing d.o redesign group be appropriate, or should we make a separate group just for this?) Currently, the discussion is splintered in too many places. There are a lot of opposing viewpoints on this topic, many different possible implementations (some of which are complementary to each other). There's a ton of interest, but very little concrete progress, since few of the proposals are clearly a step forward and have someone actively working on implementing them.

So, we need:
A) To collect all the possible proposals in 1 place (e.g. a g.d.o wiki page with links to the appropriate issue or g.d.o post for every proposal related to project quality indicators of all kinds). I can help populate this page once it exists, since I have a bunch of these proposals bookmarked or otherwise know how to find them.

B) Meta-discussion on the proposals: which ones are clearly a good idea now, which ones need more thought, which ones aren't going to work at all, etc.

C) Volunteers to work on the "low hanging fruit" -- the proposals that we can right now agree would be worth doing, but which won't take a huge amount of code to accomplish.

D) Further discussion on the specific proposals that need to be fine-tuned or otherwise better thought-out. I consider this particular proposal for end-user ratings and reviews (issue #50605) in this category. There are a lot of potential problems this will create, so we need to have good answers for those if we want to continue exploring this.

Therefore, I'm marking _this_ thread postponed, in favor of creating the g.d.o group about the broader topic of project quality indicators (or finding the appropriate existing group to use for the purpose), and the 4 steps I outlined above.

Cheers,
-Derek

Caleb G2’s picture

Heh, one person's hijacking is another person's way of trying not to get 'stuck in committee'. ;)

I can see that the route to solving the issue is obviously a bit more circuitous than I had first imagined. Fair enough. I defer to the project maintainer. :-)

dww’s picture

@calebg: Don't just defer... ;) Feel free to start the steps I outlined above. Your help would be great, and you're clearly inspired/motivated to do something -- I just don't have time to do all of this myself. Thanks.

Caleb G2’s picture

Dww, ok with your green light I guess I will try to scoot things along as I can. It would be good to hear from Amazon about this before a new group is made though:

Amazon: would the existing d.o redesign group be appropriate, or should we make a separate group just for this?

Amazon’s picture

Starting a thread on module quality measures and recommendations as part of the Drupal.org redesign is appropriate. It's the most requested feature according to the "State of Drupal 2007" survey.

moshe weitzman’s picture

On the "gaming" problem, I wrote some code that I need to contribute for votingapi. It is an integration between voting and buddylist. So, in the extreme only the scores of your buddies count toward the averages that you see. The formula can be tweaked so that buddy votes are x% of the score and overall is the other 100-x%. This basically gets rid of the gaming problem, since unknown voters won't affect the scores i see. Further, it answers the "what module should i use" question in precisely the way i want it answered. Namely, I want to use the modules that Earl, Eaton, Dries, Angie, dww, etc. are using. Those are my trusted sources. I'm sure others have theirs.

One small issue with this is that new drupal people don't have a trusted network. I think thats easily remedied by showing on the site which users have been befriended often. I think this is a smaller, solvable problem.

I was thinking of running this on my own domain for a while until drupal.org got its act together. But I haven't dedicated enough time.

Anyway, thoughts on this approach are welcome.

Caleb G2’s picture

Okey dokey. Thread created within D.O. Redesign Group. :-)

dww’s picture

Ok, thread updated with initial pile of links to all the stuff about this topic I've been trying to keep track of.

Caleb G2’s picture

Thanks, that's totally awesome. :-)

catch’s picture

Title: Project.module: Add user ratings for projects » Add user ratings for projects
Project: Project » Drupal.org infrastructure
Version: 5.x-1.x-dev »
Component: Projects » Drupal.org module

http://drupal.org/node/271933 was duplicate. Moving this to infrastructure - IMO it's not a project.module issue, or not yet anyway.

Amazon’s picture

Derek, could you point to your work on project voting for this?

catch’s picture

That work is for project issue voting, not projects themselves: http://drupal.org/project/project_issue_voting

danithaca’s picture

Once rating is added to projects, it might be able to feed us quality data to generate better module recommendations via the pivots system. The outcome would probably look like recommendations on amazon.com.

Gábor Hojtsy’s picture

Issue tags: +drupal.org redesign
Gábor Hojtsy’s picture

Status: Postponed » Active

Being an active issue for the drupal.org redesign, not marking this postponed.

Amazon’s picture

I've asked Daniel to write a blog post summarizing research on ratings systems and how the are gamed. If this is moving forward, let's make sure it's done with our eyes wide open the manipulation that is inevitable.

Kieran

Gábor Hojtsy’s picture

Kieran: we discussed the following metrics might be easy to game: downloads, usage, rating, reviews, alphabetic (although that is a bit harder, since we look at project names and vet 00aa00__ prefixes), activity (independent of whether we measure it by cvs commits or issues). These are basically *the* metrics we are going to have, and looks like each of them can be gamed in different ways. Looking forward to all kinds of ideas to make them non-gameable in their respective issues (look at at http://drupal.org/project/issues-term/799) :)

dww’s picture

There's a long discussion about all this at Project Quality Metrics on Drupal.org (meta document) -- please let's not duplicate that discussion here, but rather contribute to that if there's more to say.

danithaca’s picture

My advisor is one of the leading researchers in preventing gaming in recommender systems. His recent academic publications in this area include Manipulation-Resistant Reputation Systems, The Influence Limiter: Provably Manipulation-Resistant Recommender Systems, etc. The basic idea is to solicit honest ratings because it's the best strategy for individuals under a cleverly designed mechanism. I'll write a blog about it after I digest the algorithm :)

The manipulation-resistance algorithm and other cool features will be implemented in the Recommender API module as part of my research. The Recommender API module is at the heart of the next version of the pivots module. It's designed to support generic purpose collaborative filtering tasks. It is different from the content-based algorithm used in Solr. Hope the new Recommender API module (developing) will be a contribution to the community.

Gábor Hojtsy’s picture

@danithaca: the recommendations and the ratings for projects show up explicitly in the redesign, so we need to have some kind of metric for the user ratings as well, independently or digged from under the hood of the recommendations.

danithaca’s picture

@goba: Thanks for the suggestion. Actually recommender systems and reputation systems are quite related. I'm marking it down in my work note about mining reputation from users activity in the Recommender API module as well.

finkatronic’s picture

Project: Drupal.org infrastructure »
Component: Drupal.org module » Code
Issue tags: -drupal.org redesign

Moving to the Redesign project. Check to see who has been assigned to implement this section and the staging server it'll live on during implementation here: http://groups.drupal.org/node/37064

lisarex’s picture

Component: Code » Modules
Issue tags: +drupal.org redesign

Assigning to new component.

drumm’s picture

Status: Active » Closed (duplicate)

Looks like the party moved to http://groups.drupal.org/node/80004.

dww’s picture

Project: » Project
Version: » 6.x-1.x-dev
Component: Modules » Projects
Assigned: eaton » Unassigned
Status: Closed (duplicate) » Active

g.d.o is for meta discussion to come up with plans. Once we're happy with a plan, we'll still need to implement it, which is what this issue is for. ;)

lisarex’s picture

Issue tags: +Usability, +UMN 2011

This came up in the UMN usability testing 2011.

Observation:
Module pages don't indicate whether a given module is of high quality.
Example: differentiation between 3 “announcement.” modules

Recommendation:
Develop metrics. Suggestions:
- Ratings
- Comments
- Downloaded # times

See also "Create project quality metrics system" proposal http://drupal.org/node/128546

eaton’s picture

If we go with ratings, I'd strongly suggest as simple a system as possible -- no separate ratings for 'code quality' and 'functionality' as others have suggested in the past. Just click to rate a particular release of a module, and display a breakdown of how many people have rated the project at different levels. The iTunes rating model, basically. On their own the ratings won't help much, but when combined with comments, download statistics, and open issue ratios, it would be useful.

geerlingguy’s picture

Subscribe, and I'm with eaton on the idea of an iTunes-like rating style. As long as ratings are only used as one of the metrics to judge a module, and not as 'the' metric, I think simple ratings will be beneficial. We can ensure that's how they're used by styling / emphasizing them less than other review-oriented sites.

A few data points:

  • Joomla Extensions has the rating as the top metric for a listing, which gives that a lot of importance (too much, imo).
  • Wordpress Plugins doesn't show the rating except on the plugin's full page... though the color in the rating widget makes it stand out pretty glaringly.
  • Drupal Modules has too much information in each rating display - it took me a while to even realize what all those colored bars meant. Though the information can be useful, I think it's a bit too much for a listing.

At this point, 90% of the modules I start using are direct recommendations from another Drupal developer. The other 10% I find through my subscription to the new modules feed at drupalmodules.com. There should be better ways :)

mparker17’s picture

Subscribe!

webchick’s picture

Subscribe.

eaton’s picture

FileSize
86.5 KB

IMO, the simplest solution is the best: allow one-rating module reviews using the Amazon/iTunes model.

  1. Allow users to post reviews of modules.
  2. Include a 1-5 rating field on the review form.
  3. Include a check box that says: "I've used this module on a live site." Use this to weight whether the review shows up at the top of the list, but don't try to weight the actual review score by it until down the line.
  4. Display results as a single five-star value, but expand or hover to display a breakdown of how MANY votes have been cast at each level.
  5. Optionally provide 'was this helpful' ratings OF each comment/review; like the "I use this on a live site" checkbox, this can be used to highlight notable reviews, and doesn't have to be used to weight the actual score.
  6. Integrate other metrics that can be automated later, treating them as a separate score.
  7. DO NOT allow people to rate a module without leaving a review. That's contrary to iTunes/Amazon but is probably worth it, since we can't easily restrict it to ONLY people who've purchased a module.

For reference, here's how both Amazon and iTunes display their ratings:

geerlingguy’s picture

I like the idea of the hover-to-show-breakdown method, though I often find the simple five stars alone helpful enough. If that ever causes any holdup or accessibility problems on this issue, I say we axe that functionality.

webchick’s picture

dww’s picture

Issue tags: +projects quality

I'm still -100 to a generic/"simple" 5-star rating that doesn't specify what you're rating. WTF does "3 stars" mean? ;) How much you like Earl? How nice Earl is when he tells you your issue is a feature, not a bug? How well it works? Did it solve the problem you thought it was solving? Did it solve the problem you wish it solved, in spite of what it said on the project page? Was it well documented? Was it easy to figure out and use? Did it break your site? ... It's utterly meaningless if it's just floating there without any context. Please see Project ratings and reviews for drupal.org redesign for more on this.

I proposed that each review (and yes, you should have to submit a review to provide any rating votes at all) has some optional questions for things you might be talking about when you're rating something. I came up with 4 questions that seem to be general enough to be relatively easy for reviewers and end-users to grok, but specific enough to be useful:

  • works as intended
  • well maintained
  • ease of use
  • documentation

I'm not totally set on exactly those questions/metrics. And you don't have to fill them all in -- if you only want to rate it on 1 axis that you're familiar with, great. But if you're going to give a module "4 out of 5 stars" on something, you should have to say on what axis you're rating it on. Otherwise, we're going to go through a HUGE amount of work and invite all kinds of gaming and drama for practically no useful info for site builders trying to make Drupal sites. Helping those people is ultimately is the goal of this issue.

See also #1637534: Implement project reviews for the less controversial aspect of allowing reviews themselves. From a technical implementation standpoint, this issue should be postponed on that, but I want to keep this "active" to hash out the debate on single vs. multi-axis ratings.

Also, tagging for the projects quality initiative.

Cheers,
-Derek

webchick’s picture

<soapbox>

So over at #1637534: Implement project reviews, unlike here, there was a rather condescending remark about how I wanted Drupal.org to "Be like Apple" because I advocated for dropping the confusing and difficult multi-axis rating and making the Drupal.org project listings match the model of a mobile "App Store" as closely as possible, because these are geared toward severely non-technical users, thus providing much better familiarity and usability for both reviewers and readers.

I want to state unequivocally that this isn't about "Be like Apple." This is about "Be like the rest of the Internet and match user expectations regarding how ratings and reviews are supposed to work." To back up this point, here are some examples.

WordPress Plugin library

Listings show titles, descriptions, and a single average rating.

WordPress's "Plugins" are basically the same as our modules. Their listings show titles, descriptions, some stats... and a single average rating across all versions. Each individual rating is put against the version of WordPress being used, as well as the version of the plugin being used, so you have this neat widget on the front of a page like http://wordpress.org/extend/plugins/all-in-one-seo-pack/ like so:

Drop-down select WordPress version + Plugin version and get an overall red/yellow/gren rating based on if other people say it works or not

I don't think we need to go that crazy, because our top-level version compatibility is built into our revision numbers, but it's interesting to see how they're handling it.

Anyway, the point is, they use a single rating value, not a multi rating value, and they are directly in our target use case. Interestingly, they don't seem to do reviews at all—probably to cut down on the amount of administration and spam fighting—but I agree that more info behind whatever rating was given to a project is really useful, so I wouldn't advocate doing that here. We might even want to make ratings require a review of some kind; I would be on board with that conceptually.

Mozilla Add-ons

Mozilla's add-ons directory is much prettier than WordPress's plugins screen (or ours), but still only uses a single axis for rating:

They require a rating and review to be paired together (but don't make reviewers specify a version, which is too bad), and print nice metrics on the spread of the ratings, like Amazon.com does too:

A rating on the sidebar shows that 143 people have rated this extension 5-stars, versus only 7 who rated it 1-star

Again. Directly in our target use case. Single axis for ratings.

Joomla! Extensions Directory

Joomla: Same use case, different website, still single rating axis.

I find their overview screen kind of overwhelming with all the doo-dads to click on, but something that's kind of interesting is the idea of a "featured" or "editor pick" for a module. In version 200 of this functionality we could possibly use something like the radioactivity module to auto-generate these sorts of badges for projects.

Joomla! de-couples their ratings from their reviews; you'll notice that 1171 users have rated this, whereas only 748 users have reviewed it. They also seem to have some kind of "+1" or "Favourite" system if I'm reading "Votes" correctly.

Header of Joomla extension includes average rating, number of times it's been voted on, and other statistical data

Drupal Apps Module

Finally, if there is any remaining doubt, download a distribution that leverages the Apps project like Panopoly and observe how a consortium of Drupal distribution authors have forked the entire frigging concept of modules just to have ratings and reviews (and icons/screenshots, apparently). And what rating system did they pick? Single-axis five-star, of course. Just like the rest of the web. This is our own community telling us how they want this very use case to work, and this weird fork-y "Apps" concept only grows in popularity for every second that Drupal's default module browser doesn't contain this information:

A listing of available apps within the Panopoly distribution with 5-star ratings

Conclusion

I don't have time to keep going here, but really, take your pick. Almost any similar project with a similar use case matches this model. We should too.

Quoting from Jakob Nielsen:

If 80% or more of the big sites do things in a single way, then this is the de-facto standard and you have to comply. Only deviate from a design standard if your alternative design has at least 100% higher measured usability.

As a side note, we've finally in Drupal core, after a struggle of many years, managed to take the attitude of "Proudly Found Elsewhere" instead of "Not Invented Here" in our approach to finding solutions. We should do so on our own website as well. Rather than try and come up with some kind of unique system that highlights what kind of special snowflakes we all are, let's just admit we're actually not all that special and solve this problem the way other people have already solved it. Our users will be thankful for it.

</soapbox>
dww’s picture

Apologies if my off-hand comment in the other thread came across as condescending. That's not at all what I intended. When I said "be like Apple" I was thinking about the various places that Update manager is (foolishly) compared to Apple's "Software update" feature, and it was frustration about that which was bleeding into my comments about this topic. Anyway, I didn't mean we were just blindly following the 800 pound gorilla for no good reason, which is perhaps what it sounded like, and probably why it came across so poorly.

With that said, thanks for the thorough research and explanations. I still think this is a bad approach, but a) clearly I'm going to loose this debate and b) I'm probably wrong. ;) So, I relent. Knock yourselves out. I need to focus on d.o D7 porting anyway, so if y'all want single-axis 5 star since that's what everyone expects and you think it's the right approach, I'm no longer going to get in the way. I'm tired of both being a SPoF and letting perfect be the enemy of good. Especially since my idea of perfect in this case is probably worse than your idea of good. ;)

Thanks/sorry,
-Derek

AmyStephen’s picture

Removing my comment due to chx's reaction.

chx’s picture

Just because others screw up it does not mean Drupal should screw up, too. There's nothing in the mob mentality we should borrow. Clueless people have no place whatsoever to place a value on a module. Just because you are so stupid you didn't read the readme you should not be allowed to post BR0KEN!!!!!!11!!! Google needed to implement reply to review-ratings functionality because of this. A very very strong -1.

If you make it a permission and give it git vetted users, we can talk.

Edit: I unfollowed this issue. I have nothing more to say.

rileyhuff’s picture

Yeah there are a few things to consider about reviewers and the module version process. How well someone knows the module they are reviewing is a rather important thing to know. You always see the review from the person "I have not actually hiked in these boots yet" useless reviews that may bring a rating down but ultimately I think a good module will get the protection of the herd pretty quickly. I like the idea of being logged into D.O in order to create a review, Five star reviews and having your review subject to thumbs up and down. If a review hits a certain number of thumbs down maybe it gets tossed along with its rating.

Also I can see great modules getting mediocre reviews because documentation is lacking, making it hard for someone new to a complex module to use it. Reviews are a great way not only to inform the community of the modules reputation and potential, but also to provide feedback beyond the issue queue to the module developer and possibly get others interested in helping with the pain points folks are having. And I am sure many of us would like to have a place to publicly 5 star and thank some of the excellent developers here whose work we are using.

I'm happy to help out with this review project in any way.

RobW’s picture

For a large site example of multi-star ratings, eBay seller ratings are 4 tiered:

  • Item as described -- (star rating) -- (number of raters)
  • Communication -- (star rating) -- (number of raters)
  • Shipping time -- (star rating) -- (number of raters)
  • Shipping and handling charges -- (star rating) -- (number of raters)

Random example: http://myworld.ebay.com/jeancompany/&_trksid=p3984.m1423.l2754

RobW’s picture

There are also "watched" and "plus one" style ratings.

Watched (e.g, github) helps show the interest around a project. Watchers aren't necessarily judging a final product. They may use the product and like it, think it's a good idea but haven't had a chance to use it yet, or aren't ready to use it in its current state (issues, incompleteness, etc.) but are interested in seeing where it goes. Watched has the benefit of being compatible with incomplete/development projects.

Plus One is usually used to avoid the problems that sometimes occur with negative responses. If a subject is divisive or intense (see the recent Drupal COC groups thread) it helps avoid bashing and downvoting. It shows how many people have been helped by a post/project/whatever, and can be made more weighty by giving voters a limited amount of upvotes, or only allowing users to upvote after reaching a certain level of experience in the community (see Stack Exchange sites, although with more experience you gain the ability to downvote too).

nedjo’s picture

On the track of doing more with the data we already have....

Re watched, we don't have a specific "watch" action for projects, but maybe there are some proxy measures we might consider:

  • Number of users who follow one or more issues on a project. Of course, this is partially determined by the number of issues. Might help to filter by issue category, e.g., followers of feature requests.
  • Number of subscriptions to the RSS feed of a project's releases. Not sure if these are much used--I've personally never used them. Probably not too useful.

I wonder about a "blogged about" measure. E.g., add a field that the maintainer can use to link to blog mentions, and maybe a checkbox to indicate if the post is by the maintainer or her/his company (to allow skipping these in summary stats). These data would have a documentation usage - provide links to blog posts relevant to the module - but could also serve as a fairly strong indication of community notice/interest.

lvto2000’s picture

This post (Post #65) is the perfect example of why Drupal needs module ratings. I don't care how right or wrong this poster/developer may be. This is just plain rude and obnoxious.

Senpai’s picture

Version: 6.x-1.x-dev » 7.x-2.x-dev

This needs to be tracking the D7 version.

hass’s picture

#61 from dww sounds best to me. It gives us a lot more details. It sounds not reasonable to require git access to allow a user a rating like chx said.

Sometimes I wish there would be a "feeled bugfreeness" rating per version, too. Today it's really a mess to find out if a module is of good quality or not. A combination of well known good quality developers with long history compared with number of installation 5000+ has just failed for me. My strong position is, don't use modules below 2000 installations because of bad quality in most cases or at least I need to expect a lot of issues. Such a rating should give more than 6 stars, it should allow a selection x of 10, no less. Allowing comments on ratings would also be very helpful to get a better understanding about the module quality.

I'm sure such a rating would be very good for module quality in a big picture. We can only win in the long term, nevertheless a few maintainers may receive a bad rating also on popular projects from time to time. This will go down in the masses if it's not reasonable. We should just limit the voting/rating to logged in users from my point of view.

naught101’s picture

I would make the point that rating on its own is not valuable. What is valuable is comparison between similar modules. If there is no way to appropriately select similar modules, and show their descriptions and stats, as well as ratings, together on the same page, then ratings may well be misleading. This is somewhat supported by the plethora of (usually out of date) module comparison pages in the docs section of d.o.

Some sites that do side-by-side comparison of software that work really well:

Drupalmodules.com has been mentioned, but like drupal.org, it is also sorely lacking in ability to view alternatives together, as search is often a poor way to find things, when you're not exactly sure what you're looking for, and the drupal.org module categories are pretty limited (See #480356: Implement Community Tags for d.o projects). I have never really found the drupal.org "related projects" to be all that reliable, except on the most popular modules.

mgifford’s picture

I quite like the way that https://www.ohloh.net does it. Also with showing recent activity. All that is useful in evaluating projects.

mgifford’s picture

Issue summary: View changes
Issue tags: +maintenance, +prairie

Noting earlier GDO discussion about ratings and more about the project page:
https://groups.drupal.org/node/144604

Also added tags to the Prairie Initiative.

EDIT: Useful video looking at rating systems like this Building Web Reputation Systems.

YesCT’s picture

Issue tags: -Drupal.org priority, -maintenance +maintain

priority tag was two years ago, removing tag.
current process is different, and https://www.drupal.org/roadmap has the priorities
also, I think maintain is the tag for similar issues.

mgifford’s picture

Thanks @YesCT for nudging these forward, cleaning up the tags and adding links to where the current process is.

drumm’s picture

Project: Project » Drupal.org customizations
Version: 7.x-2.x-dev » 7.x-3.x-dev
Component: Projects » User interface

Any implementation here should be separate from project itself. So we can keep project module focussed.

dww’s picture

...or an optional add-on module that enhances Project module. Something like Project issue voting, which extends project_issue to add issue voting functionality. This doesn't necessarily *have* to be totally d.o-specific, but I agree it shouldn't be in the main parent module.

Thanks,
-Derek