Over the summer the APC (Association for Progressive Communication) "Action Kit" team engaged Web Networks to generate a report on the usability of the Drupal content management system.

The report has been ready for a little while now, but we wanted a green light from APC to release the document to the wild. The document is released under the Creative Commons Attribution-Noncommercial-ShareAlike 2.5 License.

The report can be accessed via http://www.web.net/support/DrupalUsabilityReport.pdf

Report Summary

The Drupal content management system (CMS) has gained a reputation for being a developer centric platform that is difficult to use. While Drupal does have usability issues, the overall conclusion of this report is that the tools Drupal provides to accomplish the most common administrative tasks associated with managing a website are all usable.

This report is divided into two areas of study: a usability guideline review and the stakeholder
review of Drupal usability.

The usability guideline review evaluates how well Drupal’s administrative tools adhere to a
subset of the Research-Based Web Design & Usability Guidelines that are specific to web-based
forms. Each of the forms Drupal provides to accomplish common administrative tasks
was examined individually to determine if they:
• Distinguish clearly and consistently between required and optional data entry fields
• Detect errors automatically
• Minimize user data entry
• Label data entry fields clearly
• Place labels in close proximity to data entry fields
• Label form buttons clearly
• Allow users to see their entered data
• Use radio buttons and checkboxes correctly
• Group data entry by method type
• Maintain proper tab indexing in forms

Each Drupal form, almost without exception, adhered to all of the usability guidelines. The
grades Drupal received for each guideline ranged between a ‘B-‘ and ‘A+’, with an overall ‘A-’
average.

The stakeholder review of Drupal usability grades the usability of common administrative tools
based on the experiences of Web Networks’ staff and the clients they serve, taking into
consideration alternate usability opinion pieces and usability surveys.
Several conclusions were made based on the stakeholder review. The first and perhaps most
important conclusion was that the tools Drupal provides to complete common administrative
tasks are sufficiently usable to maintain an installed website. The second conclusion was that
many usability issues that have been identified in the past are regularly addressed in
subsequent releases of Drupal. There is an expectation that this trend will continue.

Comments

jmarkantes’s picture

This is a must-read for people really interested in improving Drupal. The highs are neat to see, but there are still some lows that should be addressed. For example, under "attaching files and asset management", the upload module received a "D-". (Which, reading their report, is really addressing a different need, notably asset management.)

Just saying, let's celebrate the highs, and then focus on bringing everything on up.
J

rbrooks00’s picture

This has some value, but after reading through this it seems like these are sort of minimum standards. So it is definitely good that we scored well in these areas, but to me it feels like overall Drupal comes in with maybe a C or B GPA for usability. The comments around the management of attachments are really good, but I think it applies more broadly than that for numerous modules - things like the image module as well.

IMHO we shouldn't be striving to meet minimum standards, we should be striving to lead. There are lots of sites out there pushing the envelope of usability, and it'd be great if Drupal could make moves toward that as well. Dries mentioned some of the admin screens for Sugar CRM as an example. I'd also put the functionality of their competitor SalesForce.com there as well.

Think about interfaces like can be found with Tabblo could do for blogging, photoblogging, image galleries, artist sites, etc. Or even products like the Slideshow Pro Director for image management.

Even something like JotSpot has things that we can all learn from. I was evaluating this and had it up and running and was being productive in a matter of maybe 30 minutes. That was not the case when I was first learning drupal almost 2 years ago.

I also think that this old article (2004) from Adaptive Path makes some good points, some of which have already begun to be addressed.

And I know, I know the usual stuff about rolling patches... but it is important for people to understand there is a problem and agree there is a problem before it can be solved :)

===
BuyBlue.org | Lullabot

escoles’s picture

Evaluating an application based other apps' AJAX-driven forms is a bit like evaluating all cars with reference to a Lexus.

Drupal is deployed in many, many different configurations on a wide variety of platforms. If the developers spend a lot of time tweaking form presentation so that it works well on medium-res laptop screens (1024x768+), then small-screen presentation (PDA, etc.) will suffer; if they use AJAX, then lightweight and cross-platform presentation suffer.

Best, I think, to focus on laying the groundwork so that people can write themes and modules that manipulate the form [X]HTML as needed, and use AJAX and related techniques in a gracefully-degrading fashion.

[edited to fix grammar]

rbrooks00’s picture

It isn't just about AJAX, that isn't the point I was making. The point I was making is that the standards for receiving a "good grade" in this usability survey are incredibly low. It is good Drupal performed well but that should in no way be taken to mean that we are ok and don't really need to focus on usability. IMHO that is Drupal's biggest weakness at this point.

Take a look at that JotSpot product for example - there is very little ajax involved there. But from the time you create your free account to being able to do productive things is less than 30 minutes. That is simply not the case with Drupal. It doesn't have to do with instructions and documentation, it has to do with intuitive user interfaces and ease of installation and configuration. Some good progress toward this is being made with 5.0, but it needs a lot more work.

===
BuyBlue.org | Lullabot

moshe weitzman’s picture

IMO, it is up to distributions to make out of the box experience better. We are all evaluating the stock drupal.org profile right now b ut there should/will/better_be more distros coming. Those imstall the right modules and config for your particular app and "get you up and running in 30 mins"

distributions are a new feature of 5.0

Dries’s picture

Printed it, and started reading it on my daily commute. So far it looks great. All the suggestions are worth considering. We're still accepting usability improvements for the Drupal 5.0 release, so let's translate some of the recommendations into code!

Thanks for the excellent report!

davidm’s picture

Are there any recent reports on Drupal's level of accessibility, for example compared to WAI standards?

sepeck’s picture

In general that's more related to the specific theme you choose or design for your site.

-Steven Peck
---------
Test site, always start with a test site.
Drupal Best Practices Guide -|- Black Mountain

-Steven Peck
---------
Test site, always start with a test site.
Drupal Best Practices Guide

tang148steve’s picture

Yet I'm not sure it satisfies all the criteria. Try creating a page with only a space in subject or title - 4.7.3 takes it in but...

Other than that, it's great !!! This is after trying out about half of dozen others and a few months of drupalling ...

escoles’s picture

I've read the report, and much as I'd like to read it in a positive light, I think it has to be read closely and carefully to understand its value. The "scores", I'm afraid, are pretty useless, as are some of the evaluations. Overall, I fear this report simply obfuscates the issues by putting metrics in places where they're not merited.

One example leapt out at me: Guideline 8, "Use Checkboxes and Radio Buttons Correctly", and I'll focus on one particular aspect of that guideline: "Checkboxes should only ever be used for options where more than one option may be selected."

That's clearly wrong on several levels:

  • It violates the "checkbox metaphor": On a physical form, checkboxes are often used to select single options. Un-checked is a pretty clear indication of the option "don't do this."
  • It violates the common usage of checkboxes as single-selectors, for example as an opt-in / opt-out indicator on registration forms. ("I would like to receive email ...")
  • It violates common sense in design practice: Making one selection (yes/no) with two elements (two radio buttons) will generally require more cognitive processing.
  • It will lead to unclear form constructions: Imagine the Menu management interactions with dual radio buttons replacing the checkboxes. As is, we have to simply see a box selected, or not: A simple linear array. With dual radio buttons, we'd have to visually/coginitively parse a two-dimensional array, and then understand the dimensions, with no common usage pattern to guide us. (For example: Which is "on", right or left?)

Interestingly, they qualify their statement later by pointing out that it's based on an "expert opinion", and not on usability research. But considering the obvious impact of replacing a simple single-select modality (a lone "on/off" checkbox) with a dual-select modality (a more visually complex choice of one "on" element versus another "off" element) throughout designs, and consdering that it's a formulation clearly in opposition to the common usage, I think a little data would be nice.

The point of picking on this is to point out that any heuristic judgement is based on some reasoning, and should also be based on some evidence. In the "Guidelines" section, mostly this is impressionistic stuff, masked by an objective tone. "We are evaluating against GUIDELINES. That we just made up."

The "stakeholder review" part is very valuable. Again, though, it should be read closely.

andremolnar’s picture

It seems as those this single guideline has generated the most discussion thus far. A couple of points.

It should be noted that Drupal received a grade of A (top marks) for its use of checkboxes and radio buttons.

The purpose of the guideline evaluation was not to define the guidelines - but rather evaluate against guidelines that have been defined by others.

Rather than review each possible use of checkboxes and radio buttons (and go into the minutia of each individual guideline) a group of guidelines were evaluated under a single amalgamated guideline that was greatly simplified (which appears to be the greatest point of contention). In short, the question posed was: does Drupal use checkboxes and radio buttons correctly? The answer is clearly YES (and we might add 'very well').

The relative importance of this guideline is low. (Compared to [say] displaying meaningful error messages). It is not because use of checkboxes and radio buttons is not important, but rather there is less research evidence surrounding the usage of these elements and most guidelines surrounding their usage are based on expert opinion.

The single area where a recommendation was made was with regard to the administrative option to expand menu items.

Currently:
[ ] Expanded
If selected and this menu item has children, the menu will always appear expanded.

The feeling was that this particular option might be clearer (and therefore marginally more usable) if it was:

If selected and this menu item has children, the menu will always appear:

(*) Collapsed
( ) Expanded

Either implementation is 'correct' (depending on your perspective, objective or who you want to listen to). But, as others have pointed out, such UI decisions are based on many more factors than strictly adhering to some guideline. If I were to prioritize the implementation of this particular change - it would be just ahead of fixing a misplaced comma.

At any rate, I anticipate that more energy will be spent addressing areas where there is clearer evidence of usability issues - rather than arguing the fine details of something that is already deemed to be usable 'as is'.

The feedback is appreciated, and future versions of the document will correct the mistake of over simplifying the guidelines surrounding checkboxes and radio buttons.

andre

leoklein’s picture

I thought the paper was good in as much as it was using heuristics to judge the usability of administering Drupal. I think a good next step would be actual usability testing of users.

whereisian’s picture

The pdf seems to have been removed. I found a copy in google's cache.

rcross’s picture

I know this is an old thread, but I came across it when looking for a usability report and figured I'd post the corrected url for any future users

http://www.web.net/support/DrupalUsabilityReport.pdf