The target for this round of testing was "sub-admins" familiar with other content management systems. They were exposed to Drupal 6 with CCK. Eye-tracking data was gathered, and we saw the animated results - not very pretty.
The first challenge - to build a form with a few fields. The first place they go - Site Building. Logical, enough, but that's not where we put "content types" (whatever the hell those are). Frankly, I would have expected the term "node" to be the one making users go "Whaaaa?", but apparently "content types", "story", "book page", etc. were enough to trip people up.
Once people finally reached content type creation, the eye-tracker showed that people just didn't see those essential tabs along the top. I tend to agree with comments from the audience that the Garland theme's very untablike rendering of form tabs probably contributes significantly to this problem.
Tasks 2 & 3 - creating a user account with appropriate permissions, and classifying content, went considerably more smoothly (although the fact that the subjects were library staff certainly helped in the taxonomy area).
Task 4 - No one made it here.
It's interesting that people use the initial front page as a step-by-step tutorial - and they blow it away when they create their first node, which is disconcerting.
Help proved unhelpful - it's not searchable, not task-oriented, and there's no glossary.
The admin panel is overwhelming. This is no surprise.
People click on everything to see what's available, so hiding fieldsets isn't as helpful to newbies as one might expect.
Drupal 6 usability improvements gone wrong:
We should be targeting Drupal 6 contrib and Drupal 7 core for improvements based on this experience (limited as it was - small sample size, one particular target role).
See the Usability group for details. How the rest of us can help: plug specifics from the detailed results into project issues (titled in the form- "Usability UMN: <title>"). Also see User Experience Goals Draft.
A couple of my own usability thoughts: