[Foundation-l] Usability Study Results (Sneak Preview)

[Erik Moeller]

2009/5/7 Brian <Brian.Mingus at colorado.edu>:
>>
>> Based on these criteria, the 2,500 users that responded to our survey were
>> filtered down to 500 viable subjects based on their answers to these
>> questions. The team, along with B|P, partnered with Davis Recruiting to
>> contact, filter, and screen these 500 participants based on their Wikipedia
>> contribution history, Wikipedia usage patterns, their given reasons for not
>> contributing, and their talkativeness and openness to discuss their thoughts
>> and actions. From 2,500 users, we ended up with 10 study participants and
>> 3-5 waitlisted participants.

> You went from 2,500 subjects to just 10?

The purpose of a study like this is focused observation of the
behavior of individual human beings. As David has pointed out, for any
study like this there are laws of diminishing returns, and any serious
observation of an individual is time-consuming and costly (raw data is
worthless if you can't analyze it). That's why usability gurus like
Nielsen suggest "5 is enough" for most tests:
http://www.useit.com/alertbox/20000319.html - due to our highly
diverse audience, we chose a larger group, and we split between remote
and lab testing to compensate for biases of both methods. This has
worked well to identify plenty of very obvious usability barriers to
focus on.

There are alternative data collection methods such as large scale
quantitative testing where the level of individual engagement is
limited; those can give you behavioral patterns etc. They can be
useful, too, but are an entirely different thing.
-- 
Erik Möller
Deputy Director, Wikimedia Foundation

Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate