[Foundation-l] Usability Study Results (Sneak Preview)

[Aryeh Gregor]

On Thu, May 7, 2009 at 11:58 PM, Brian <Brian.Mingus at colorado.edu> wrote:
> Quite frankly the advice that you should only use five subjects makes no
> sense. The appeal to Nielsen's authority is not going to work on me or
> anyone else who understands why the scientific method exists.

Experience shows that most people end up being very similar when it
comes to usability.  Most problems show up repeatedly even with groups
of five people.  If you run the tests on a hundred people, you're
going to get a somewhat more accurate picture, but not enough to
justify the extra expense.  It's much better to run a five-person
study, assume that any objections raised by (say) at least three are
representative, fix those, and run another few five-person studies on
the fixed software for the same cost.

You don't need large sample sizes if something is regular enough.  You
only need high sample sizes if the object of your study is variable
enough to require it.  That's usually the case in pharmacological
studies, for instance, but that doesn't mean it's true everywhere.  If
you have a tiny standard deviation, then a study of five people could
provide very clear conclusions.  It all depends on your data.