[Foundation-l] Usability Study Results (Sneak Preview)

[Delirium]

Brian wrote:
> Quite frankly the advice that you should only use five subjects makes no
> sense. The appeal to Nielsen's authority is not going to work on me or
> anyone else who understands why the scientific method exists. It's
> unscientific thinking and it's going cause to you waste money. You're going
> to draw conclusions based on results that simply aren't valid, and you won't
> know it until the study is over and you didn't make progress.
>   

I would caution people unfamiliar with the scientific literature in the 
area from making overbroad about fields outside their area of expertise. 
There are pros and cons of quantitative and qualitative studies in 
human-computer interaction (HCI), and in many cases qualitative studies 
give more useful information, especially for fairly early stage design 
questions. In particular, it's quite easy to get statistically rigorous 
but in-practice useless data from quantitative studies. Those are very 
good when you have precisely one variable you want to measure (or maybe 
a few), but in many HCI studies, especially with open-ended interface 
design questions (like "make Wikipedia better") there are thousands of 
possible variables, and a major purpose of a study is to better 
understand the concerns of the domain and figure out what the relevant 
variables even *are*. Quantitative studies are much more useful when 
you've fixed your interface but have a handful of final decisions to 
make---do we want a particular feature to have slight variant A or B, 
while everything else stays fixed.

-Mark