[Foundation-l] Usability Study Results (Sneak Preview)

[Sage Ross]

On Thu, May 7, 2009 at 11:58 PM, Brian <Brian.Mingus at colorado.edu> wrote:
> Quite frankly the advice that you should only use five subjects makes no
> sense. The appeal to Nielsen's authority is not going to work on me or
> anyone else who understands why the scientific method exists. It's
> unscientific thinking and it's going cause to you waste money. You're going
> to draw conclusions based on results that simply aren't valid, and you won't
> know it until the study is over and you didn't make progress.
>
> Careful analysis of site data could allow you to draw some conclusions. I'm
> curious how you're planning to go about that. Dependent/independent
> variables?
>

An exercise in statistical thinking: when everyone or almost everyone
cites problem X, how many people does it take to reach statistical
significance that X is a problem worth addressing?  Even if the
results are a statistical fluke and in reality only 20% of new users
run into trouble with problem X, that's still a problem worth
addressing.

The fact that so many of the 15 people had the same problems, and
those problems also align with common sense, is a strong indication
that the study has found some things worth fixing.

There is more than one way to come to reliable conclusions.  Any time
I see someone invoking "the [singular] scientific method", as if there
is only one and it is set in stone and universally agreed upon by all
rational people, I have trouble taking them seriously.  See
[[Talk:Scientific method]].

-Sage (User:Ragesoss)