On Thu, May 7, 2009 at 11:58 PM, Brian <Brian.Mingus(a)colorado.edu> wrote:
Quite frankly the advice that you should only use five
subjects makes no
sense. The appeal to Nielsen's authority is not going to work on me or
anyone else who understands why the scientific method exists. It's
unscientific thinking and it's going cause to you waste money. You're going
to draw conclusions based on results that simply aren't valid, and you won't
know it until the study is over and you didn't make progress.
Careful analysis of site data could allow you to draw some conclusions. I'm
curious how you're planning to go about that. Dependent/independent
variables?
An exercise in statistical thinking: when everyone or almost everyone
cites problem X, how many people does it take to reach statistical
significance that X is a problem worth addressing? Even if the
results are a statistical fluke and in reality only 20% of new users
run into trouble with problem X, that's still a problem worth
addressing.
The fact that so many of the 15 people had the same problems, and
those problems also align with common sense, is a strong indication
that the study has found some things worth fixing.
There is more than one way to come to reliable conclusions. Any time
I see someone invoking "the [singular] scientific method", as if there
is only one and it is set in stone and universally agreed upon by all
rational people, I have trouble taking them seriously. See
[[Talk:Scientific method]].
-Sage (User:Ragesoss)