You don't have to be an expert (more formally defined as someone with ten
years of experience in a field) to spot unscientific thinking. I don't think
you're an expert either so maybe you should just leave expertise out of it.
More to the point, the question in my mind is, how much money did the
Foundation waste on this "study" that you can't draw valid, generalizable
The distinction between quantitative and qualitative is for naught -
you can, and should, quantify what you mean when making qualitative
assessments. Especially if you will use them to bolster your arguments
about how to spend millions of dollars of donated money.
On Fri, May 8, 2009 at 3:51 AM, Delirium <delirium(a)hackish.org> wrote:
Quite frankly the advice that you should only use
five subjects makes no
sense. The appeal to Nielsen's authority is not going to work on me or
anyone else who understands why the scientific method exists. It's
unscientific thinking and it's going cause to you waste money. You're
to draw conclusions based on results that simply
aren't valid, and you
know it until the study is over and you
didn't make progress.
I would caution people unfamiliar with the scientific literature in the
area from making overbroad about fields outside their area of expertise.
There are pros and cons of quantitative and qualitative studies in
human-computer interaction (HCI), and in many cases qualitative studies
give more useful information, especially for fairly early stage design
questions. In particular, it's quite easy to get statistically rigorous
but in-practice useless data from quantitative studies. Those are very
good when you have precisely one variable you want to measure (or maybe
a few), but in many HCI studies, especially with open-ended interface
design questions (like "make Wikipedia better") there are thousands of
possible variables, and a major purpose of a study is to better
understand the concerns of the domain and figure out what the relevant
variables even *are*. Quantitative studies are much more useful when
you've fixed your interface but have a handful of final decisions to
make---do we want a particular feature to have slight variant A or B,
while everything else stays fixed.
foundation-l mailing list