[Foundation-l] Usability Study Results (Sneak Preview)

[Brian]

You don't have to be an expert (more formally defined as someone with ten
years of experience in a field) to spot unscientific thinking. I don't think
you're an expert either so maybe you should just leave expertise out of it.
More to the point, the question in my mind is, how much money did the
Foundation waste on this "study" that you can't draw valid, generalizable
conclusions from?
The distinction between quantitative and qualitative is for naught -
you can, and should, quantify what you mean when making qualitative
assessments. Especially if you will use them to bolster your arguments
about how to spend millions of dollars of donated money.

On Fri, May 8, 2009 at 3:51 AM, Delirium <delirium at hackish.org> wrote:

> Brian wrote:
> > Quite frankly the advice that you should only use five subjects makes no
> > sense. The appeal to Nielsen's authority is not going to work on me or
> > anyone else who understands why the scientific method exists. It's
> > unscientific thinking and it's going cause to you waste money. You're
> going
> > to draw conclusions based on results that simply aren't valid, and you
> won't
> > know it until the study is over and you didn't make progress.
> >
>
> I would caution people unfamiliar with the scientific literature in the
> area from making overbroad about fields outside their area of expertise.
> There are pros and cons of quantitative and qualitative studies in
> human-computer interaction (HCI), and in many cases qualitative studies
> give more useful information, especially for fairly early stage design
> questions. In particular, it's quite easy to get statistically rigorous
> but in-practice useless data from quantitative studies. Those are very
> good when you have precisely one variable you want to measure (or maybe
> a few), but in many HCI studies, especially with open-ended interface
> design questions (like "make Wikipedia better") there are thousands of
> possible variables, and a major purpose of a study is to better
> understand the concerns of the domain and figure out what the relevant
> variables even *are*. Quantitative studies are much more useful when
> you've fixed your interface but have a handful of final decisions to
> make---do we want a particular feature to have slight variant A or B,
> while everything else stays fixed.
>
> -Mark
>
>
> _______________________________________________
> foundation-l mailing list
> foundation-l at lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>