You don't have to be an expert (more formally defined as someone with ten years of experience in a field) to spot unscientific thinking. I don't think you're an expert either so maybe you should just leave expertise out of it. More to the point, the question in my mind is, how much money did the Foundation waste on this "study" that you can't draw valid, generalizable conclusions from? The distinction between quantitative and qualitative is for naught - you can, and should, quantify what you mean when making qualitative assessments. Especially if you will use them to bolster your arguments about how to spend millions of dollars of donated money.
On Fri, May 8, 2009 at 3:51 AM, Delirium delirium@hackish.org wrote:
Brian wrote:
Quite frankly the advice that you should only use five subjects makes no sense. The appeal to Nielsen's authority is not going to work on me or anyone else who understands why the scientific method exists. It's unscientific thinking and it's going cause to you waste money. You're
going
to draw conclusions based on results that simply aren't valid, and you
won't
know it until the study is over and you didn't make progress.
I would caution people unfamiliar with the scientific literature in the area from making overbroad about fields outside their area of expertise. There are pros and cons of quantitative and qualitative studies in human-computer interaction (HCI), and in many cases qualitative studies give more useful information, especially for fairly early stage design questions. In particular, it's quite easy to get statistically rigorous but in-practice useless data from quantitative studies. Those are very good when you have precisely one variable you want to measure (or maybe a few), but in many HCI studies, especially with open-ended interface design questions (like "make Wikipedia better") there are thousands of possible variables, and a major purpose of a study is to better understand the concerns of the domain and figure out what the relevant variables even *are*. Quantitative studies are much more useful when you've fixed your interface but have a handful of final decisions to make---do we want a particular feature to have slight variant A or B, while everything else stays fixed.
-Mark
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l