On Thu, Dec 3, 2009 at 2:13 PM, Charles Matthews charles.r.matthews@ntlworld.com wrote:
No, but I can think much better ways of framing the question than following up WSJ article would lead to. Studies and articles written by people not really aware of how our communities function are not really good places to start, if the issue is how to improve that functioning. It seems pretty clear that if you frame the question too loosely, you get a recital of some beefs that are brought up whatever the occasion.
Indeed.
Please read this page, starting at "Other kinds of errors are more characteristic of poor science." down to "But not paying attention to experiments like that is a characteristic of cargo cult science." "
http://www.lhup.edu/~DSIMANEK/cargocul.htm
If you care about the research on Wikipedia at all, or making policies from that research... and you are only going to read one thing suggested me this year— this should be the thing you read. (Well, I recommend everyone go and buy and read the feynman biographies…)
I'm not suggesting that anyone is engaging in cargo cult science so much as just saying that we do not yet know much of anything about researching Wikipedia. Unfortunately, people are being rewarded for making loud conclusions even though many are easily dismissed as uncertain because of confounding factors which are obvious to people experienced in Wikipedia. This does not encourage the kind of careful fundamentals research required before we can make real progress.
I'm interested in knowing which, if any, experienced Wikipedians have ever participated in the peer review of an article studying Wikipedia. I have not. Peer review isn't a magic bullet, but I'm surprised at the number of obvious and easily correctable flawed clams (e.g. that the dumps contain ALL edits so no effort needs to be taken to correct for any sampling bias), and I wonder if anyone with significant first hand experience is providing input.