On Thu, Dec 3, 2009 at 2:13 PM, Charles Matthews <charles.r.matthews(a)ntlworld.com> wrote: Indeed. Please read this page, starting at "Other kinds of errors are more characteristic of poor science." down to "But not paying attention to experiments like that is a characteristic of cargo cult science." " http://www.lhup.edu/~DSIMANEK/cargocul.htm If you care about the research on Wikipedia at all, or making policies from that research... and you are only going to read one thing suggested me this year— this should be the thing you read. (Well, I recommend everyone go and buy and read the feynman biographies…) I'm not suggesting that anyone is engaging in cargo cult science so much as just saying that we do not yet know much of anything about researching Wikipedia. Unfortunately, people are being rewarded for making loud conclusions even though many are easily dismissed as uncertain because of confounding factors which are obvious to people experienced in Wikipedia. This does not encourage the kind of careful fundamentals research required before we can make real progress. I'm interested in knowing which, if any, experienced Wikipedians have ever participated in the peer review of an article studying Wikipedia. I have not. Peer review isn't a magic bullet, but I'm surprised at the number of obvious and easily correctable flawed clams (e.g. that the dumps contain ALL edits so no effort needs to be taken to correct for any sampling bias), and I wonder if anyone with significant first hand experience is providing input.