On Thu, Dec 3, 2009 at 2:13 PM, Charles Matthews charles.r.matthews@ntlworld.com wrote:
No, but I can think much better ways of framing the question than following up WSJ article would lead to. Studies and articles written by people not really aware of how our communities function are not really good places to start, if the issue is how to improve that functioning. It seems pretty clear that if you frame the question too loosely, you get a recital of some beefs that are brought up whatever the occasion.
Indeed.
Please read this page, starting at "Other kinds of errors are more characteristic of poor science." down to "But not paying attention to experiments like that is a characteristic of cargo cult science." "
http://www.lhup.edu/~DSIMANEK/cargocul.htm
If you care about the research on Wikipedia at all, or making policies from that research... and you are only going to read one thing suggested me this year— this should be the thing you read. (Well, I recommend everyone go and buy and read the feynman biographies…)
I'm not suggesting that anyone is engaging in cargo cult science so much as just saying that we do not yet know much of anything about researching Wikipedia. Unfortunately, people are being rewarded for making loud conclusions even though many are easily dismissed as uncertain because of confounding factors which are obvious to people experienced in Wikipedia. This does not encourage the kind of careful fundamentals research required before we can make real progress.
I'm interested in knowing which, if any, experienced Wikipedians have ever participated in the peer review of an article studying Wikipedia. I have not. Peer review isn't a magic bullet, but I'm surprised at the number of obvious and easily correctable flawed clams (e.g. that the dumps contain ALL edits so no effort needs to be taken to correct for any sampling bias), and I wonder if anyone with significant first hand experience is providing input.
Gregory Maxwell wrote:
On Thu, Dec 3, 2009 at 2:13 PM, Charles Matthews charles.r.matthews@ntlworld.com wrote:
No, but I can think much better ways of framing the question than following up WSJ article would lead to. Studies and articles written by people not really aware of how our communities function are not really good places to start, if the issue is how to improve that functioning. It seems pretty clear that if you frame the question too loosely, you get a recital of some beefs that are brought up whatever the occasion.
I'm interested in knowing which, if any, experienced Wikipedians have ever participated in the peer review of an article studying Wikipedia. I have not. Peer review isn't a magic bullet, but I'm surprised at the number of obvious and easily correctable flawed clams (e.g. that the dumps contain ALL edits so no effort needs to be taken to correct for any sampling bias), and I wonder if anyone with significant first hand experience is providing input.
Good points. But as (few threads earlier) be lack any dedicated publication to Wikiepdia (and as far as I know Wikimania doesn't issue peer review compilations, and is not seen as a "real" academic conference), works on Wikipedia are submitted to various traditional outlets, and reviewed by traditional experts - which may be really good in their branch of academia, but probably 1) are not Wikipedians and 2) have not read much research on Wikipedia.
I also wonder how many research pieces on Wikipedians are written by scholars who 1) are not Wikipedians 2) do not realize that there is an already large body of literature on the subject...
I am not sure how this could be changed, other than very slowly and over time, as Wikipedia researches and the field itself become more estabilished.
Piotr Konieczny wrote:
Good points. But as (few threads earlier) be lack any dedicated publication to Wikiepdia (and as far as I know Wikimania doesn't issue peer review compilations, and is not seen as a "real" academic conference), works on Wikipedia are submitted to various traditional outlets, and reviewed by traditional experts - which may be really good in their branch of academia, but probably 1) are not Wikipedians and 2) have not read much research on Wikipedia.
I also wonder how many research pieces on Wikipedians are written by scholars who 1) are not Wikipedians 2) do not realize that there is an already large body of literature on the subject...
I'm not sure to what degree this is of concern. Personally, I have on the order of 500 Wikipedia revisions and have a vague notion of how WP policy works - certainly not expert status, but I'd guess on the order of 95th percentile of WP users who have edited at least once. Thinking of my colleagues who also write and review Wikipedia work, this is not atypical. Certainly my GroupLens colleagues understand policy far better than I.
I regularly review for CHI, CSCW, and other high-reputation "traditional" venues, and I also regularly reject work which is lacking in understanding of how Wikipedia works and the existing literature. I see other reviewers doing this as well, and the associate chairs (who manage each paper's review) are generally well-versed in Wikipedia and have similar high standards.
In general, understanding the user community in question is a must for any credible research, so if you can provide evidence that the flaws you identify are leading to systematic problems in the work that is happening, I strongly encourage you to write it up and submit it to the traditional venues. This kind of process critique would be very well received.
HTH,
Reid
wiki-research-l@lists.wikimedia.org