Piotr Konieczny wrote:
Good points. But as (few threads earlier) be lack any dedicated publication to Wikiepdia (and as far as I know Wikimania doesn't issue peer review compilations, and is not seen as a "real" academic conference), works on Wikipedia are submitted to various traditional outlets, and reviewed by traditional experts - which may be really good in their branch of academia, but probably 1) are not Wikipedians and 2) have not read much research on Wikipedia.
I also wonder how many research pieces on Wikipedians are written by scholars who 1) are not Wikipedians 2) do not realize that there is an already large body of literature on the subject...
I'm not sure to what degree this is of concern. Personally, I have on the order of 500 Wikipedia revisions and have a vague notion of how WP policy works - certainly not expert status, but I'd guess on the order of 95th percentile of WP users who have edited at least once. Thinking of my colleagues who also write and review Wikipedia work, this is not atypical. Certainly my GroupLens colleagues understand policy far better than I.
I regularly review for CHI, CSCW, and other high-reputation "traditional" venues, and I also regularly reject work which is lacking in understanding of how Wikipedia works and the existing literature. I see other reviewers doing this as well, and the associate chairs (who manage each paper's review) are generally well-versed in Wikipedia and have similar high standards.
In general, understanding the user community in question is a must for any credible research, so if you can provide evidence that the flaws you identify are leading to systematic problems in the work that is happening, I strongly encourage you to write it up and submit it to the traditional venues. This kind of process critique would be very well received.
HTH,
Reid