Hi there, Great project; massive but will be much appreciated. We did something similar for empirical studies of Open Source, recently accepted at ACM Computing Surveys (PDF pre-print available here [1], article not in print until 2012 (!! that's another email entirely, bah)) I recognize the need to cut down the number of articles for review, we reviewed around 600 and that was a multi-year effort. We did that mainly by excluding conceptual (hence empirical) or passing reference articles (ie we did a two-step filter on many more articles), but were forced to only do journal articles for updates during the (long) revision process. I regret that necessity, it decreases the utility of the work. Given the publication venues of choice for many academics in this community I do wonder if you aren't shooting yourself in the foot by excluding peer-reviewed conferences and restricting to journals. Personally I'd rather read a review that included the top journals and top conferences than one that included all journals. Or even rather read a review over a shorter time period that included publications over journals and conferences, or on more specified topics. The interesting question is, "what do we know about wikipedia" not "what did we publish in journals about wikipedia". In particular you will find you have systematically excluded the contribution of HCI authors. Given the commendable and massive effort you are providing (and your approach to coverage below is really interesting), getting that wrong at the outset seems a shame. Best regards, James Howison [1] Crowston, K., Wei, K., Howison, J., and Wiggins, A. (2012). Free (libre) open source software development: What we know and what we do not know. ACM Computing Surveys, 44(2): http://floss.syr.edu/content/freelibre-open-source-software-development-wha… On Mar 14, 2011, at 13:58, Chitu Okoli wrote: _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

As an HCI/CS researcher who has published at top peer-reviewed conferences about Wikipedia, but not journals, I'd like to echo James' statements. Journals are not the norm in CS/HCI research. Knowledge is shared through conferences, not journals. On 3/14/11 11:32 AM, James Howison wrote: > Hi there, > > Great project; massive but will be much appreciated. We did something similar for empirical studies of Open Source, recently accepted at ACM Computing Surveys (PDF pre-print available here [1], article not in print until 2012 (!! that's another email entirely, bah)) > > I recognize the need to cut down the number of articles for review, we reviewed around 600 and that was a multi-year effort. We did that mainly by excluding conceptual (hence empirical) or passing reference articles (ie we did a two-step filter on many more articles), but were forced to only do journal articles for updates during the (long) revision process. I regret that necessity, it decreases the utility of the work. > > Given the publication venues of choice for many academics in this community I do wonder if you aren't shooting yourself in the foot by excluding peer-reviewed conferences and restricting to journals. Personally I'd rather read a review that included the top journals and top conferences than one that included all journals. Or even rather read a review over a shorter time period that included publications over journals and conferences, or on more specified topics. The interesting question is, "what do we know about wikipedia" not "what did we publish in journals about wikipedia". In particular you will find you have systematically excluded the contribution of HCI authors. > > Given the commendable and massive effort you are providing (and your approach to coverage below is really interesting), getting that wrong at the outset seems a shame. > > Best regards, > James Howison > > [1] Crowston, K., Wei, K., Howison, J., and Wiggins, A. (2012). Free (libre) open source software development: What we know and what we do not know. ACM Computing Surveys, 44(2): > http://floss.syr.edu/content/freelibre-open-source-software-development-wha… > > > On Mar 14, 2011, at 13:58, Chitu Okoli wrote: > >> Hi everyone, >> >> We are a research group conducting a systematic literature review on Wikipedia-related peer-reviewed academic studies published in the English language. (Although there are many excellent studies in other languages, we unfortunately do not have the resources to systematically review these at any kind of acceptable scholarly level. Also, our study is about Wikipedia only, not about other Wikimedia Foundation projects. However, we do include studies about other language Wikipedias, as long as the studies are published in English.) We have completed a search using many major databases of scholarly research. In a separate thread, we will also talk about research questions related to our review. >> >> Thanks for your help. >> >> Chitu Okoli, Concordia University, Montreal, Canada >> (http://chitu.okoli.org/professional/open-content/wikipedia-and-open-content…) >> Arto Lanamäki, University of Agder, Kristiansand, Norway >> Mohamad Mehdi, Concordia University, Montreal, Canada >> Mostafa Mesgari, Concordia University, Montreal, Canada

James and Travis, you bring up a point that we have struggled back and forth with for several months. We really, really would like to include conference articles, but we just can't see how we could handle many more articles than what we've got now. We've been working on and off on this project for over two years now. (You can find works in progress at the link at the bottom to my website.) We'd like to get it done eventually, and we can only handle so many articles. We considered including top-tier conferences, but the question is, what is a "top conference"? In trying to answer this, we looked at a couple of sources: * Top Tier and 2nd tier conferences from http://webdocs.cs.ualberta.ca/~zaiane/htmldocs/ConfRanking.html * A-ranked conferences in Information and Computing Sciences from http://lamp.infosys.deakin.edu.au/era/?page=cforsel10 * We also considered including all WikiSym articles on Wikipedia We identified which of the 1,500 conference papers from http://en.wikipedia.org/wiki/User:Moudy83/conference_papers were "top conferences" by those definitions, and we found over 400. On top of our 600 journal articles and doctoral theses, we think 1,000 papers is just too much for us to handle. If we could somehow narrow it down to 100 relevant conference papers, we could add that in, but no more. However, how do we select which conferences are "must includes" while unfortunately leaving out the rest? We just don't know how to do this in a non-arbitrary, objective manner that would truly identify the top 100 conference papers on Wikipedia that contribute to scholarly knowledge. Any ideas on how to do this would be very much appreciated. Regards, Chitu -------- Message original -------- Sujet: Re: [Wiki-research-l] Request to verify articles for Wikipedia  literature review De : Travis Kriplean <travis(a)cs.washington.edu> Pour : Research into Wikimedia content and communities     <wiki-research-l(a)lists.wikimedia.org> Date : 14/03/2011 3:46 PM As an HCI/CS researcher who has published at top peer-reviewed conferences about Wikipedia, but not journals, I'd like to echo James' statements. Journals are not the norm in CS/HCI research. Knowledge is shared through conferences, not journals. On 3/14/11 11:32 AM, James Howison wrote: Hi there, Great project; massive but will be much appreciated.  We did something similar for empirical studies of Open Source, recently accepted at ACM Computing Surveys (PDF pre-print available here [1], article not in print until 2012 (!! that's another email entirely, bah)) I recognize the need to cut down the number of articles for review, we reviewed around 600 and that was a multi-year effort. We did that mainly by excluding conceptual (hence empirical) or passing reference articles (ie we did a two-step filter on many more articles), but were forced to only do journal articles for updates during the (long) revision process.  I regret that necessity, it decreases the utility of the work. Given the publication venues of choice for many academics in this community I do wonder if you aren't shooting yourself in the foot by excluding peer-reviewed conferences and restricting to journals.  Personally I'd rather read a review that included the top journals and top conferences than one that included all journals.  Or even rather read a review over a shorter time period that included publications over journals and conferences, or on more specified topics. The interesting question is, "what do we know about wikipedia" not "what did we publish in journals about wikipedia".  In particular you will find you have systematically excluded the contribution of HCI authors. Given the commendable and massive effort you are providing (and your approach to coverage below is really interesting), getting that wrong at the outset seems a shame. Best regards, James Howison [1] Crowston, K., Wei, K., Howison, J., and Wiggins, A. (2012). Free (libre) open source software development: What we know and what we do not know. ACM Computing Surveys, 44(2): http://floss.syr.edu/content/freelibre-open-source-software-development-wha… On Mar 14, 2011, at 13:58, Chitu Okoli wrote: Hi everyone, We are a research group conducting a systematic literature review on Wikipedia-related peer-reviewed academic studies published in the English language. (Although there are many excellent studies in other languages, we unfortunately do not have the resources to systematically review these at any kind of acceptable scholarly level. Also, our study is about Wikipedia only, not about other Wikimedia Foundation projects. However, we do include studies about other language Wikipedias, as long as the studies are published in English.) We have completed a search using many major databases of scholarly research. In a separate thread, we will also talk about research questions related to our review. Thanks for your help. Chitu Okoli, Concordia University, Montreal, Canada (http://chitu.okoli.org/professional/open-content/wikipedia-and-open-content…) Arto Lanamäki, University of Agder, Kristiansand, Norway Mohamad Mehdi, Concordia University, Montreal, Canada Mostafa Mesgari, Concordia University, Montreal, Canada _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

I am a little sheepish; clearly you've really struggled with this, it's certainly a huge amount of papers. I'm tempted to ask what happens if you cut by publication date, but I suspect that that doesn't help much because of the accelerating rate of publication. In any case not entirely sure of the justification for not including older things, it's not as though one stops knowing them :) Ah, I know: randomize ;) Ok, that's not really in the spirit of a review article. Have you considered cutting by some first quick pass characteristics, such as topic (using some framework relevant to your interests, we used Input-Process-Output for organizing studies of FLOSS)/empirical vs conceptual/perhaps even quant. vs qual. That is, of course, a lot of work just there but it seems to deal with the selection bias the best. That would also help give a conceptual focus to the review article. To avoid the full selection bias of excluding conferences, perhaps you could include only those that are cited in your journal articles? (hmmm, issues there, but perhaps worth thinking about; could one seek out some variant of "the connected set" of articles, with some cutting factor on the strength of linkage to bring the number down to something managable?). Adding people to your review team is another option, I'm sure you've thought about that. Difficulties there are obvious (a good review goes beyond 'tagging' articles and conducts cross-cutting conceptually organized perspective, hard to coordinate or build through disconnected work). Best wishes for the work, James On Mar 15, 2011, at 14:56, Chitu Okoli wrote: > James and Travis, you bring up a point that we have struggled back and forth with for several months. We really, really would like to include conference articles, but we just can't see how we could handle many more articles than what we've got now. We've been working on and off on this project for over two years now. (You can find works in progress at the link at the bottom to my website.) We'd like to get it done eventually, and we can only handle so many articles.

When you consider a "top tier conference", how do you know you are not excluding contributions that might be not just novel but also truly important? It seems that page rank plays the role of beauty contest in the sense that top-ranked pages are those already in the view of others. I have seen comments that this filters against novelty, possibly crucial novelty. Jack On Tue, Mar 15, 2011 at 11:56 AM, Chitu Okoli<Chitu.Okoli(a)concordia.ca> wrote: > We considered including top-tier conferences, but the question is, what is a > "top conference"? In trying to answer this, we looked at a couple of > sources: > * Top Tier and 2nd tier conferences from > http://webdocs.cs.ualberta.ca/~zaiane/htmldocs/ConfRanking.html > * A-ranked conferences in Information and Computing Sciences from > http://lamp.infosys.deakin.edu.au/era/?pageÏorsel10 > * We also considered including all WikiSym articles on Wikipedia

We have now updated the following page with the peer-reviewed journal articles and doctoral theses we have identified: http://en.wikipedia.org/wiki/Wikipedia:Academic_studies_of_Wikipedia. Please note that we have only updated the sections on peer-reviewed journal articles and on theses; we have not updated other sections with newly identified studies, except for correcting some misclassified items.

To help us in identifying all eligible studies, we would really appreciate it if you could look at the sections on peer-reviewed journal articles and theses in http://en.wikipedia.org/wiki/Wikipedia:Academic_studies_of_Wikipedia, and send us any citations (by yourself or others) that you know are missing. In particular, please inform us of:

...

* A-ranked conferences in Information and Computing Sciences from http://lamp.infosys.deakin.edu.au/era/?page=cforsel10: This is the most exhaustive journal ranking exercise I have ever found anywhere.

Unfortunately, I like you have serious questions about the face validity of these rankings; I think they heavily overrate many conferences in my own field of information systems; I assume the same is true with other fields that I don't know so well. (My primary reservation with conference or journal rankings by professors is that I strongly suspect that one of the main criteria for their rankings is whether or not they have published in that outlet before.) Unfortunately, I don't know of anything that approaches this ranking in comprehensiveness.