That seems a sensible system, I can see how a 6 and a 7 were enough to reject one of my proposals, though I'd have appreciated a bit more feedback as to why.
But how strong did the divergence have to be if 5 and 8 didn't count as a "strong divergence"?
WereSpielChequers
On 4 Feb 2016, at 11:55, Sebastian Wallroth sebastian@wallroth.de wrote:
Hi Tomasz,
this is what actually happend. Please refer to https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evalu...
: In order to achieve the greatest possible neutrality, the submissions will : be evaluated online using a *double-blind peer-review process*. This : means that two evaluators will review the submission without knowing : the name of its author. *If there are strong divergences among the two : evaluations, at least one other review will be made.*
Kind regards, Sebastian
Am 04.02.2016 um 11:31 schrieb Tomasz Ganicz: Also - normally in Academia - if there are two strongly opposite reviews (one very positive, one very negative) a typical procedure is to send the submission to the third one.
2016-02-04 10:54 GMT+01:00 Tomasz Ganicz <polimerek@gmail.com mailto:polimerek@gmail.com>:
Well I think that double blind peer review hardly make sense here, from the reviewer POV, as we are in fact small community and it is easy to guess who was a submiter in most cases. For example - if there is a submission about project X in country Y, which was funded by WMF grant - it is very easy to find out who was grantee and it is rather obvious that that person is a submitter :-)
Also judging from the several reviewers comments which I saw already - they did not follow the very vague criteria which was posted here:
https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evalu...
Normally - at least in Academia - reviewers are forced directly (by the review form) to address their opinion in relation to the criteria. The criteria were:
"
1. problems and possible solutions in a specific field 2. proposals for others to replicate 3. issues (positive or negative) which have emerged from projects 4. issues you want to raise which you feel have not been discussed yet 5. issues which are at the centre of an online debate that you would like to address offline"
1-4 are IMHO relatively easy to evaluate - I would expect from the reviews to answer yes or no to them. 5 is a bit tricky as it depends strongly of what the reviewer think is "at the centre" - but I would expect that they at least explain in few words here what they think is "at the centre" or not :-)
2016-02-04 0:22 GMT+01:00 Dariusz Jemielniak <darekj@alk.edu.pl mailto:darekj@alk.edu.pl>:
hi, I have some comments as a person from Academia (and not involved in Wikimania process in any way): 1. Short reviews are definitely not helping in addressing the frustration of rejection, yet are quite common in academic peer reviewing, especially for conferences. 2. Double blind peer review (not knowing who is reviewed, and not knowing who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions). 3. Two reviewers per submission is absolutely on par with the conference standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much). 4. It could be useful to sensitize the reviewers that the main purpose of the review is to help the author to do better next time. 5. All this is volunteer work. We should be, generally, grateful to reviewers (but in the same time grateful to the contributors, too). best, dj On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers <maarten@mdammers.nl <mailto:maarten@mdammers.nl>> wrote: What kind of ridiculous process is this? This is all I got: =============== ----------------------- REVIEW 1 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers OVERALL EVALUATION: 8 (Very good) ----------- REVIEW ----------- 8 ----------------------- REVIEW 2 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers OVERALL EVALUATION: 6 (Rather interesting) ----------- REVIEW ----------- 6 ============== So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1]. Maarten [1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5 Op 3-2-2016 om 23:15 schreef Andy Mabbett:I've just received feedback on one of my pitches saying, in part: "Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair." -- Andy Mabbett @pigsonthewing http://pigsonthewing.org.uk _______________________________________________ Wikimania-l mailing list Wikimania-l@lists.wikimedia.org <mailto:Wikimania-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikimania-l_______________________________________________ Wikimania-l mailing list Wikimania-l@lists.wikimedia.org <mailto:Wikimania-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikimania-l -- __________________________ prof. dr hab. Dariusz Jemielniak kierownik katedry Zarządzania Międzynarodowego i grupy badawczej NeRDS Akademia Leona Koźmińskiego http://n <http://www.crow.alk.edu.pl/>wrds.kozminski.edu.pl <http://wrds.kozminski.edu.pl> członek Akademii Młodych Uczonych Polskiej Akademii Nauk członek Komitetu Polityki Naukowej MNiSW Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010 Recenzje Forbes: http://www.forbes.com/fdc/welcome_mjx.shtml Pacific Standard: http://www.psmag.com/navigation/books-and-culture/killed-wikipedia-93777/ Motherboard: http://motherboard.vice.com/read/an-ethnography-of-wikipedia The Wikipedian: http://thewikipedian.net/2014/10/10/dariusz-jemielniak-common-knowledge _______________________________________________ Wikimania-l mailing list Wikimania-l@lists.wikimedia.org <mailto:Wikimania-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikimania-l-- Tomek "Polimerek" Ganicz http://pl.wikimedia.org/wiki/User:Polimerek http://www.ganicz.pl/poli/
-- Tomek "Polimerek" Ganicz http://pl.wikimedia.org/wiki/User:Polimerek http://www.ganicz.pl/poli/
Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l
-- Sebastian Wallroth mobile +4917615154002 http://about.me/real68er PGP Key https://de.wikipedia.org/wiki/Benutzer:Sebastian_Wallroth/PGP
Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l