Hi Michele, lots of people were quite surprised by the outcomes of the review. Will you be sharing the feedback as well? I know easychair has that option. And how definitive are these programmings? I mean, it is quite common that a significant number of accepted people don't make it after all. Lodewijk On Wed, Feb 3, 2016 at 2:00 PM, Michele Lavazza <michele.lavazza(a)gmail.com _______________________________________________ Wikimania-l mailing list Wikimania-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Hi Michele, posting on wiki is fine with me too - but not exactly what i had in mind (that would be a huge amount of copy & paste work). What I rather would suggest, is to use the Easychair built in functionality to share the feedback with the proposers. Please get in touch if you don't know how. Best, Lodewijk On Wed, Feb 3, 2016 at 7:12 PM, Michele Lavazza <michele.lavazza(a)gmail.com _______________________________________________ Wikimania-l mailing list Wikimania-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

What kind of ridiculous process is this? This is all I got: =============== ----------------------- REVIEW 1 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers OVERALL EVALUATION: 8 (Very good) ----------- REVIEW ----------- 8 ----------------------- REVIEW 2 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers OVERALL EVALUATION: 6 (Rather interesting) ----------- REVIEW ----------- 6 ============== So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1]. Maarten [1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5 Op 3-2-2016 om 23:15 schreef Andy Mabbett: I've just received feedback on one of my pitches saying, in part: "Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair." -- Andy Mabbett @pigsonthewing http://pigsonthewing.org.uk _______________________________________________ Wikimania-l mailing listWikimania-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/wikimania-l _______________________________________________ Wikimania-l mailing list Wikimania-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

I've just received feedback on one of my pitches saying, in part: "Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair." -- Andy Mabbett @pigsonthewing http://pigsonthewing.org.uk _______________________________________________ Wikimania-l mailing list Wikimania-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Just to say that I agree with Dariusz's assessment of how academic reviewing is done. That doesn't address the tension of whether Wikimania is an academic conference or not, though - a question that has existed since early days. There's good arguments both ways, and each Wikimania team has approached this question differently.* It should, however, be made clear to submitters what is happening. Phoebe * many teams, and participants, have wanted a more academic conference for various reasons. Many participants find the socializing and practical knowledge sharing the most useful last. In recent years, we've tried to balance this by doing both. If I ever run Wikimania again, I think I'd try having *no* formal talks: only discussions and lightning talks. On Feb 3, 2016 6:23 PM, "Dariusz Jemielniak" <darekj(a)alk.edu.pl> wrote: Wikimania-l mailing list Wikimania-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

lots of people were quite surprised by the outcomes of the review.

hi, I have some comments as a person from Academia (and not involved in Wikimania process in any way): 1. Short reviews are definitely not helping in addressing the frustration of rejection, yet are quite common in academic peer reviewing, especially for conferences. 2. Double blind peer review (not knowing who is reviewed, and not knowing who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions). 3. Two reviewers per submission is absolutely on par with the conference standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much). 4. It could be useful to sensitize the reviewers that the main purpose of the review is to help the author to do better next time. 5. All this is volunteer work. We should be, generally, grateful to reviewers (but in the same time grateful to the contributors, too). best, dj On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers &lt;maarten(a)mdammers.nl&gt; wrote: -- __________________________ prof. dr hab. Dariusz Jemielniak kierownik katedry Zarządzania Międzynarodowego i grupy badawczej NeRDS Akademia Leona Koźmińskiego http://n <http://www.crow.alk.edu.pl/>wrds.kozminski.edu.pl członek Akademii Młodych Uczonych Polskiej Akademii Nauk członek Komitetu Polityki Naukowej MNiSW Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010 Recenzje Forbes: http://www.forbes.com/fdc/welcome_mjx.shtml Pacific Standard: http://www.psmag.com/navigation/books-and-culture/killed-wikipedia-93777/ Motherboard: http://motherboard.vice.com/read/an-ethnography-of-wikipedia The Wikipedian: http://thewikipedian.net/2014/10/10/dariusz-jemielniak-common-knowledge _______________________________________________ Wikimania-l mailing list Wikimania-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Well I think that double blind peer review hardly make sense here, from the reviewer POV, as we are in fact small community and it is easy to guess who was a submiter in most cases. For example - if there is a submission about project X in country Y, which was funded by WMF grant - it is very easy to find out who was grantee and it is rather obvious that that person is a submitter :-) Also judging from the several reviewers comments which I saw already - they did not follow the very vague criteria which was posted here: https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Eval… Normally - at least in Academia - reviewers are forced directly (by the review form) to address their opinion in relation to the criteria. The criteria were: " 1. problems and possible solutions in a specific field 2. proposals for others to replicate 3. issues (positive or negative) which have emerged from projects 4. issues you want to raise which you feel have not been discussed yet 5. issues which are at the centre of an online debate that you would like to address offline " 1-4 are IMHO relatively easy to evaluate - I would expect from the reviews to answer yes or no to them. 5 is a bit tricky as it depends strongly of what the reviewer think is "at the centre" - but I would expect that they at least explain in few words here what they think is "at the centre" or not :-) 2016-02-04 0:22 GMT+01:00 Dariusz Jemielniak &lt;darekj(a)alk.edu.pl&gt;pl>: -- Tomek "Polimerek" Ganicz http://pl.wikimedia.org/wiki/User:Polimerek http://www.ganicz.pl/poli/

Also - normally in Academia - if there are two strongly opposite reviews (one very positive, one very negative) a typical procedure is to send the submission to the third one. 2016-02-04 10:54 GMT+01:00 Tomasz Ganicz &lt;polimerek(a)gmail.com <mailto:polimerek@gmail.com>>: Well I think that double blind peer review hardly make sense here, from the reviewer POV, as we are in fact small community and it is easy to guess who was a submiter in most cases. For example - if there is a submission about project X in country Y, which was funded by WMF grant - it is very easy to find out who was grantee and it is rather obvious that that person is a submitter :-) Also judging from the several reviewers comments which I saw already - they did not follow the very vague criteria which was posted here: https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Eval… Normally - at least in Academia - reviewers are forced directly (by the review form) to address their opinion in relation to the criteria. The criteria were: " 1. problems and possible solutions in a specific field 2. proposals for others to replicate 3. issues (positive or negative) which have emerged from projects 4. issues you want to raise which you feel have not been discussed yet 5. issues which are at the centre of an online debate that you would like to address offline " 1-4 are IMHO relatively easy to evaluate - I would expect from the reviews to answer yes or no to them. 5 is a bit tricky as it depends strongly of what the reviewer think is "at the centre" - but I would expect that they at least explain in few words here what they think is "at the centre" or not :-) 2016-02-04 0:22 GMT+01:00 Dariusz Jemielniak &lt;darekj(a)alk.edu.pl <mailto:darekj@alk.edu.pl>>: hi, I have some comments as a person from Academia (and not involved in Wikimania process in any way): 1. Short reviews are definitely not helping in addressing the frustration of rejection, yet are quite common in academic peer reviewing, especially for conferences. 2. Double blind peer review (not knowing who is reviewed, and not knowing who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions). 3. Two reviewers per submission is absolutely on par with the conference standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much). 4. It could be useful to sensitize the reviewers that the main purpose of the review is to help the author to do better next time. 5. All this is volunteer work. We should be, generally, grateful to reviewers (but in the same time grateful to the contributors, too). best, dj On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers &lt;maarten(a)mdammers.nl <mailto:maarten@mdammers.nl>> wrote: What kind of ridiculous process is this? This is all I got: =============== ----------------------- REVIEW 1 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers OVERALL EVALUATION: 8 (Very good) ----------- REVIEW ----------- 8 ----------------------- REVIEW 2 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers OVERALL EVALUATION: 6 (Rather interesting) ----------- REVIEW ----------- 6 ============== So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1]. Maarten [1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5 Op 3-2-2016 om 23:15 schreef Andy Mabbett: _______________________________________________ Wikimania-l mailing list Wikimania-l(a)lists.wikimedia.org <mailto:Wikimania-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikimania-l -- __________________________ prof. dr hab. Dariusz Jemielniak kierownik katedry Zarządzania Międzynarodowego i grupy badawczej NeRDS Akademia Leona Koźmińskiego http://n <http://www.crow.alk.edu.pl/>wrds.kozminski.edu.pl <http://wrds.kozminski.edu.pl> członek Akademii Młodych Uczonych Polskiej Akademii Nauk członek Komitetu Polityki Naukowej MNiSW Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010 Recenzje Forbes: http://www.forbes.com/fdc/welcome_mjx.shtml Pacific Standard: http://www.psmag.com/navigation/books-and-culture/killed-wikipedia-93777/ Motherboard: http://motherboard.vice.com/read/an-ethnography-of-wikipedia The Wikipedian: http://thewikipedian.net/2014/10/10/dariusz-jemielniak-common-knowledge _______________________________________________ Wikimania-l mailing list Wikimania-l(a)lists.wikimedia.org <mailto:Wikimania-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikimania-l -- Tomek "Polimerek" Ganicz http://pl.wikimedia.org/wiki/User:Polimerek http://www.ganicz.pl/poli/ -- Tomek "Polimerek" Ganicz http://pl.wikimedia.org/wiki/User:Polimerek http://www.ganicz.pl/poli/ _______________________________________________ Wikimania-l mailing list Wikimania-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

But when you get the same presentation being marked as 5 and 8

I got a very similar comment assuming I knew little about Wikidata. As its product manager...

Hope Montreal manages something a bit better,

Following Andrew's comments, here's what I know -- 1 - User Digest is just one session of about 30 minutes. Liaisons pick the speakers to that. It's supposed to give a review of the thematic subject - GLAM, EDU, etc. 2 - I'm posting here what I've sent to the Cultural Partners Mailing List - *​"*I've just updated the GLAM part on the program liaison page on Meta <https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Program/Liaisons#GLAM>, putting there everything from our joint google doc. Now that the "critical issues" submission part is over, *it's high time to submit your suggestions *to all the other aspects of our GLAM track, if you haven't done so thus far. This includes suggestions for: ** Discussions* ** Workshops / Training* ** Posters* ** Lightning talks* * *Anything else we might have forgotten* Some of you have already contacted me privately about *discussions *and *workshops,* so please feel free to update the relevant part on meta. Try to keep it in the same format as suggested below, so it's easier to follow - * Title: * Purpose: * Target audience: * Length: * Max number of people (only if there is a limitation on your part): * Facilitator(s): * any other detail that will help others get a sense of the workshop and what you want to achieve. Please see an example I posted on behalf of Barbara Fischer -- building the GLAM KIT library <https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Program/Liaisons#Building_the_GLAM_KIT_library> .* "* In other words*, use the liaison page for now,* till the organizing team has the separate pages ready. - Follow the format suggest, so it's cohesive and easier to follow. - Show your support to proposals, the wiki-way. The organizing team will take that into consideration. - When the organizing team opens submissions for the remaining parts -- submit! Hope that helps, Shani. On Thu, Feb 4, 2016 at 7:39 PM, Marc A. Pelletier &lt;marc(a)uberbox.org&gt; wrote: _______________________________________________ Wikimania-l mailing list Wikimania-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

@Andrew pleas take a look at https://wikimania2016.wikimedia.org/wiki/Programme for "the proportion, appropriateness or overall relationship of user digest presentations, critical issues or discussions". @Marc-André thank you for your remarks, this is just to say that in the end there was no need to implement the so-called "WMF quota" because less than 10 WMF staff submission were among the top 40. @Those who have doubts about the consistency of our implementation of a third evaluation in those cases where the first two were strongly divergent: this was a technically arduous thing, since clearly assigning a third review presupposes having two already. Having a deadline to respect, we tried to make sure to have the first two reviews for each submission a few days before the deadline (which proved impossible because of personal difficulties of some members of our international reviewing team). We would then add the third reviews when needed. This still was a heavy bulk of extra work, so we decided not to request the extra review for those submissions whose average score after the first two reviews was too low to make it possible for a third review to make the relevant submission competitive for acceptance. (For example: if the top 50 submissions, with 2 reviews or 3 where needed, already had an average score higher than 7.5, we didn't assign a third reviewer to a submission whose first two scores were 3 and 8, because, even if the third review were a 10, the average would still be lower than that 7.5). I hope I answered a few of the most urgent questions. Thank you all for expressing you perplexities, criticism, or questions. I realise some aspects of the double blind peer review process have been suboptimal and we're trying to make sure both that the conference sessions will be great nevertheless, and that our experience is useful for the future Wikimanias. Thank you very much, yours, Michele On Thu, Feb 4, 2016 at 11:38 PM, Lodewijk &lt;lodewijk(a)effeietsanders.org&gt; wrote: _______________________________________________ Wikimania-l mailing list Wikimania-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

One thing that would help make sense of where we are now: remind us how the overall structure of the conference is going to include not just the 42 “critical issues” sessions picked out of EasyChair, but also ones via other processes. Make it clear, repeat it constantly, and give links to people to understand it.

- “User digest presentations" - When the page says “Contact the Thematic Liaison,” the user is almost always at least two clicks away from finding a way to contact that person. Even worse, for many users, clicking on their name sends you to a confusing page: “https://wikimania2016.wikimedia.org/wiki/Special:CentralAuth/Username“ What is the average user supposed to do with that?

- “Discussions” page is a red link. There is not even a simple description of what this is. Same thing with “Community Village”

- Even a brief paragraph or a diagram showing the 10,000 foot/3048 meter view of the overall plan would be welcome on the submissions page. Right now there is no prose, only six big categories. This requires a lot of haphazard clicking and piecing together of the conference narrative.