Sure, we will upload the comments and feedback of pc members to the Wikimania 2016 wiki too.

We already planned to keep a margin of around 20% considering that some authors won't make it. We have 42 papers accepted and 8 "accepted as reserve". If necessary, we will increase this reserve.

Yours,

Michele

On Wed, Feb 3, 2016 at 3:49 PM, Lodewijk lodewijk@effeietsanders.org wrote:

Hi Michele,

lots of people were quite surprised by the outcomes of the review. Will you be sharing the feedback as well? I know easychair has that option.

And how definitive are these programmings? I mean, it is quite common that a significant number of accepted people don't make it after all.

Lodewijk

On Wed, Feb 3, 2016 at 2:00 PM, Michele Lavazza <michele.lavazza@gmail.com

...

wrote:

...

Dear all,

this is just to inform you that the notifications of acceptance and of rejection for the Wikimania 2016 "Critical issues" talks were issued yesterday.

In the next few days, we will work to upload the submissions and the evalutations to the Wikimania 2016 wiki.

Thank you very much, best regards,

Michele Lavazza (Wikimania 2016 Programme Committee Deputy Chair)

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Sure, you're right. I will send the feedback to the authors immediately. That is indeed helpful for their understanding of the reasons for acceptance and, most importantly, rejection. I'm sorry I didn't do it before.

For the sake of transparency, however, the submissions and results will be published on the wiki too, even if you are right: that's quite an amount of work I'll share with Marco Chemello, so please be patient :)

Thank you very much for your help, yours,

Michele

On Wed, Feb 3, 2016 at 7:27 PM, Lodewijk lodewijk@effeietsanders.org wrote:

Hi Michele,

posting on wiki is fine with me too - but not exactly what i had in mind (that would be a huge amount of copy & paste work). What I rather would suggest, is to use the Easychair built in functionality to share the feedback with the proposers. Please get in touch if you don't know how.

Best, Lodewijk

On Wed, Feb 3, 2016 at 7:12 PM, Michele Lavazza <michele.lavazza@gmail.com

...

wrote:

...

Sure, we will upload the comments and feedback of pc members to the Wikimania 2016 wiki too.

We already planned to keep a margin of around 20% considering that some authors won't make it. We have 42 papers accepted and 8 "accepted as reserve". If necessary, we will increase this reserve.

Yours,

Michele

On Wed, Feb 3, 2016 at 3:49 PM, Lodewijk lodewijk@effeietsanders.org wrote:

...

Hi Michele,

lots of people were quite surprised by the outcomes of the review. Will you be sharing the feedback as well? I know easychair has that option.

And how definitive are these programmings? I mean, it is quite common that a significant number of accepted people don't make it after all.

Lodewijk

On Wed, Feb 3, 2016 at 2:00 PM, Michele Lavazza < michele.lavazza@gmail.com> wrote:

...

Dear all,

this is just to inform you that the notifications of acceptance and of rejection for the Wikimania 2016 "Critical issues" talks were issued yesterday.

In the next few days, we will work to upload the submissions and the evalutations to the Wikimania 2016 wiki.

Thank you very much, best regards,

Michele Lavazza (Wikimania 2016 Programme Committee Deputy Chair)

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Here's another rejection-review, in full:

"Someone knows what wikispecies is about, and is worried about its future. Sounds like complaining to keep a project. No arguments were given. Topic to be discussed on mail lists."

Dear Maarten,

I'm sorry that the process looks ridiculous to you. About the fact that only two people evaluated your submission, please take a look at https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evalu... . I can only add that this year PC members, instead of evaluating all papers, had to bid for those they wanted to review. Therefore you have less reviews, but they are more qualified because your submission was chosen by someone who found it fell in his/her area of interest or of expertise.

About the fact that in your case, unfortunately, there are no verbal motivations for the numeric score, that depends on the fact that PC members were not obliged to give such motivations. Some did nevertheless.

About the fact that you've been sent the reviews without the names of the reviewers associated to them, I must blame the limitations of the EasyChair sofware, which from some points of view has been infernal to deal with. I'm sorry, as soon as we upload the submissions to the wiki you will see this information, too.

I hope I answered your questions. Best regards,

Michele

On Wed, Feb 3, 2016 at 11:26 PM, Maarten Dammers maarten@mdammers.nl wrote:

What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW ----------- 8

----------------------- REVIEW 2 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW ----------- 6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

-- Andy Mabbett @pigsonthewing http://pigsonthewing.org.uk

---

Wikimania-l mailing listWikimania-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Just to say that I agree with Dariusz's assessment of how academic reviewing is done. That doesn't address the tension of whether Wikimania is an academic conference or not, though - a question that has existed since early days. There's good arguments both ways, and each Wikimania team has approached this question differently.* It should, however, be made clear to submitters what is happening.

Phoebe

* many teams, and participants, have wanted a more academic conference for various reasons. Many participants find the socializing and practical knowledge sharing the most useful last. In recent years, we've tried to balance this by doing both. If I ever run Wikimania again, I think I'd try having *no* formal talks: only discussions and lightning talks.

On Feb 3, 2016 6:23 PM, "Dariusz Jemielniak" darekj@alk.edu.pl wrote:

hi,

I have some comments as a person from Academia (and not involved in Wikimania process in any way):

1. Short reviews are definitely not helping in addressing the frustration

of rejection, yet are quite common in academic peer reviewing, especially for conferences.

2. Double blind peer review (not knowing who is reviewed, and not knowing

who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions).

3. Two reviewers per submission is absolutely on par with the conference

standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much).

4. It could be useful to sensitize the reviewers that the main purpose of

the review is to help the author to do better next time.

5. All this is volunteer work. We should be, generally, grateful to

reviewers (but in the same time grateful to the contributors, too).

best,

dj

On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers maarten@mdammers.nl wrote:

...

What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW ----------- 8

----------------------- REVIEW 2 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW ----------- 6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

-- Andy Mabbett @pigsonthewing http://pigsonthewing.org.uk

---

Wikimania-l mailing listWikimania-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

--

---

prof. dr hab. Dariusz Jemielniak kierownik katedry Zarządzania Międzynarodowego i grupy badawczej NeRDS Akademia Leona Koźmińskiego http://n http://www.crow.alk.edu.pl/wrds.kozminski.edu.pl

członek Akademii Młodych Uczonych Polskiej Akademii Nauk członek Komitetu Polityki Naukowej MNiSW

Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010

Recenzje Forbes: http://www.forbes.com/fdc/welcome_mjx.shtml Pacific Standard: http://www.psmag.com/navigation/books-and-culture/killed-wikipedia-93777/ Motherboard: http://motherboard.vice.com/read/an-ethnography-of-wikipedia The Wikipedian: http://thewikipedian.net/2014/10/10/dariusz-jemielniak-common-knowledge

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Two reviews per submission might work if we had a clear set of criteria that the reviewers were following and sufficient training of the reviewers that they were broadly consistent in their marking. But when you get the same presentation being marked as 5 and 8, as one of mine was then the suspicion is that the assessors are not working to the same criteria as each other. That wouldn't matter so much if they were all assessing all submissions, except that an assessor who varied between 0 and ten points would have far more influence than assessors who usually voted 6, 7 or 8. But having that level of inconsistency and only two reviews per submission makes the process a lottery that depends on who the two reviewers are for your submission.

As for the content of the reviews, I don't consider that either "5 (average)" or "6 (rather interesting) tell me anything as to why my submissions were rejected.

The other two reviews at least managed one or two lines. One of them even stretched to two sentences.

Hope Montreal manages something a bit better, I'm sure either Manilla or Perth would have done.

WereSpielChequers

On 3 Feb 2016, at 23:22, Dariusz Jemielniak darekj@alk.edu.pl wrote:

hi,

I have some comments as a person from Academia (and not involved in Wikimania process in any way):

1. Short reviews are definitely not helping in addressing the frustration of rejection, yet are quite common in academic peer reviewing, especially for conferences.

2. Double blind peer review (not knowing who is reviewed, and not knowing who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions).

3. Two reviewers per submission is absolutely on par with the conference standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much).

4. It could be useful to sensitize the reviewers that the main purpose of the review is to help the author to do better next time.

5. All this is volunteer work. We should be, generally, grateful to reviewers (but in the same time grateful to the contributors, too).

best,

dj

...

On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers maarten@mdammers.nl wrote: What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW ----------- 8

----------------------- REVIEW 2 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW ----------- 6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

...

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

-- Andy Mabbett @pigsonthewing http://pigsonthewing.org.uk

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

--

---

prof. dr hab. Dariusz Jemielniak kierownik katedry Zarządzania Międzynarodowego i grupy badawczej NeRDS Akademia Leona Koźmińskiego http://nwrds.kozminski.edu.pl

członek Akademii Młodych Uczonych Polskiej Akademii Nauk członek Komitetu Polityki Naukowej MNiSW

Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010

Recenzje Forbes: http://www.forbes.com/fdc/welcome_mjx.shtml Pacific Standard: http://www.psmag.com/navigation/books-and-culture/killed-wikipedia-93777/ Motherboard: http://motherboard.vice.com/read/an-ethnography-of-wikipedia The Wikipedian: http://thewikipedian.net/2014/10/10/dariusz-jemielniak-common-knowledge _______________________________________________ Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

On 2016-02-04 3:22 AM, WereSpielChequers wrote:

But when you get the same presentation being marked as 5 and 8

I got a 4 and a 8; the former noting in the comment that it was not clear that I knew the subject. Subject being external tools to the project. Like found on Tool Labs.

I'm not sure that review process was entirely successful.

-- Marc

On 4 February 2016 at 14:50, Lydia Pintscher Lydia.Pintscher@wikimedia.de wrote:

I got a very similar comment assuming I knew little about Wikidata. As its product manager...

Anyone who has been burdened with doing these sorts of reviews will feel some sympathy for those giving the feedback. It is easy to upset a lot of people if the process is not well thought out. Where there are marking discrepancies, the workflow should mean it goes to another independent reviewer and there is a meeting (like 2 minutes in a Hangout discussion) where there is final agreement on the rating/mark *and* the feedback that should be given.

Even without discrepancies in marks, feedback needs to be positive and supportive, this is all volunteers giving their time after all, not postgrads getting critical essay feedback. That means the workflow also needs to include regular checks and team meetings to talk about how to best ensure marks and feedback remains consistent, even when the experience and viewpoints of the reviewers may be highly varied.

Lots of lessons to be summarized for later, and probably a need to consider whether now is a good time put up your hands and formally admit to problems in consistency. Asking submitters to give their feedback and suggestions on-wiki, even if is too late to change any decision, was a good response.

Fae

Andrew Lih, 04/02/2016 16:57:

One thing that would help make sense of where we are now: remind us how the overall structure of the conference is going to include not just the 42 “critical issues” sessions picked out of EasyChair, but also ones via other processes. Make it clear, repeat it constantly, and give links to people to understand it.

+1 For the rest, huge +1 on Shani's "wiki way" paragraph.

• “User digest presentations" - When the page says “Contact the Thematic

Liaison,” the user is almost always at least two clicks away from finding a way to contact that person. Even worse, for many users, clicking on their name sends you to a confusing page: “https://wikimania2016.wikimedia.org/wiki/Special:CentralAuth/Username%E2%80%... What is the average user supposed to do with that?

The page can be edited. :-) Information on most people listed there is public here or there, only needs some editing help to be presented at best.

I added the user accounts link for the case where no link is provided, thinking that the "average user" of a Wikimania wiki knows how to use talk pages. How does your "average user" look like?

• “Discussions” page is a red link. There is not even a simple

description of what this is. Same thing with “Community Village”

These are repeat models, so any former Wikimania attendee can help fill in. :-)

• Even a brief paragraph or a diagram showing the 10,000 foot/3048 meter

view of the overall plan would be welcome on the submissions page. Right now there is no prose, only six big categories. This requires a lot of haphazard clicking and piecing together of the conference narrative.

The "general narrative" is https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Pillars , AFAIK.

Nemo

Following Andrew's comments, here's what I know --

1 - User Digest is just one session of about 30 minutes. Liaisons pick the speakers to that. It's supposed to give a review of the thematic subject - GLAM, EDU, etc.

2 - I'm posting here what I've sent to the Cultural Partners Mailing List -

*​"*I've just updated the GLAM part on the program liaison page on Meta https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Program/Liaisons#GLAM, putting there everything from our joint google doc.

Now that the "critical issues" submission part is over, *it's high time to submit your suggestions *to all the other aspects of our GLAM track, if you haven't done so thus far. This includes suggestions for: ** Discussions* ** Workshops / Training* ** Posters* ** Lightning talks* * *Anything else we might have forgotten*

Some of you have already contacted me privately about *discussions *and *workshops,* so please feel free to update the relevant part on meta. Try to keep it in the same format as suggested below, so it's easier to follow - * Title: * Purpose: * Target audience: * Length: * Max number of people (only if there is a limitation on your part): * Facilitator(s): * any other detail that will help others get a sense of the workshop and what you want to achieve. Please see an example I posted on behalf of Barbara Fischer -- building the GLAM KIT library https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Program/Liaisons#Building_the_GLAM_KIT_library .* "*

In other words*, use the liaison page for now,* till the organizing team has the separate pages ready. - Follow the format suggest, so it's cohesive and easier to follow. - Show your support to proposals, the wiki-way. The organizing team will take that into consideration. - When the organizing team opens submissions for the remaining parts -- submit!

Hope that helps, Shani.

On Thu, Feb 4, 2016 at 7:39 PM, Marc A. Pelletier marc@uberbox.org wrote:

On 2016-02-04 3:22 AM, WereSpielChequers wrote:

...

Hope Montreal manages something a bit better,

I don't know about "better", nor do I think it quite fair to slam the 2016 team either for what was clearly intended to be an attempt to improve the process - even if some of the results appear suboptimal in retrospect.

FWIW, the Montreal team is keeping a close eye on the experiments being done by the Italian team - no doubt there will be a valuable set of lessons learned and we may be able to translate some of the things that worked well into improvements to future Wikimanias.

As for the programme selection, we are gunning for a process that splits about 30% invited, 40% community CFP, and 30% unconference-style, with the selection process for the CFP being very close to past years (i.e.: public review on-wiki). We also don't intend to make a distinction between submissions by Foundation staff and the other community members, though we expect that many presentations that would have been proposals by staff will end up being invited directly by the programming committee leaving more "slots" available to the CFP.

-- Marc

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

@Andrew pleas take a look at https://wikimania2016.wikimedia.org/wiki/Programme for "the proportion, appropriateness or overall relationship of user digest presentations, critical issues or discussions".

@Marc-André thank you for your remarks, this is just to say that in the end there was no need to implement the so-called "WMF quota" because less than 10 WMF staff submission were among the top 40.

@Those who have doubts about the consistency of our implementation of a third evaluation in those cases where the first two were strongly divergent: this was a technically arduous thing, since clearly assigning a third review presupposes having two already. Having a deadline to respect, we tried to make sure to have the first two reviews for each submission a few days before the deadline (which proved impossible because of personal difficulties of some members of our international reviewing team). We would then add the third reviews when needed. This still was a heavy bulk of extra work, so we decided not to request the extra review for those submissions whose average score after the first two reviews was too low to make it possible for a third review to make the relevant submission competitive for acceptance. (For example: if the top 50 submissions, with 2 reviews or 3 where needed, already had an average score higher than 7.5, we didn't assign a third reviewer to a submission whose first two scores were 3 and 8, because, even if the third review were a 10, the average would still be lower than that 7.5).

I hope I answered a few of the most urgent questions. Thank you all for expressing you perplexities, criticism, or questions. I realise some aspects of the double blind peer review process have been suboptimal and we're trying to make sure both that the conference sessions will be great nevertheless, and that our experience is useful for the future Wikimanias.

Thank you very much, yours,

Michele

On Thu, Feb 4, 2016 at 11:38 PM, Lodewijk lodewijk@effeietsanders.org wrote:

Hi all,

quick update about the discussions track: I'm coordinating that together with a few others, and this will probably be a bit later in time that most of the other sessions. This is a continuation of the Discussion Room of 2014/2015 https://wikimania2015.wikimedia.org/wiki/Discussion_Room, and focuses on roundtabe discussions of 40-45 minutes each on a specific topic. That is mostly because there's less preparation required for those discussions, so we love to have them decided a bit later and have them be more 'hot and current'. We hope to open improve the descriptions of that soon though, and open suggestions for that too.

We're currently planning to formally open discussion suggestions on Feb 25 or a bit before. But in the mean time, if you're afraid to forget to submit it by then, feel free to leave behind ideas here: https://wikimania2016.wikimedia.org/w/index.php?title=Talk:Discussions&a... (Shani's list of points sounds good in general, although proposals with less information are also welcome for roundtable discussions)

Best, Lodewijk

On Thu, Feb 4, 2016 at 9:32 PM, Shani shani.even@gmail.com wrote:

...

Following Andrew's comments, here's what I know --

1 - User Digest is just one session of about 30 minutes. Liaisons pick the speakers to that. It's supposed to give a review of the thematic subject - GLAM, EDU, etc.

2 - I'm posting here what I've sent to the Cultural Partners Mailing List

*​"*I've just updated the GLAM part on the program liaison page on Meta https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Program/Liaisons#GLAM, putting there everything from our joint google doc.

Now that the "critical issues" submission part is over, *it's high time to submit your suggestions *to all the other aspects of our GLAM track, if you haven't done so thus far. This includes suggestions for: ** Discussions* ** Workshops / Training* ** Posters* ** Lightning talks*

• *Anything else we might have forgotten*

Some of you have already contacted me privately about *discussions *and *workshops,* so please feel free to update the relevant part on meta. Try to keep it in the same format as suggested below, so it's easier to follow -

• Title:
• Purpose:
• Target audience:
• Length:
• Max number of people (only if there is a limitation on your part):
• Facilitator(s):
• any other detail that will help others get a sense of the workshop and

what you want to achieve. Please see an example I posted on behalf of Barbara Fischer -- building the GLAM KIT library https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Program/Liaisons#Building_the_GLAM_KIT_library .* "*

In other words*, use the liaison page for now,* till the organizing team has the separate pages ready.

• Follow the format suggest, so it's cohesive and easier to follow.
• Show your support to proposals, the wiki-way. The organizing team will

take that into consideration.

• When the organizing team opens submissions for the remaining parts --

submit!

Hope that helps, Shani.

On Thu, Feb 4, 2016 at 7:39 PM, Marc A. Pelletier marc@uberbox.org wrote:

...

On 2016-02-04 3:22 AM, WereSpielChequers wrote:

...

Hope Montreal manages something a bit better,

I don't know about "better", nor do I think it quite fair to slam the 2016 team either for what was clearly intended to be an attempt to improve the process - even if some of the results appear suboptimal in retrospect.

FWIW, the Montreal team is keeping a close eye on the experiments being done by the Italian team - no doubt there will be a valuable set of lessons learned and we may be able to translate some of the things that worked well into improvements to future Wikimanias.

As for the programme selection, we are gunning for a process that splits about 30% invited, 40% community CFP, and 30% unconference-style, with the selection process for the CFP being very close to past years (i.e.: public review on-wiki). We also don't intend to make a distinction between submissions by Foundation staff and the other community members, though we expect that many presentations that would have been proposals by staff will end up being invited directly by the programming committee leaving more "slots" available to the CFP.

-- Marc

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

That would indeed be better, but it would require a huge amount of extra work for PC chairs and a much more sophisticated software than EasyChair...

On Fri, Feb 5, 2016 at 11:32 AM, Gnangarra gnangarra@gmail.com wrote:

Wouldnt it be better to have redacted the anomaly and aggregated to two remaining scores rather then three scores because other the review is meaningless as it the anomaly in place and therefore create the wasted effort, where an 8 and 3 occurs if the third score is a 10 then the 3 is the anomaly and the average should be 9 where the third score is 4 then the 8 is the anomaly and the average is 3.5

On 5 February 2016 at 18:19, Michele Lavazza michele.lavazza@gmail.com wrote:

...

@Andrew pleas take a look at https://wikimania2016.wikimedia.org/wiki/Programme for "the proportion, appropriateness or overall relationship of user digest presentations, critical issues or discussions".

@Marc-André thank you for your remarks, this is just to say that in the end there was no need to implement the so-called "WMF quota" because less than 10 WMF staff submission were among the top 40.

@Those who have doubts about the consistency of our implementation of a third evaluation in those cases where the first two were strongly divergent: this was a technically arduous thing, since clearly assigning a third review presupposes having two already. Having a deadline to respect, we tried to make sure to have the first two reviews for each submission a few days before the deadline (which proved impossible because of personal difficulties of some members of our international reviewing team). We would then add the third reviews when needed. This still was a heavy bulk of extra work, so we decided not to request the extra review for those submissions whose average score after the first two reviews was too low to make it possible for a third review to make the relevant submission competitive for acceptance. (For example: if the top 50 submissions, with 2 reviews or 3 where needed, already had an average score higher than 7.5, we didn't assign a third reviewer to a submission whose first two scores were 3 and 8, because, even if the third review were a 10, the average would still be lower than that 7.5).

I hope I answered a few of the most urgent questions. Thank you all for expressing you perplexities, criticism, or questions. I realise some aspects of the double blind peer review process have been suboptimal and we're trying to make sure both that the conference sessions will be great nevertheless, and that our experience is useful for the future Wikimanias.

Thank you very much, yours,

Michele

On Thu, Feb 4, 2016 at 11:38 PM, Lodewijk lodewijk@effeietsanders.org wrote:

...

Hi all,

quick update about the discussions track: I'm coordinating that together with a few others, and this will probably be a bit later in time that most of the other sessions. This is a continuation of the Discussion Room of 2014/2015 https://wikimania2015.wikimedia.org/wiki/Discussion_Room, and focuses on roundtabe discussions of 40-45 minutes each on a specific topic. That is mostly because there's less preparation required for those discussions, so we love to have them decided a bit later and have them be more 'hot and current'. We hope to open improve the descriptions of that soon though, and open suggestions for that too.

We're currently planning to formally open discussion suggestions on Feb 25 or a bit before. But in the mean time, if you're afraid to forget to submit it by then, feel free to leave behind ideas here: https://wikimania2016.wikimedia.org/w/index.php?title=Talk:Discussions&a... (Shani's list of points sounds good in general, although proposals with less information are also welcome for roundtable discussions)

Best, Lodewijk

On Thu, Feb 4, 2016 at 9:32 PM, Shani shani.even@gmail.com wrote:

...

Following Andrew's comments, here's what I know --

1 - User Digest is just one session of about 30 minutes. Liaisons pick the speakers to that. It's supposed to give a review of the thematic subject - GLAM, EDU, etc.

2 - I'm posting here what I've sent to the Cultural Partners Mailing List -

*​"*I've just updated the GLAM part on the program liaison page on Meta https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Program/Liaisons#GLAM, putting there everything from our joint google doc.

Now that the "critical issues" submission part is over, *it's high time to submit your suggestions *to all the other aspects of our GLAM track, if you haven't done so thus far. This includes suggestions for: ** Discussions* ** Workshops / Training* ** Posters* ** Lightning talks*

• *Anything else we might have forgotten*

Some of you have already contacted me privately about *discussions * and *workshops,* so please feel free to update the relevant part on meta. Try to keep it in the same format as suggested below, so it's easier to follow -

• Title:
• Purpose:
• Target audience:
• Length:
• Max number of people (only if there is a limitation on your part):
• Facilitator(s):
• any other detail that will help others get a sense of the workshop

and what you want to achieve. Please see an example I posted on behalf of Barbara Fischer -- building the GLAM KIT library https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Program/Liaisons#Building_the_GLAM_KIT_library .* "*

In other words*, use the liaison page for now,* till the organizing team has the separate pages ready.

• Follow the format suggest, so it's cohesive and easier to follow.
• Show your support to proposals, the wiki-way. The organizing team

will take that into consideration.

• When the organizing team opens submissions for the remaining parts --

submit!

Hope that helps, Shani.

On Thu, Feb 4, 2016 at 7:39 PM, Marc A. Pelletier marc@uberbox.org wrote:

...

On 2016-02-04 3:22 AM, WereSpielChequers wrote:

...

Hope Montreal manages something a bit better,

I don't know about "better", nor do I think it quite fair to slam the 2016 team either for what was clearly intended to be an attempt to improve the process - even if some of the results appear suboptimal in retrospect.

FWIW, the Montreal team is keeping a close eye on the experiments being done by the Italian team - no doubt there will be a valuable set of lessons learned and we may be able to translate some of the things that worked well into improvements to future Wikimanias.

As for the programme selection, we are gunning for a process that splits about 30% invited, 40% community CFP, and 30% unconference-style, with the selection process for the CFP being very close to past years (i.e.: public review on-wiki). We also don't intend to make a distinction between submissions by Foundation staff and the other community members, though we expect that many presentations that would have been proposals by staff will end up being invited directly by the programming committee leaving more "slots" available to the CFP.

-- Marc

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

-- GN. President Wikimedia Australia WMAU: http://www.wikimedia.org.au/wiki/User:Gnangarra Photo Gallery: http://gnangarra.redbubble.com

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Tomasz Ganicz, 04/02/2016 10:54:

1-4 are IMHO relatively easy to evaluate - I would expect from the reviews to answer yes or no to them. 5 is a bit tricky as it depends strongly of what the reviewer think is "at the centre" - but I would expect that they at least explain in few words here what they think is "at the centre" or not :-)

Yet, reviews like the one Andy mentioned clearly address this: his review was a clear "no" for point 5 as it pointed to (the absence of) a mailing list discussion. (I can't check whether that's true.)

Nemo

Also - normally in Academia - if there are two strongly opposite reviews (one very positive, one very negative) a typical procedure is to send the submission to the third one.

2016-02-04 10:54 GMT+01:00 Tomasz Ganicz polimerek@gmail.com:

Well I think that double blind peer review hardly make sense here, from the reviewer POV, as we are in fact small community and it is easy to guess who was a submiter in most cases. For example - if there is a submission about project X in country Y, which was funded by WMF grant - it is very easy to find out who was grantee and it is rather obvious that that person is a submitter :-)

Also judging from the several reviewers comments which I saw already - they did not follow the very vague criteria which was posted here:

https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evalu...

Normally - at least in Academia - reviewers are forced directly (by the review form) to address their opinion in relation to the criteria. The criteria were:

"

1. problems and possible solutions in a specific field
2. proposals for others to replicate
3. issues (positive or negative) which have emerged from projects
4. issues you want to raise which you feel have not been discussed yet
5. issues which are at the centre of an online debate that you would

like to address offline

"

1-4 are IMHO relatively easy to evaluate - I would expect from the reviews to answer yes or no to them. 5 is a bit tricky as it depends strongly of what the reviewer think is "at the centre" - but I would expect that they at least explain in few words here what they think is "at the centre" or not :-)

2016-02-04 0:22 GMT+01:00 Dariusz Jemielniak darekj@alk.edu.pl:

...

hi,

I have some comments as a person from Academia (and not involved in Wikimania process in any way):

1. Short reviews are definitely not helping in addressing the frustration

of rejection, yet are quite common in academic peer reviewing, especially for conferences.

2. Double blind peer review (not knowing who is reviewed, and not knowing

who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions).

3. Two reviewers per submission is absolutely on par with the conference

standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much).

4. It could be useful to sensitize the reviewers that the main purpose of

the review is to help the author to do better next time.

5. All this is volunteer work. We should be, generally, grateful to

reviewers (but in the same time grateful to the contributors, too).

best,

dj

On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers maarten@mdammers.nl wrote:

...

What kind of ridiculous process is this? This is all I got:

===============

----------------------- REVIEW 1 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 8 (Very good)

----------- REVIEW ----------- 8

----------------------- REVIEW 2 --------------------- PAPER: 194 TITLE: GLAM+Wikidata AUTHORS: Sandra Fauconnier and Maarten Dammers

OVERALL EVALUATION: 6 (Rather interesting)

----------- REVIEW ----------- 6

==============

So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

Maarten

[1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5

Op 3-2-2016 om 23:15 schreef Andy Mabbett:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

-- Andy Mabbett @pigsonthewing http://pigsonthewing.org.uk

---

Wikimania-l mailing listWikimania-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/wikimania-l

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

--

---

prof. dr hab. Dariusz Jemielniak kierownik katedry Zarządzania Międzynarodowego i grupy badawczej NeRDS Akademia Leona Koźmińskiego http://n http://www.crow.alk.edu.pl/wrds.kozminski.edu.pl

członek Akademii Młodych Uczonych Polskiej Akademii Nauk członek Komitetu Polityki Naukowej MNiSW

Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa http://www.sup.org/book.cgi?id=24010

Recenzje Forbes: http://www.forbes.com/fdc/welcome_mjx.shtml Pacific Standard: http://www.psmag.com/navigation/books-and-culture/killed-wikipedia-93777/ Motherboard: http://motherboard.vice.com/read/an-ethnography-of-wikipedia The Wikipedian: http://thewikipedian.net/2014/10/10/dariusz-jemielniak-common-knowledge

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

-- Tomek "Polimerek" Ganicz http://pl.wikimedia.org/wiki/User:Polimerek http://www.ganicz.pl/poli/

That seems a sensible system, I can see how a 6 and a 7 were enough to reject one of my proposals, though I'd have appreciated a bit more feedback as to why.

But how strong did the divergence have to be if 5 and 8 didn't count as a "strong divergence"?

WereSpielChequers

On 4 Feb 2016, at 11:55, Sebastian Wallroth sebastian@wallroth.de wrote:

Hi Tomasz,

this is what actually happend. Please refer to https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evalu...

: In order to achieve the greatest possible neutrality, the submissions will : be evaluated online using a *double-blind peer-review process*. This : means that two evaluators will review the submission without knowing : the name of its author. *If there are strong divergences among the two : evaluations, at least one other review will be made.*

Kind regards, Sebastian

...

Am 04.02.2016 um 11:31 schrieb Tomasz Ganicz: Also - normally in Academia - if there are two strongly opposite reviews (one very positive, one very negative) a typical procedure is to send the submission to the third one.

2016-02-04 10:54 GMT+01:00 Tomasz Ganicz <polimerek@gmail.com mailto:polimerek@gmail.com>:

Well I think that double blind peer review hardly make sense here, from the reviewer POV, as we are in fact small community and it is easy to guess who was a submiter in most cases. For example - if there is a submission about project X in country Y, which was funded by WMF grant - it is very easy to find out who was grantee and it is rather obvious that that person is a submitter :-)

Also judging from the several reviewers comments which I saw already - they did not follow the very vague criteria which was posted here:

https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evalu...

Normally - at least in Academia - reviewers are forced directly (by the review form) to address their opinion in relation to the criteria. The criteria were:

"

1. problems and possible solutions in a specific field
2. proposals for others to replicate
3. issues (positive or negative) which have emerged from projects
4. issues you want to raise which you feel have not been
   discussed yet
5. issues which are at the centre of an online debate that you
   would like to address offline

"

1-4 are IMHO relatively easy to evaluate - I would expect from the reviews to answer yes or no to them. 5 is a bit tricky as it depends strongly of what the reviewer think is "at the centre" - but I would expect that they at least explain in few words here what they think is "at the centre" or not :-)

2016-02-04 0:22 GMT+01:00 Dariusz Jemielniak <darekj@alk.edu.pl mailto:darekj@alk.edu.pl>:

   hi,

   I have some comments as a person from Academia (and not
   involved in Wikimania process in any way):

   1. Short reviews are definitely not helping in addressing the
   frustration of rejection, yet are quite common in academic
   peer reviewing, especially for conferences. 

   2. Double blind peer review (not knowing who is reviewed, and
   not knowing who reviews) is a standard in Academia, although
   some perceive it as contributing to lack of responsibility
   (especially true in competitive journal submissions).

   3. Two reviewers per submission is absolutely on par with the
   conference standards I'm used to. Sometimes there are three,
   but two is absolutely acceptable (although a third opinion
   should be used if the two disagree too much). 

   4. It could be useful to sensitize the reviewers that the main
   purpose of the review is to help the author to do better next
   time. 

   5. All this is volunteer work. We should be, generally,
   grateful to reviewers (but in the same time grateful to the
   contributors, too). 

   best,

   dj

   On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers
   <maarten@mdammers.nl <mailto:maarten@mdammers.nl>> wrote:

       What kind of ridiculous process is this? This is all I got:

       ===============

       ----------------------- REVIEW 1 ---------------------
       PAPER: 194
       TITLE: GLAM+Wikidata
       AUTHORS: Sandra Fauconnier and Maarten Dammers

       OVERALL EVALUATION: 8 (Very good)

       ----------- REVIEW -----------
       8


       ----------------------- REVIEW 2 ---------------------
       PAPER: 194
       TITLE: GLAM+Wikidata
       AUTHORS: Sandra Fauconnier and Maarten Dammers

       OVERALL EVALUATION: 6 (Rather interesting)

       ----------- REVIEW -----------
       6

       ==============

       So only two people reviewed this? Who are these people? Why is this secret? Last year I had 5 people reviewing my submission [1].

       Maarten

       [1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5


       Op 3-2-2016 om 23:15 schreef Andy Mabbett:

...

       I've just received feedback on one of my pitches saying,
       in part:

       "Bad boy Andy! This is supposed to be an anonymous review
       process, so starting your abstract with your own name, is
       not entirely fair."

       -- 
       Andy Mabbett
       @pigsonthewing
       http://pigsonthewing.org.uk



       _______________________________________________
       Wikimania-l mailing list
       Wikimania-l@lists.wikimedia.org
       <mailto:Wikimania-l@lists.wikimedia.org>
       https://lists.wikimedia.org/mailman/listinfo/wikimania-l

       _______________________________________________
       Wikimania-l mailing list
       Wikimania-l@lists.wikimedia.org
       <mailto:Wikimania-l@lists.wikimedia.org>
       https://lists.wikimedia.org/mailman/listinfo/wikimania-l




   -- 

   __________________________
   prof. dr hab. Dariusz Jemielniak
   kierownik katedry Zarządzania Międzynarodowego
   i grupy badawczej NeRDS
   Akademia Leona Koźmińskiego
   http://n <http://www.crow.alk.edu.pl/>wrds.kozminski.edu.pl
   <http://wrds.kozminski.edu.pl> 

   członek Akademii Młodych Uczonych Polskiej Akademii Nauk
   członek Komitetu Polityki Naukowej MNiSW

   Wyszła pierwsza na świecie etnografia Wikipedii "Common
   Knowledge? An Ethnography of Wikipedia" (2014, Stanford
   University Press) mojego
   autorstwa http://www.sup.org/book.cgi?id=24010

   Recenzje
   Forbes: http://www.forbes.com/fdc/welcome_mjx.shtml
   Pacific
   Standard: http://www.psmag.com/navigation/books-and-culture/killed-wikipedia-93777/
   Motherboard: http://motherboard.vice.com/read/an-ethnography-of-wikipedia
   The
   Wikipedian: http://thewikipedian.net/2014/10/10/dariusz-jemielniak-common-knowledge

   _______________________________________________
   Wikimania-l mailing list
   Wikimania-l@lists.wikimedia.org
   <mailto:Wikimania-l@lists.wikimedia.org>
   https://lists.wikimedia.org/mailman/listinfo/wikimania-l

-- Tomek "Polimerek" Ganicz http://pl.wikimedia.org/wiki/User:Polimerek http://www.ganicz.pl/poli/

-- Tomek "Polimerek" Ganicz http://pl.wikimedia.org/wiki/User:Polimerek http://www.ganicz.pl/poli/

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

-- Sebastian Wallroth mobile +4917615154002 http://about.me/real68er PGP Key https://de.wikipedia.org/wiki/Benutzer:Sebastian_Wallroth/PGP

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l

I would have thought knowing who was proposing the presentation was a significant factor in its evaluation as that provides the critical point of whether the person has the knowledge to speak about the issue. Such games make the process appear to be unprofessional offering very little in the way of addressing critical issues and potentially opening up the community to abuse.

Annon review is fine as that enables the reviewers to be critical without a concern of some backlash, I hope that before you publish the reviews publicly that you consider the impact of those reviews, what is actually and ensure the reviews dont do harm to the person receiving them

On 4 February 2016 at 06:15, Andy Mabbett andy@pigsonthewing.org.uk wrote:

I've just received feedback on one of my pitches saying, in part:

"Bad boy Andy! This is supposed to be an anonymous review process, so starting your abstract with your own name, is not entirely fair."

-- Andy Mabbett @pigsonthewing http://pigsonthewing.org.uk

---

Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l