I'd be interested to hear from countries that have experience from previous years how you pre-filtered the submissions for the jury. As this is the first year for the UK, we have no idea how many entries we will get, but it may well be too many for out three-person jury to handle.
I know that some countries have used special panels for pre-filtering, and others have used the Wiki community. What were your experiences, and what technical means did you use to help with the filtering?
Any insights on how best to handle this would be much appreciated.
Michael
Our filtering was done by the Wikimedians organizing the competition. We used last year's judging tool, which allows as many images per page as you want, which makes it easy to filter literally thousands of images a day. Only one person needs to go over each image (because we trust our organizers). You should immediately filter out the pictures that have absolutely no chance of getting to one of the top places, which means you'll have immediately filtered out 3/4 or more of the images. After that it's a bit more meticulous, but I'm sure one person can do it.
Ideally you should have in the area of 200 images for the final jury (I think this is the statistic for many countries, not just Israel).
On Mon, Aug 26, 2013 at 8:14 PM, Michael Maggs Michael@maggs.name wrote:
I'd be interested to hear from countries that have experience from previous years how you pre-filtered the submissions for the jury. As this is the first year for the UK, we have no idea how many entries we will get, but it may well be too many for out three-person jury to handle.
I know that some countries have used special panels for pre-filtering, and others have used the Wiki community. What were your experiences, and what technical means did you use to help with the filtering?
Any insights on how best to handle this would be much appreciated.
Michael
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
In Mexico we had a very similar process, made by 5 volunteers. We deliver 100 photos to the final jury. I think the filter needs to be make by different people to avoid saturation and visual fatigue. For this year we will have two rounds of training for volunteers who will support us in the process.
2013/8/26 Ynhockey ynhockey@gmail.com
Our filtering was done by the Wikimedians organizing the competition. We used last year's judging tool, which allows as many images per page as you want, which makes it easy to filter literally thousands of images a day. Only one person needs to go over each image (because we trust our organizers). You should immediately filter out the pictures that have absolutely no chance of getting to one of the top places, which means you'll have immediately filtered out 3/4 or more of the images. After that it's a bit more meticulous, but I'm sure one person can do it.
Ideally you should have in the area of 200 images for the final jury (I think this is the statistic for many countries, not just Israel).
On Mon, Aug 26, 2013 at 8:14 PM, Michael Maggs Michael@maggs.name wrote:
I'd be interested to hear from countries that have experience from previous years how you pre-filtered the submissions for the jury. As this is the first year for the UK, we have no idea how many entries we will get, but it may well be too many for out three-person jury to handle.
I know that some countries have used special panels for pre-filtering, and others have used the Wiki community. What were your experiences, and what technical means did you use to help with the filtering?
Any insights on how best to handle this would be much appreciated.
Michael
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Yep, just to clarify, we had more than one person do the filtering, but just saying that timewise one person can easily handle thousands of images in this fashion. It's always best to have more than one person though, if possible.
On Mon, Aug 26, 2013 at 9:36 PM, Ivan Martínez galaver@gmail.com wrote:
In Mexico we had a very similar process, made by 5 volunteers. We deliver 100 photos to the final jury. I think the filter needs to be make by different people to avoid saturation and visual fatigue. For this year we will have two rounds of training for volunteers who will support us in the process.
2013/8/26 Ynhockey ynhockey@gmail.com
Our filtering was done by the Wikimedians organizing the competition. We used last year's judging tool, which allows as many images per page as you want, which makes it easy to filter literally thousands of images a day. Only one person needs to go over each image (because we trust our organizers). You should immediately filter out the pictures that have absolutely no chance of getting to one of the top places, which means you'll have immediately filtered out 3/4 or more of the images. After that it's a bit more meticulous, but I'm sure one person can do it.
Ideally you should have in the area of 200 images for the final jury (I think this is the statistic for many countries, not just Israel).
On Mon, Aug 26, 2013 at 8:14 PM, Michael Maggs Michael@maggs.namewrote:
I'd be interested to hear from countries that have experience from previous years how you pre-filtered the submissions for the jury. As this is the first year for the UK, we have no idea how many entries we will get, but it may well be too many for out three-person jury to handle.
I know that some countries have used special panels for pre-filtering, and others have used the Wiki community. What were your experiences, and what technical means did you use to help with the filtering?
Any insights on how best to handle this would be much appreciated.
Michael
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
-- *Atentamente:
Iván Martínez Presidente Wikimedia México A.C. wikimedia.mx
Imagina un mundo en donde cada persona del planeta pueda tener acceso libre a la suma total del conocimiento humano. Eso es lo que estamos haciendo http://es.wikipedia.org. *
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
In Chile, a small group of volunteers (7) make the filters (too small, licensing, etc.). The jury (5) saw "only" the 10% of images in the contest (we had 4100 images uploaded).
The work to reach only 400 finalist images took around 6 hours and 1 week to confirm the free license (CC in Flickr or other sites or original work).
Regards,
On Mon, Aug 26, 2013 at 2:37 PM, Ynhockey ynhockey@gmail.com wrote:
Yep, just to clarify, we had more than one person do the filtering, but just saying that timewise one person can easily handle thousands of images in this fashion. It's always best to have more than one person though, if possible.
On Mon, Aug 26, 2013 at 9:36 PM, Ivan Martínez galaver@gmail.com wrote:
In Mexico we had a very similar process, made by 5 volunteers. We deliver 100 photos to the final jury. I think the filter needs to be make by different people to avoid saturation and visual fatigue. For this year we will have two rounds of training for volunteers who will support us in the process.
2013/8/26 Ynhockey ynhockey@gmail.com
Our filtering was done by the Wikimedians organizing the competition. We used last year's judging tool, which allows as many images per page as you want, which makes it easy to filter literally thousands of images a day. Only one person needs to go over each image (because we trust our organizers). You should immediately filter out the pictures that have absolutely no chance of getting to one of the top places, which means you'll have immediately filtered out 3/4 or more of the images. After that it's a bit more meticulous, but I'm sure one person can do it.
Ideally you should have in the area of 200 images for the final jury (I think this is the statistic for many countries, not just Israel).
On Mon, Aug 26, 2013 at 8:14 PM, Michael Maggs Michael@maggs.namewrote:
I'd be interested to hear from countries that have experience from previous years how you pre-filtered the submissions for the jury. As this is the first year for the UK, we have no idea how many entries we will get, but it may well be too many for out three-person jury to handle.
I know that some countries have used special panels for pre-filtering, and others have used the Wiki community. What were your experiences, and what technical means did you use to help with the filtering?
Any insights on how best to handle this would be much appreciated.
Michael
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
-- *Atentamente:
Iván Martínez Presidente Wikimedia México A.C. wikimedia.mx
Imagina un mundo en donde cada persona del planeta pueda tener acceso libre a la suma total del conocimiento humano. Eso es lo que estamos haciendo http://es.wikipedia.org. *
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
In case of Poland - we had over 50000 pictures to select - in fact the highest number of all local WLM contests ;-) Our jury consisted of 3 wikimedians and 4 external experts. In fact only wikimedians were engaged in first selection. It was organized spontaneously by secretary of the jury. He has collected lists of pictures in alphabetical order and put it on his discussion page of Wikimedia Polska wiki:
http://pl.wikimedia.org/wiki/U%C5%BCytkownik:Karol007/WLZ
He created general rules of selecting pictures - which was almost purely technical. As it was too much work for 3 people he asked several trusted wikimedians to help the jury (including me ;-) ). On average it resulted in selecting aroun 6% of the pictures - but it was still a lot (around 3000). In the second round - each member of the jury was asked to select no more than 100 out of the pre-selected 3000. It resulted in around 500 pictures. From this - each jury member was asked to select top 12. It resulted in around 60 finalist. The discussion and final selection of 10 best from 60 finalist was made on real-life meeting which took several hours of hard discussion ;-)
2013/8/26 Michael Maggs Michael@maggs.name:
I'd be interested to hear from countries that have experience from previous years how you pre-filtered the submissions for the jury. As this is the first year for the UK, we have no idea how many entries we will get, but it may well be too many for out three-person jury to handle.
I know that some countries have used special panels for pre-filtering, and others have used the Wiki community. What were your experiences, and what technical means did you use to help with the filtering?
Any insights on how best to handle this would be much appreciated.
Michael
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wikimedia Italia had a pre-jury who selected 500 pics for the Jury. We did not use the judging tool, as it would be better (more reliable, more formal) to have every picture seen and evaluated from multiple persons (and not just one). If the judging tool could be set up like that, it would be *very* useful also for different workflows.
Aubrey
On Mon, Aug 26, 2013 at 10:50 PM, Tomasz Ganicz polimerek@gmail.com wrote:
In case of Poland - we had over 50000 pictures to select - in fact the highest number of all local WLM contests ;-) Our jury consisted of 3 wikimedians and 4 external experts. In fact only wikimedians were engaged in first selection. It was organized spontaneously by secretary of the jury. He has collected lists of pictures in alphabetical order and put it on his discussion page of Wikimedia Polska wiki:
http://pl.wikimedia.org/wiki/U%C5%BCytkownik:Karol007/WLZ
He created general rules of selecting pictures - which was almost purely technical. As it was too much work for 3 people he asked several trusted wikimedians to help the jury (including me ;-) ). On average it resulted in selecting aroun 6% of the pictures - but it was still a lot (around 3000). In the second round - each member of the jury was asked to select no more than 100 out of the pre-selected 3000. It resulted in around 500 pictures. From this - each jury member was asked to select top 12. It resulted in around 60 finalist. The discussion and final selection of 10 best from 60 finalist was made on real-life meeting which took several hours of hard discussion ;-)
2013/8/26 Michael Maggs Michael@maggs.name:
I'd be interested to hear from countries that have experience from
previous years how you pre-filtered the submissions for the jury. As this is the first year for the UK, we have no idea how many entries we will get, but it may well be too many for out three-person jury to handle.
I know that some countries have used special panels for pre-filtering,
and others have used the Wiki community. What were your experiences, and what technical means did you use to help with the filtering?
Any insights on how best to handle this would be much appreciated.
Michael
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
-- Tomek "Polimerek" Ganicz http://pl.wikimedia.org/wiki/User:Polimerek http://www.ganicz.pl/poli/ http://www.cbmm.lodz.pl/work.php?id=29&title=tomasz-ganicz
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
preselection actually doesn't happen in all countries, so I feel I have to break this chorus :)
In the Netherlands we have always had a system where every jury member had to see every image submitted (more or less). We learned that professional photographers actually are very well able to handle this kind of job, especially if they have some kind of editing function too. We learned that they can process actually thousands of images per hour - so if they are willing to spend a few hours and are helped by the right equipment (jury tool!) they can process a /lot/. For reference, the Netherlands had roughly 12k, 13k and 6k submissions in the past years. We asked the jury members to come up with their top-50/100, added those together and then discussed them during a real life jury meeting where the final decisions were made.
In other countries, this was undoable, and the list was split up in a way that each image was initially seen by one jury member (I believe this is the system used in Germany). They would then select a small number from their own stack, and together this would get a second round treatment.
Basically, what you see in all countries is that there are multiple rounds. First there is a round where, one way or another, the big stack is reduced to a much smaller number. Usually between 100 and 1000, depending on your starting set (Poland is probably an exception - they needed a round in between). Then, there us another round to reduce the set further to somewhere between 10 and 100 - which makes it comprehensible for the human brain. From this point on, you can start discussing the order the images have to be in.
Each of these rounds can take multiple forms. They can be a small group of people (or even one person) making quick decisions yes/no, it can crowdsourced (community) or it can be the jury itself.
I am myself no fan of the system where a single person makes the decision in any round. I feel that there should /always/ be at least two people involved - otherwise the personal taste plays too much a role. This means twice the amount of work, but I think it is worth it because it means less very good images falling through the cracks. (if I dismiss one picture, that should not mean that it doesn't make a chance anywhere).
I think all the models offered above are now supported in the jury tool - which allows a multi-round system setup, where each round can use a different selection method ranging from binary single person decision to crowdsourced rating.
Whatever method or setup you choose, I advise a few things: 1) make sure that you have your 10 nominations for the international round submitted well in time! The deadline is 31 October (UTC), and this deadline is NOT FLEXIBLE! Keep some space before that date, to be able to manage sickness or hit-by-bus-situations. After this deadline, the nominations will be closed, and the international jury starts their work. We will send more specific instructions about this when that time approaches. 2) Appoint a non-voting jury moderator that will manage the process and poke people. This can be a staff member in your case. 3) Set up a temporary mailing list, where each juror and the jury moderator is subscribed to. 4) describe the jury process publicly, and describe your deliberations (including reasons why the top-10/20/30 was selected) in a jury report. 5) Finally, make sure that your 10 nominations are submitted before the deadline. (Yes, this is mentioned twice, because it would be a pity if after all the work, your photos are not considered for the international round!)
Best, Lodewijk
2013/8/26 Andrea Zanni zanni.andrea84@gmail.com
Wikimedia Italia had a pre-jury who selected 500 pics for the Jury. We did not use the judging tool, as it would be better (more reliable, more formal) to have every picture seen and evaluated from multiple persons (and not just one). If the judging tool could be set up like that, it would be *very* useful also for different workflows.
Aubrey
On Mon, Aug 26, 2013 at 10:50 PM, Tomasz Ganicz polimerek@gmail.comwrote:
In case of Poland - we had over 50000 pictures to select - in fact the highest number of all local WLM contests ;-) Our jury consisted of 3 wikimedians and 4 external experts. In fact only wikimedians were engaged in first selection. It was organized spontaneously by secretary of the jury. He has collected lists of pictures in alphabetical order and put it on his discussion page of Wikimedia Polska wiki:
http://pl.wikimedia.org/wiki/U%C5%BCytkownik:Karol007/WLZ
He created general rules of selecting pictures - which was almost purely technical. As it was too much work for 3 people he asked several trusted wikimedians to help the jury (including me ;-) ). On average it resulted in selecting aroun 6% of the pictures - but it was still a lot (around 3000). In the second round - each member of the jury was asked to select no more than 100 out of the pre-selected 3000. It resulted in around 500 pictures. From this - each jury member was asked to select top 12. It resulted in around 60 finalist. The discussion and final selection of 10 best from 60 finalist was made on real-life meeting which took several hours of hard discussion ;-)
2013/8/26 Michael Maggs Michael@maggs.name:
I'd be interested to hear from countries that have experience from
previous years how you pre-filtered the submissions for the jury. As this is the first year for the UK, we have no idea how many entries we will get, but it may well be too many for out three-person jury to handle.
I know that some countries have used special panels for pre-filtering,
and others have used the Wiki community. What were your experiences, and what technical means did you use to help with the filtering?
Any insights on how best to handle this would be much appreciated.
Michael
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
-- Tomek "Polimerek" Ganicz http://pl.wikimedia.org/wiki/User:Polimerek http://www.ganicz.pl/poli/ http://www.cbmm.lodz.pl/work.php?id=29&title=tomasz-ganicz
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
On 27/08/13 13:32, Lodewijk wrote:
preselection actually doesn't happen in all countries, so I feel I have to break this chorus :)
In the Netherlands we have always had a system where every jury member had to see every image submitted (...)
In Spain we also had every jury member review all the images. WLM-ES worked with a jury of five members. Each member of the jury reviewed the 39460 participating photos and chose the 25 they deemed to be the best.
Joining the results of each of them, we made a pool of 117 photos (they could be up to 125, but there were photos selected by several members). The juries did not know what photos were each selected by each of the other ones.
Then in round 2 they gave points from 0 to 10 to each of them on artistic, technical and encyclopedic categories. This was done through an application which shuffled the images, to avoid a bias in the order of presentation. They were also unaware of what had voted each of them. We then sorted them by the number of points. Additionally, 15 of them turned out to be ineligible at this step.
Finally, they met in irc to straighten a few ties which appeared on the top-10 of that ordered list.
Whatever method or setup you choose, I advise a few things:
- make sure that you have your 10 nominations for the international
round submitted well in time! The deadline is 31 October (UTC), and this deadline is NOT FLEXIBLE! Keep some space before that date, to be able to manage sickness or hit-by-bus-situations. After this deadline, the nominations will be closed, and the international jury starts their work. We will send more specific instructions about this when that time approaches.
Prepare for an earlier delivery date: There may be silly delays discussing how the voting should be done (which would ideally have been fixed in Sep), you may need to create a zip with all photos, the server may break serving the file, there may be errors opening the files...
- Appoint a non-voting jury moderator that will manage the process and
poke people. This can be a staff member in your case.
+1 It's important to have someone in charge of poking the jury. Some members won't need it, but one guy can delay the whole process.
- Set up a temporary mailing list, where each juror and the jury
moderator is subscribed to. 4) describe the jury process publicly, and describe your deliberations (including reasons why the top-10/20/30 was selected) in a jury report. 5) Finally, make sure that your 10 nominations are submitted before the deadline. (Yes, this is mentioned twice, because it would be a pity if after all the work, your photos are not considered for the international round!)
Don't miss the deadline. If your nominations aren't available at that date, your photos won't enter the international round. This will also make your country participations not count for the Guinness Record. **This happened last year** (with sadness from all sides)
Finally, if your country is having difficulties with the process, *speak up*. It may be possible for other organisers to get you some extra juries or technical help. But we won't be able to help you if we didn't know that you needed it!
God save all our brave juries!
wikilovesmonuments@lists.wikimedia.org