It seems to me that you can't possibly give a jury a 1,000 photos and expect them to come up with anything reasonable. That type of system would also drive away quality jurors - the best jurors simply wouldn't have time for all that. And if we're talking about 10,000 photos, it just gets worse. There has to be some sort of pre-screening, whether we like it or not.
A couple of suggestions for pre-screening: 1. Let the photographer decide which of his photos is best - say 1 for the entire contest or 1 for each day he/she uploads. 2. Have a contest each day, with a each photographer who uploaded that day nominating a single photo, and letting the community vote (I'd say +1 for each photo you like) then after a few days a selected screener from the community selects 2 or 3 photos from the group that has the highest score. After 30 days, you'd have 60-90 photos that the jury can deal with, each photog would have had the chance to nominate his best photos (multiple times), the community would have their say, and the screeners would not have to deal with 1,000s of photos.
Pete Ekman User:Smallbones
Date: Fri, 22 Jun 2012 00:16:01 +0200
From: Lodewijk <lodewijk@effeietsanders.org javascript:;> To: Wiki Loves Monuments Photograph Competition <wikilovesmonuments@lists.wikimedia.org javascript:;> Subject: Re: [Wiki Loves Monuments] jury process - easy and neat? Message-ID: <CACf6BesGfNFEVuXoQapbahbTmCiyMh0FdAOgZpGBungyENpZzA@mail.gmail.comjavascript:;
Content-Type: text/plain; charset="iso-8859-1"
Of course if you only use this tool as a means to get the best 1% of the pictures... it doesn't have to be as precise. Just have the final selection in a second round, with zero scores - then vote again for those or have a discussion on the mailing list.
Personally, I think every country can decide its own process - so lets make the tool somewhat dynamic, but not shoot down major options. Last year Russia had a public voting, and it seems they were happy with it. I'm no big fan of it, but who am I to stop it? :)
Lodewijk
2012/6/22 Platonides <platonides@gmail.com javascript:;>
On 21/06/12 18:29, aude wrote:
We have a technical volunteer "intern" helping us in DC and this is something he wants to work on.
I'm not exactly sure how the process worked last year, but we're thinking either integrating something into the WordPress site so that people can browse photos (pulled from Commons, like InstantCommons) and rate them. A public voting phase could help narrow the selection for the jury, which could also make use of the tool.
Thoughts? suggestions? brilliant ideas?
Each instance used its own system. Nuno made a web application for voting, from which you could start. There is also a toolserver tool which facilitates downloading all the files from a category. Last year we provided zips with all the images which our jury downloaded. Then they chose those they deemed the best, which went to a second round. If you're joining the jury in one room (with multiple computers), it may be simpler if they are provided directly a copy of the images in an external drive. Remember that not all your jury members will be
tech-savvy.
If I were designing such system, I'd make it an interface where the jury would go giving a mark from 1 to 10 to each photo. But not actually restricted to that. So after 200 photos, it could give a 12 if needed (instead of recalculating all previous votes), or even provide marks with decimal points. The interface itself would be just a (zoomable) gallery of the photos that he didn't review yet (plus auxiliar pages, to view the best ranked by you, change a vote, etc.).
What would be interesting is that they could be using it from the first day, so instead of reviewing all files after the competition closes, they could keep up with the upload rate. As far as a 10 given on Sep 1st is the same as one given on 30th Sep (which is easy by things like moving the bar higher up to a 14), it'd be equivalent. You then fetch the N most ranked from each member to next round, so different scales aren't a problem.
As for a public voting, I don't think it would work. You would need each person to review a significant number of files, otherwise the noise given by each different reviewer (a 10 by me could be an 8 by you). And you won't be getting volunteers to review thousands of photos. Only the jury will do that, because they agreed to. (Obviously, anyone is able to volunteer to be jury. We were discussing on wlm-iberconf ml giving a jury for another country and getting one).
What I had thought as a possibility for involving the public was to allow it to choose a number of photos that pass to the next round (just as each jury does), thus ensuring they get attention. But that won't the jury task of having to view all of them.
Regards
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.eu
1. I agree with Peter. There should be a reasonable limit for uploads; If there is not, each participant will be free to take dozens of photos of the same monument and upload them all to see if any of them is lucky! The limit must be put with caution, however, as we won't like to lose valuable photos. Here, in Colombia, we are considering to put a limit of 10 (maybe even 5) photos per monument per participant. That way, each participant still can send "unlimited" photos, but they would have to be of different monuments.
2. However, I think that even 4000 is a reasonable number for a jury IF some kind of supporting system (like the ones mentioned before) is used.
3. Be careful when involving "the community" for selecting, as we don't want to transform this into a popularity contest. Extreme case: people giving 10 to their friends photos, and 1 to everything else.
4. Comment: check Worth 1000's voting karma system: http://all.worth1000.com/faq#voting. Something like that could be implemented when designing open-voting systems.
Racso
2012/6/21 Peter Ekman pdekman@gmail.com
It seems to me that you can't possibly give a jury a 1,000 photos and expect them to come up with anything reasonable. That type of system would also drive away quality jurors - the best jurors simply wouldn't have time for all that. And if we're talking about 10,000 photos, it just gets worse. There has to be some sort of pre-screening, whether we like it or not.
A couple of suggestions for pre-screening:
- Let the photographer decide which of his photos is best - say 1 for
the entire contest or 1 for each day he/she uploads. 2. Have a contest each day, with a each photographer who uploaded that day nominating a single photo, and letting the community vote (I'd say +1 for each photo you like) then after a few days a selected screener from the community selects 2 or 3 photos from the group that has the highest score. After 30 days, you'd have 60-90 photos that the jury can deal with, each photog would have had the chance to nominate his best photos (multiple times), the community would have their say, and the screeners would not have to deal with 1,000s of photos.
Pete Ekman User:Smallbones
On Fri, Jun 22, 2012 at 4:29 AM, Racso wrote:
- I agree with Peter. There should be a reasonable limit for uploads; If
there is not, each participant will be free to take dozens of photos of the same monument and upload them all to see if any of them is lucky! The limit must be put with caution, however, as we won't like to lose valuable photos. Here, in Colombia, we are considering to put a limit of 10 (maybe even 5) photos per monument per participant. That way, each participant still can send "unlimited" photos, but they would have to be of different monuments.
Last year in my country we had one user submitting like this, *a lot* of pictures, sometime over 60, for the same monument, with some of them almost identical, not adding anything new. I think this was useless, but respected his will to put into the commons every little detail (he's also an article contributor to Wikipedia, so he may want to cover special things).
I personally do not want to limit in any way the contributions, if people want to donate, let them do it. But they should not expect a prize for that, I want to reward quality more than quantity.
This year for some monuments I plan to send myself up to 10 different pictures, since some are big and deserve it (but my photos won't enter the contest, I am an organizer and a member of the local jury).
- However, I think that even 4000 is a reasonable number for a jury IF some
kind of supporting system (like the ones mentioned before) is used.
We had close to 6000 images and judging was NOT a pain.
- Be careful when involving "the community" for selecting, as we don't want
to transform this into a popularity contest. Extreme case: people giving 10 to their friends photos, and 1 to everything else.
This is not democracy, is meritocracy. With popular vote you may end with the *good* images out of the selection, just look at how such contests happen on facebook where the contestants "fight" with their "friends" numbers and getting "likes". I prefer a jury who understand what a photo is and what a historical monument is.
- Comment: check Worth 1000's voting karma
system: http://all.worth1000.com/faq#voting. Something like that could be implemented when designing open-voting systems.
On Fri, Jun 22, 2012 at 3:19 AM, Peter Ekman wrote:
It seems to me that you can't possibly give a jury a 1,000 photos and expect them to come up with anything reasonable. That type of system would also drive away quality jurors - the best jurors simply wouldn't have time for all that. And if we're talking about 10,000 photos, it just gets worse. There has to be some sort of pre-screening, whether we like it or not.
A jury put in front of 1000 or more photos would have a difficult job, but with a pre-screening you can get to a few hundreds of images and a reasonable amount of work.
A couple of suggestions for pre-screening:
- Let the photographer decide which of his photos is best - say 1 for the
entire contest or 1 for each day he/she uploads. 2. Have a contest each day, with a each photographer who uploaded that day nominating a single photo, and letting the community vote (I'd say +1 for each photo you like) then after a few days a selected screener from the community selects 2 or 3 photos from the group that has the highest score. After 30 days, you'd have 60-90 photos that the jury can deal with, each photog would have had the chance to nominate his best photos (multiple times), the community would have their say, and the screeners would not have to deal with 1,000s of photos.
Something like that would require a large organizational effort and a large community, which is not the case for most of the participating countries. Small teams and small communities will have to "Keep It Simple and Stupid". [1]
[1] http://en.wikipedia.org/wiki/KISS_principle
While making it as easy as possible for the jury, let's not forget the purpose of the contest, to get educational pictures of as many monuments as possible. There need to be an incentive for photographers to not be satisfied with just uploading their best image. Uploading with metadata is a pain, and if they are not entering the contest the risk is that we will miss out of some educational pictures that may not be the prettiest. The contest is our carrot to make people upload, and if the carrot is smaller not all will chase for it.
I think Racso is on to something though, by limiting it to a certain number per monument. The limit must be higher than one, eg. for a church the interior is at least as interesting as the exterior for one thing and while the photographer thinks one of them has better chance to win, maybe the other is of most value to the projects. Perhaps ten is enough, that could make those photographers that upload hundreds of picture of each monument less overwhelming. Can we do such a limitation technically, or do we make it as a strong recommendation to the contestants or solve it in another way?
Hi all,
two issues are getting mixed up here. So I'm going to split the thread. I'll start a new thread about a maximum number of photos per photographer in a minute.
About the jury. In the Netherlands we had contact with professional newspaper photographers, and I was told by them it was totally not uncommon for them to go through thousands and thousands of pictures to find the right one. They even have special software to make that easier. I didn't get the impression that more photos lead to lower quality jurywork, so I'm curious how you got to that assumption, Peter!
In my experience, juries find numbers around 500-1000 ideal because it allows enough choice but is like totally easy to go through. Up to 10.000 seems to be quite acceptable with the right tools. If it gets above 5000 you may want to include a step in the process to lower the number of photos per juror. Some ways to accomplish that are to cut the set in a fer parts, and let each juror make a pre-selection of their set. Then combine those preselections for the whole jury. Another option would be to have a preselection by the organizers (read: we get some volunteers to do it), or just by the Wikipedian jury members. Please note that in either case the whole process can only take up to a month. The results should be in by the beginning of November.
Best, Lodewijk
2012/6/22 Jan Ainali jan.ainali@wikimedia.se
While making it as easy as possible for the jury, let's not forget the purpose of the contest, to get educational pictures of as many monuments as possible. There need to be an incentive for photographers to not be satisfied with just uploading their best image. Uploading with metadata is a pain, and if they are not entering the contest the risk is that we will miss out of some educational pictures that may not be the prettiest. The contest is our carrot to make people upload, and if the carrot is smaller not all will chase for it.
I think Racso is on to something though, by limiting it to a certain number per monument. The limit must be higher than one, eg. for a church the interior is at least as interesting as the exterior for one thing and while the photographer thinks one of them has better chance to win, maybe the other is of most value to the projects. Perhaps ten is enough, that could make those photographers that upload hundreds of picture of each monument less overwhelming. Can we do such a limitation technically, or do we make it as a strong recommendation to the contestants or solve it in another way?
-- Best Jan Ainali Chairman, Wikimedia Sverige http://se.wikimedia.org/wiki/Huvudsida
2012/6/22 Nicu Buculei nicubunu@gmail.com
On Fri, Jun 22, 2012 at 3:19 AM, Peter Ekman wrote:
It seems to me that you can't possibly give a jury a 1,000 photos and
expect
them to come up with anything reasonable. That type of system would
also
drive away quality jurors - the best jurors simply wouldn't have time
for
all that. And if we're talking about 10,000 photos, it just gets worse. There has to be some sort of pre-screening, whether we like it or not.
A jury put in front of 1000 or more photos would have a difficult job, but with a pre-screening you can get to a few hundreds of images and a reasonable amount of work.
A couple of suggestions for pre-screening:
- Let the photographer decide which of his photos is best - say 1 for
the
entire contest or 1 for each day he/she uploads. 2. Have a contest each day, with a each photographer who uploaded that
day
nominating a single photo, and letting the community vote (I'd say +1
for
each photo you like) then after a few days a selected screener from the community selects 2 or 3 photos from the group that has the highest
score.
After 30 days, you'd have 60-90 photos that the jury can deal with,
each
photog would have had the chance to nominate his best photos (multiple times), the community would have their say, and the screeners would not
have
to deal with 1,000s of photos.
Something like that would require a large organizational effort and a large community, which is not the case for most of the participating countries. Small teams and small communities will have to "Keep It Simple and Stupid". [1]
[1] http://en.wikipedia.org/wiki/KISS_principle
-- nicu :: http://photoblog.nicubunu.ro/
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.eu
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.eu
Hi Peter,
On Fri, Jun 22, 2012 at 2:19 AM, Peter Ekman pdekman@gmail.com wrote:
- Let the photographer decide which of his photos is best - say 1 for the
entire contest or 1 for each day he/she uploads.
I strongly oppose this idea. From my experience last year, this will ruin much of the fun. Almost every winner I spoke told me that if he or she had been given the possibility to choose which of their pictures went into the contest, they wouldn't have selected a single one or at least not the one that actually got a prize. Many really great pictures wouldn't have made it (in Germany, many top 10 pictures would be missing). That's especially true for new contributors.
Technically a competition with this kind of preselection wouldn't be a big photo competition anymore but rather a pretty small one. (And if you don't want your photo to be judged, you can just upload it to Commons without the templates).
- Have a contest each day, with a each photographer who uploaded that day
nominating a single photo, and letting the community vote (I'd say +1 for each photo you like) then after a few days a selected screener from the community selects 2 or 3 photos from the group that has the highest score. After 30 days, you'd have 60-90 photos that the jury can deal with, each photog would have had the chance to nominate his best photos (multiple times), the community would have their say, and the screeners would not have to deal with 1,000s of photos.
I believe that's much too difficult and time consuming. As a new contributor, I might come across the site notice, look up some monuments, take some pictures during my sunday afternoon walk and upload them on monday. That'll be difficult enough, I certainly don't want to participate in a voting schedule.
When you have a bunch of people in charge of the pre-selection process (last year, we had 7 jury members select from roughly 4000 pictures each, which was a bit too much, though), it's doable and doesn't take too much time. Have 20 to 30 community members take a look at 1/20th to 1/30th of all uploaded pictures (It's useful to not have them judge a day each because of the batches of similar pictures) and narrow it down to 20-30 pictures each. Then you bring in the actual jury.
Regards,
Kilian
wikilovesmonuments@lists.wikimedia.org