I'll suggest that we won't effect much of anything this year with this conversation, so let's leave it for after the contest is over. I'll ask everybody to keep a journal on their jury procedures this year so we can improve them next year.

A couple of issues with Plotonides comments first: Nobody is gaming the system by self-nominating - that was built in as a compromise in the beginning. I'd take it out next year.

The "positive version" of Kaldari's tool would output the file names of favorite pix to a special sandbox page for further processing. The only problems that I have with Nuno's tool is that I don't know how to output data, or how to scale it up for general use. That's because I'm not a coder, not because of any fault in the software.

Anonymity in the screening process is probably necessary. Without some anonymity things might get very tricky and emotional. I'll readily admit that I've made some mistakes in screening - if you go through 1,000s of pix fairly quickly how could there not be mistakes? and I think most people will understand this. Except when somebody puts forth his special "baby" to the public ...

I'll admit to 2 types of mistakes - Type I (the most important) removing pix that shouldn't have been removed at this stage. I'll admit to a 1% error rate here, and if you look at the numbers - the implications are truly horrendous. Type II errors - leaving in pix that shouldn't have been left in - I'm not really very concerned about . I'll admit to a 10% error rate here.

If somebody were to complain about his baby being removed when it shouldn't have been, and then compares it to some Type II errors - if I were publicly identified, all I could do is give the seemingly lame excuse "everybody makes mistakes" But of course everybody does.

I think with "crowd sourcing" or just crowd voting, we have the chance to make this contest something very, very special. Everything that's put in front of the final jury may not be perfect, but at the very least, it could be "crowd pleasing." A voting system for masses of photos - I've come to believe - is a very hard technical project. Perhaps, if we can define exactly what we want, we could get the Foundation to work on this for us before next year.

I'll suggest a 3 stage general process. 1. Review for obvious technical problems (probably 2 strikes and you're out, rather than 1) 2. Crowd voting to get down to a limited number of crowd pleasing photos, and 3. A final round of judging by highly qualified judges. After all photography is an art, and experts must play a role.

Hope this helps (for next year) Pete

Message: 8 Date: Sat, 15 Sep 2012 00:25:14 +0200 From: Platonides platonides@gmail.com To: Wiki Loves Monuments Photograph Competition wikilovesmonuments@lists.wikimedia.org Subject: Re: [Wiki Loves Monuments] jury tools Message-ID: 5053AECA.5090809@gmail.com Content-Type: text/plain; charset=ISO-8859-1

On 13/09/12 05:05, Peter Ekman wrote:

Kaldari came up with a tool for WLM-US which works quite well IMHO as a part of the jury process, but is not the whole answer.(...)

The logic is that you can include a photo for further consideration by clicking "Nominate", or you can eliminate it from further consideration by clicking "Decline for nomination" and when you can't decide click "can't decide." It works by manipulating categories on the photo's file "Category:WLM-US 2012 unreviewed" to start off, then "reviewed" and/or "nominated" We're aiming to eliminate 90% at this point - keeping up as the photos come in. But in reality we're keeping 14%. I think one of the quirks is when folks manually change categories - it can just stay in the unreviewed category forever. The fix here is just manually changing the categories to the correct state.

It's a tough system (but quick) "one strike and you're out." But it is needed if you have several thousand photos to go through and want to get down to a few hundred in a reasonable time so that you can properly consider the best ones.

One drawback of this is accountability. There's no trace of who voted a picture through the app (categories are changed by the bot).

Another problem is that authors are adding themselves the categories nominating the images they just uploaded: https://commons.wikimedia.org/w/index.php?title=File:Comac_Church.jpg&di... https://commons.wikimedia.org/w/index.php?title=File%3ADramatic_Obelisk.jpg&... https://commons.wikimedia.org/w/index.php?title=File:Protestant_Children%27s... https://commons.wikimedia.org/w/index.php?title=File:Bellows_House_Missoula,...

(these are all different users tricking the system, random images provided by the tool)

I'd love it if there could be a positive version of this - say click keep and the bot places the file name in your sandbox - but the problem there would be you don't eliminate anything and the possible selection just keeps on growing.

You mean something like Nuno tool?