Responding to Lodewijk, David N.
I'm guessing that there are about 10 countries that would have the possible
problem of reviewing something like 1,000 photos a day, so an "as you go"
jury tool will likely be critical to all of them. France already said
something about reviewing photos "on the fly" which I think is the same
thing. Is it possible to break down the upload categories from something
like [[Category:WLM US 2013]] into a set of daily categories, e.g.
[[Category:WLM US 2013 9-1]] or weekly categories, e.g. [[Category:WLM US
2013 Week 1]]. If that is the case, then reviewing could take place "as
you go" for each day or week, and then the roughly screened output combined
for the official jury to decide on. I assume getting the output for each
day or week and combining them would not be a problem.
I don't see a reasonable technical solution for the gaming problem I
described. Rather we might just keep an eye peeled to see if it is a real
problem.
The "starting at a random place on the list" looks like a good solution to
the "everybody rates the same photos" problem I described.
I'd guess renaming or deleting files would not involve many files at all,
and will be technically bothersome no matter what. Is there a way just to
flag these rare events for an organizer, who can just make sure that the
photo gets on some list to be reviewed?
In short, you're definitely making progress in calming my fears.
Pete
User:Smallbones
Date: Tue, 20 Aug 2013 11:48:34 +0200
From: Lodewijk <lodewijk(a)effeietsanders.org>
To: Wiki Loves Monuments Photograph Competition
<wikilovesmonuments(a)lists.wikimedia.org>
Subject: Re: [Wiki Loves Monuments] Jury tool - practical
considerations
Message-ID:
<CACf6BesC+GS+EKQrDytdE7K9xdKyz=GpKD27Z=bTCwPmQjw35g(a)mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
2013/8/20 David Narvaez <david.narvaez(a)computer.org>
> On Mon, Aug 19, 2013 at 12:50 PM, Peter Ekman <pdekman(a)gmail.com> wrote:
> > B) Will it be able to handle 1,000 entries per day? Or a better
> question -
> > What will it take to make it work well with 1,000 entries per day?
> > - technically, I'd think one upload (perhaps of just one day's worth of
> > entries each day) would seem to be needed. If each reviewer needs to
> upload
> > 30,000 images each time they use it, it could be pretty difficult.
>
>
> Well, nothing is really uploaded to the tool, but some database
> entries (and 1,000 db entries are manageable, probably with some
> tweaks I have in mind), but I think this question is oriented towards
> the "rate as you go" mode you mention later, is that right?
>
> > -technically, can it be set so that new images (or perhaps randomly
> selected
> > images) are judged by different reviewers. We wouldn't want the same
500
> > images being at the front of the line that get judged by 10 different
> > people, but the other 29,500 don't get seen at all.
>
> That is what the configurations for policy of the round are for: With
> a "All photos are reviewed exactly once" policy, no photo is ever
> served to two different juries, with a "All photos are reviewed by at
> least x juries" all photos are also reviewed, and more than once.
>
> > -sociologically, how to deal with folks who go through and select the
> photos
> > they uploaded and rate poorly other very good photos (gaming)?
>
> This would only apply to Open rounds and has no way to be fixed.
> Invitational rounds, on the other hand, would require juries to be
> pre-approved by the organizers of the contest, and that means the
> organizers won't put a jury that uploaded a photo.
>
> > -what happens to photos not rated in cases there are too many photos and
> not
> > enough reviewers? Are they just left behind?
>
> Haven't thought of that. In my small head all photos have to be reviewed.
>
Would this be solved if you start at a random point in the review list
rather than always at the beginning? Then every jury member would start
somewhere else, and the odds of images having zero reviews is minimized.
>
> > -is "rate as you go" a possibility, or would you have to wait until the
> end
> > of the contest?
>
> Good question. The tool is not really designed for that, but could be
> extended to support this. I'd like to hear other opininos about this.
>
I think if you allow feeding multiple txt files through the contest (taking
out doublures) that might work?
>
> > -presentation, could you include a box where the local designer (who
> selects
> > the various options) explains how they work, i.e. what are the mechanics
> for
> > a photo to get to the next round.
>
> That's doable, and shouldn't be too hard. I would only need help with
> the wording (see the other thread about design).
>
I think if you just present this as an option, you could have this as an
empty text box in the configuration, and let the configurer worry about the
wording (and language).
A question of my own: what happens if an image gets renamed/deleted. Will
the tool be able to handle that?
Lodewijk