RE: Sylvain's comments and practical "user experience"or sociological considerations.
A) the tool looks great! B) Will it be able to handle 1,000 entries per day? Or a better question - What will it take to make it work well with 1,000 entries per day? - technically, I'd think one upload (perhaps of just one day's worth of entries each day) would seem to be needed. If each reviewer needs to upload 30,000 images each time they use it, it could be pretty difficult. -technically, can it be set so that new images (or perhaps randomly selected images) are judged by different reviewers. We wouldn't want the same 500 images being at the front of the line that get judged by 10 different people, but the other 29,500 don't get seen at all. -sociologically, how to deal with folks who go through and select the photos they uploaded and rate poorly other very good photos (gaming)? -what happens to photos not rated in cases there are too many photos and not enough reviewers? Are they just left behind? -is "rate as you go" a possibility, or would you have to wait until the end of the contest? -presentation, could you include a box where the local designer (who selects the various options) explains how they work, i.e. what are the mechanics for a photo to get to the next round.
Perhaps these aren't the easiest questions, and maybe some can be ignored for the time being, but I'd think the success of the system will depend on questions like this being addressed.
Sincerely, Pete User:Smallbones
Message: 4 Date: Mon, 19 Aug 2013 16:06:39 +0200 From: Sylvain Boissel sylvainboissel@gmail.com To: Wiki Loves Monuments Photograph Competition wikilovesmonuments@lists.wikimedia.org Subject: Re: [Wiki Loves Monuments] [Jury Tool] Technical Details Message-ID: CAPVch_tAwmOw9XZ0YKPZpT0XDyPOG2qwC-yx0nsFSfHHQe7ydw@mail.gmail.com Content-Type: text/plain; charset="utf-8"
Well, except for the part where my server's PHP version was too old to use Composer, the installation went smoothly, the steps are very clear.
But looking at the config part and I'd appreciate some explanations about what the options mean : how do the rounds system work, and the binary scoring (if two jury members review a file, one saying "good" and the other "bad", what does it do ?), etc.
For this part : Commons category Plain text file
I guess the "plain text file" is a one-shot import ? But what about the "Commons Category" ? Do we have to import everything at once or can the jury make the first round of evaluation "on the fly" during the contest ? In France, we had 27 000 photos to evaluate. In 20 days, it is very short so if the jury has 50 days (assuming they start sorting out the photos the first day of the contest) its better for them.
And I want to thank you for creating the tool, it looks awesome !
Best regards, Sylvain.
Hi Pete,
excellent questions. But I suggest we seperate the technical from the more social ones. Because the social ones mostly depend on what exact setup you choose for your jury process and are very country specific (so probably rather something to ask yourself internally).
I didn't consider myself the usecase scenario yet where the judging would start during the contest - so that is interesting. I automatically assumed that it would only start on Oct 1, which means that you only feed in a single amount of pictures once. If you want to feed it chunks of images over time, I guess a text file input (one every day) works better technically?
The box where the local jury moderator explains things is a nice idea - could perhaps be an intro/help screen?
Lodewijk
2013/8/19 Peter Ekman pdekman@gmail.com
RE: Sylvain's comments and practical "user experience"or sociological considerations.
A) the tool looks great! B) Will it be able to handle 1,000 entries per day? Or a better question
- What will it take to make it work well with 1,000 entries per day?
- technically, I'd think one upload (perhaps of just one day's worth of
entries each day) would seem to be needed. If each reviewer needs to upload 30,000 images each time they use it, it could be pretty difficult. -technically, can it be set so that new images (or perhaps randomly selected images) are judged by different reviewers. We wouldn't want the same 500 images being at the front of the line that get judged by 10 different people, but the other 29,500 don't get seen at all. -sociologically, how to deal with folks who go through and select the photos they uploaded and rate poorly other very good photos (gaming)? -what happens to photos not rated in cases there are too many photos and not enough reviewers? Are they just left behind? -is "rate as you go" a possibility, or would you have to wait until the end of the contest? -presentation, could you include a box where the local designer (who selects the various options) explains how they work, i.e. what are the mechanics for a photo to get to the next round.
Perhaps these aren't the easiest questions, and maybe some can be ignored for the time being, but I'd think the success of the system will depend on questions like this being addressed.
Sincerely, Pete User:Smallbones
Message: 4 Date: Mon, 19 Aug 2013 16:06:39 +0200 From: Sylvain Boissel sylvainboissel@gmail.com To: Wiki Loves Monuments Photograph Competition wikilovesmonuments@lists.wikimedia.org Subject: Re: [Wiki Loves Monuments] [Jury Tool] Technical Details Message-ID: < CAPVch_tAwmOw9XZ0YKPZpT0XDyPOG2qwC-yx0nsFSfHHQe7ydw@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Well, except for the part where my server's PHP version was too old to use Composer, the installation went smoothly, the steps are very clear.
But looking at the config part and I'd appreciate some explanations about what the options mean : how do the rounds system work, and the binary scoring (if two jury members review a file, one saying "good" and the other "bad", what does it do ?), etc.
For this part : Commons category Plain text file
I guess the "plain text file" is a one-shot import ? But what about the "Commons Category" ? Do we have to import everything at once or can the jury make the first round of evaluation "on the fly" during the contest ? In France, we had 27 000 photos to evaluate. In 20 days, it is very short so if the jury has 50 days (assuming they start sorting out the photos the first day of the contest) its better for them.
And I want to thank you for creating the tool, it looks awesome !
Best regards, Sylvain.
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
On Mon, Aug 19, 2013 at 12:50 PM, Peter Ekman pdekman@gmail.com wrote:
B) Will it be able to handle 1,000 entries per day? Or a better question - What will it take to make it work well with 1,000 entries per day?
- technically, I'd think one upload (perhaps of just one day's worth of
entries each day) would seem to be needed. If each reviewer needs to upload 30,000 images each time they use it, it could be pretty difficult.
Well, nothing is really uploaded to the tool, but some database entries (and 1,000 db entries are manageable, probably with some tweaks I have in mind), but I think this question is oriented towards the "rate as you go" mode you mention later, is that right?
-technically, can it be set so that new images (or perhaps randomly selected images) are judged by different reviewers. We wouldn't want the same 500 images being at the front of the line that get judged by 10 different people, but the other 29,500 don't get seen at all.
That is what the configurations for policy of the round are for: With a "All photos are reviewed exactly once" policy, no photo is ever served to two different juries, with a "All photos are reviewed by at least x juries" all photos are also reviewed, and more than once.
-sociologically, how to deal with folks who go through and select the photos they uploaded and rate poorly other very good photos (gaming)?
This would only apply to Open rounds and has no way to be fixed. Invitational rounds, on the other hand, would require juries to be pre-approved by the organizers of the contest, and that means the organizers won't put a jury that uploaded a photo.
-what happens to photos not rated in cases there are too many photos and not enough reviewers? Are they just left behind?
Haven't thought of that. In my small head all photos have to be reviewed.
-is "rate as you go" a possibility, or would you have to wait until the end of the contest?
Good question. The tool is not really designed for that, but could be extended to support this. I'd like to hear other opininos about this.
-presentation, could you include a box where the local designer (who selects the various options) explains how they work, i.e. what are the mechanics for a photo to get to the next round.
That's doable, and shouldn't be too hard. I would only need help with the wording (see the other thread about design).
David E. Narvaez
2013/8/20 David Narvaez david.narvaez@computer.org
On Mon, Aug 19, 2013 at 12:50 PM, Peter Ekman pdekman@gmail.com wrote:
B) Will it be able to handle 1,000 entries per day? Or a better
question -
What will it take to make it work well with 1,000 entries per day?
- technically, I'd think one upload (perhaps of just one day's worth of
entries each day) would seem to be needed. If each reviewer needs to
upload
30,000 images each time they use it, it could be pretty difficult.
Well, nothing is really uploaded to the tool, but some database entries (and 1,000 db entries are manageable, probably with some tweaks I have in mind), but I think this question is oriented towards the "rate as you go" mode you mention later, is that right?
-technically, can it be set so that new images (or perhaps randomly
selected
images) are judged by different reviewers. We wouldn't want the same 500 images being at the front of the line that get judged by 10 different people, but the other 29,500 don't get seen at all.
That is what the configurations for policy of the round are for: With a "All photos are reviewed exactly once" policy, no photo is ever served to two different juries, with a "All photos are reviewed by at least x juries" all photos are also reviewed, and more than once.
-sociologically, how to deal with folks who go through and select the
photos
they uploaded and rate poorly other very good photos (gaming)?
This would only apply to Open rounds and has no way to be fixed. Invitational rounds, on the other hand, would require juries to be pre-approved by the organizers of the contest, and that means the organizers won't put a jury that uploaded a photo.
-what happens to photos not rated in cases there are too many photos and
not
enough reviewers? Are they just left behind?
Haven't thought of that. In my small head all photos have to be reviewed.
Would this be solved if you start at a random point in the review list rather than always at the beginning? Then every jury member would start somewhere else, and the odds of images having zero reviews is minimized.
-is "rate as you go" a possibility, or would you have to wait until the
end
of the contest?
Good question. The tool is not really designed for that, but could be extended to support this. I'd like to hear other opininos about this.
I think if you allow feeding multiple txt files through the contest (taking out doublures) that might work?
-presentation, could you include a box where the local designer (who
selects
the various options) explains how they work, i.e. what are the mechanics
for
a photo to get to the next round.
That's doable, and shouldn't be too hard. I would only need help with the wording (see the other thread about design).
I think if you just present this as an option, you could have this as an empty text box in the configuration, and let the configurer worry about the wording (and language).
A question of my own: what happens if an image gets renamed/deleted. Will the tool be able to handle that?
Lodewijk
On Tue, Aug 20, 2013 at 4:48 AM, Lodewijk lodewijk@effeietsanders.org wrote:
-what happens to photos not rated in cases there are too many photos and not enough reviewers? Are they just left behind?
Haven't thought of that. In my small head all photos have to be reviewed.
Would this be solved if you start at a random point in the review list rather than always at the beginning? Then every jury member would start somewhere else, and the odds of images having zero reviews is minimized.
If we agree that all photos must be reviewed before advancing to the next round, then there are several ways to guarantee all photos are reviewed. The question is whether we want to enforce all photos to be reviewed or not.
-is "rate as you go" a possibility, or would you have to wait until the end of the contest?
Good question. The tool is not really designed for that, but could be extended to support this. I'd like to hear other opininos about this.
I think if you allow feeding multiple txt files through the contest (taking out doublures) that might work?
What would the source of the txt file be? A list of new uploads? If so, then we could add a feature where the tool updates itself with newly added items in a category every day.
-presentation, could you include a box where the local designer (who selects the various options) explains how they work, i.e. what are the mechanics for a photo to get to the next round.
That's doable, and shouldn't be too hard. I would only need help with the wording (see the other thread about design).
I think if you just present this as an option, you could have this as an empty text box in the configuration, and let the configurer worry about the wording (and language).
Ok.
A question of my own: what happens if an image gets renamed/deleted. Will the tool be able to handle that?
Haven't thought of that, good point. This is somewhat related to the rate-as-you-go mode discussed above, so let me think about these and I'll write back with a plan (or no plan ;))
David E. Narvaez
2013/8/20 David Narvaez david.narvaez@computer.org
On Tue, Aug 20, 2013 at 4:48 AM, Lodewijk lodewijk@effeietsanders.org wrote:
-what happens to photos not rated in cases there are too many photos
and
not enough reviewers? Are they just left behind?
Haven't thought of that. In my small head all photos have to be
reviewed.
Would this be solved if you start at a random point in the review list rather than always at the beginning? Then every jury member would start somewhere else, and the odds of images having zero reviews is minimized.
If we agree that all photos must be reviewed before advancing to the next round, then there are several ways to guarantee all photos are reviewed. The question is whether we want to enforce all photos to be reviewed or not.
-is "rate as you go" a possibility, or would you have to wait until
the
end of the contest?
Good question. The tool is not really designed for that, but could be extended to support this. I'd like to hear other opininos about this.
I think if you allow feeding multiple txt files through the contest
(taking
out doublures) that might work?
What would the source of the txt file be? A list of new uploads? If so, then we could add a feature where the tool updates itself with newly added items in a category every day.
Sounds even better - my suggestion was just trying to make your life easier :) But this option should definitely be optional and possible to turn off after a while - I suspect most countries will not start judging before the end of the month, and we don't want pollution of the category trickle through into the jury tool.
-presentation, could you include a box where the local designer (who selects the various options) explains how they work, i.e. what are the
mechanics
for a photo to get to the next round.
That's doable, and shouldn't be too hard. I would only need help with the wording (see the other thread about design).
I think if you just present this as an option, you could have this as an empty text box in the configuration, and let the configurer worry about
the
wording (and language).
Ok.
A question of my own: what happens if an image gets renamed/deleted. Will the tool be able to handle that?
Haven't thought of that, good point. This is somewhat related to the rate-as-you-go mode discussed above, so let me think about these and I'll write back with a plan (or no plan ;))
Also important when you get the images you display directly from commons (or do you download everything) and to keep the links working.
A possible way of dealing with it, is keeping an eye on move- and deletion logs (which are public) and processing that. Not sure how that would work automatically, but the data is there.
David E. Narvaez
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
2013/8/20 Lodewijk:
2013/8/20 David Narvaez:
What would the source of the txt file be? A list of new uploads? If so, then we could add a feature where the tool updates itself with newly added items in a category every day.
If you are simply loading from the files/category, it should be automatic (perhaps too automatic!).
> A question of my own: what happens if an image gets renamed/deleted. Will > the tool be able to handle that? Haven't thought of that, good point. This is somewhat related to the rate-as-you-go mode discussed above, so let me think about these and I'll write back with a plan (or no plan ;))Also important when you get the images you display directly from commons (or do you download everything) and to keep the links working.
A possible way of dealing with it, is keeping an eye on move- and deletion logs (which are public) and processing that. Not sure how that would work automatically, but the data is there.
In my experience the trickiest cases are going back from an image page to the upload log after it was renamed, and an image being splitted (ie. Upload A, reupload A, Delete A, restore A1, move A → B, delete A, restore A2).
Can your take the files from an existing db? It would be nice if it could pull from the db with the list generated by my tools. I wasn't able to find your database schema, though.
On Tue, Aug 20, 2013 at 9:08 AM, Platonides platonides@gmail.com wrote:
Can your take the files from an existing db? It would be nice if it could pull from the db with the list generated by my tools. I wasn't able to find your database schema, though.
Sorry, totally forgot about this question. If you can export your results as a plain text file of image names (without the File: prefix) then you can upload that text file into the tool.
David E. Narvaez
Thanks a lot for the amazing effort here, David!
Lodewijk
2013/8/31 David Narvaez david.narvaez@computer.org
On Tue, Aug 20, 2013 at 9:08 AM, Platonides platonides@gmail.com wrote:
Can your take the files from an existing db? It would be nice if it could pull from the db with the list generated by my tools. I wasn't able to
find
your database schema, though.
Sorry, totally forgot about this question. If you can export your results as a plain text file of image names (without the File: prefix) then you can upload that text file into the tool.
David E. Narvaez
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Yes, indeed. Thanks David!
Although we will in the UK be screening on a daily basis, I would like to avoid having to ask the WMUK IT consultants to update the tool too often, so it would be great if you could post a note here with the latest install/update instructions once you have things in a stable situation that our screeners can work with for at least the first week or so.
Michael
On 31 Aug 2013, at 10:15, Lodewijk wrote:
Thanks a lot for the amazing effort here, David!
Lodewijk
2013/8/31 David Narvaez david.narvaez@computer.org On Tue, Aug 20, 2013 at 9:08 AM, Platonides platonides@gmail.com wrote:
Can your take the files from an existing db? It would be nice if it could pull from the db with the list generated by my tools. I wasn't able to find your database schema, though.
Sorry, totally forgot about this question. If you can export your results as a plain text file of image names (without the File: prefix) then you can upload that text file into the tool.
David E. Narvaez
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
On Tue, Aug 20, 2013 at 7:24 AM, David Narvaez david.narvaez@computer.org wrote:
A question of my own: what happens if an image gets renamed/deleted. Will the tool be able to handle that?
Haven't thought of that, good point. This is somewhat related to the rate-as-you-go mode discussed above, so let me think about these and I'll write back with a plan (or no plan ;))
I just updated the code of the tool to support querying a category daily to update the photos of a round with new uploaded photos. Still doesn't address renaming/deletion but should be usable enough to start scoring next week. The implementation is based on cron jobs. If you do not know what that means it is probably not important because we are hosting most of the tools in Labs and we can take care of that configuration, yet it is useful to know that if your local team wants to try this feature of the tool.
David E. Narvaez
wikilovesmonuments@lists.wikimedia.org