Hi Katie,

I'm particularly referring to the national processes here. I saw someone mention that India might get 100.000 images... (worst case?) and that got me thinking ;) Every country will have a slightly different process.

I think it is a nice approach to have people rate. If we can just then add access options (access for all, or for a specified group). I don't have a clou if WP is the best option. But bear in mind the interface should be able to hold up to 100.000 images. Ideally:

* It would allow organizers to simply insert a list of commons file names for the photos * It would allow organizers to specify who should have access to the voting interface * It would allow organizers to have an overview of who voted, how often * it would allow the user to show X images on a page, on a specified size and rank each of them * It would allow the user to sort based on name, date, number of rankings and current ranking * It would include a simple option (clicking) to enlarge the picture, and include a link to the detail page on commons.

Does that make sense?

Lodewijk

2012/6/21 aude aude.wiki@gmail.com

On Thu, Jun 21, 2012 at 12:21 PM, Lodewijk lodewijk@effeietsanders.orgwrote:

...

Hi all,

is there anyone who would be willing to help set up a jury process tool that can be used in multiple countries? I'm looking for a software tool that makes juring more easy than going through commons categories. Would be happy to brainstorm if someone is willing to work it out.

We have a technical volunteer "intern" helping us in DC and this is something he wants to work on.

I'm not exactly sure how the process worked last year, but we're thinking either integrating something into the WordPress site so that people can browse photos (pulled from Commons, like InstantCommons) and rate them. A public voting phase could help narrow the selection for the jury, which could also make use of the tool.

Thoughts? suggestions? brilliant ideas?

Cheers, Katie

...

Thanks!

Lodewijk

---

Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.eu

-- Board member, Wikimedia District of Columbia http://wikimediadc.org @wikimediadc / @wikimania2012

---

Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.eu

No voter will watch all images (or maybe not even a big part), so I think a public voting system will only work with lists up to a maximum of 500 pictures. Watching 10.000 pictures (and some will have 50k) just takes quite some time (1-2 days). Last year I scrolled to the categories (looking 200 pictures in 10 seconds), every picture that I liked or jumped out was then opened in a new tab by me, and if I liked the picture I copied the link and added it to a page with wikistyle pictures on big size (700px) this page I watched in preview edit mode and this way narrowed down the selection. I think a basic but strong to tool to speed this up very much is to skip the copy the link and place it somewhere else part of that and make that a simple mouseclick. so you start with a huge category, you go throught that page by page (200 pictures a time), and select those that you like (the add pictures to list mode). These will then show in a list (where you can pick the pixel size), there you have a delete mode to delete pictures not good enough. The only thing is, no saving on Commons should be involved because participants/everybody shouldn't see what the juries select (at least not to soon). You can also consider voting, but I think that only voting for all the 10-50k pictures in some country will not work at all, because then you at least need like 20 people who watch all pictures (and are a bit neutral (eg not vote own pictures or ask friends)), if people only look to the most voted pictures, or some 500 pictures picked some way (by name etc) then very good pictures will be missed. You just need some neutral people to watch everything and make selections. After that you might choose to let the public vote on these selections (say the selections consist of 50-500 pictures. I simply think that getting a rating on every image will be impossible, that will take people days. Mvg, Bas From: lodewijk@effeietsanders.org Date: Thu, 21 Jun 2012 18:47:11 +0200 To: wikilovesmonuments@lists.wikimedia.org Subject: Re: [Wiki Loves Monuments] jury process - easy and neat?

Hi Katie, I'm particularly referring to the national processes here. I saw someone mention that India might get 100.000 images... (worst case?) and that got me thinking ;) Every country will have a slightly different process.

I think it is a nice approach to have people rate. If we can just then add access options (access for all, or for a specified group). I don't have a clou if WP is the best option. But bear in mind the interface should be able to hold up to 100.000 images. Ideally:

* It would allow organizers to simply insert a list of commons file names for the photos* It would allow organizers to specify who should have access to the voting interface* It would allow organizers to have an overview of who voted, how often

* it would allow the user to show X images on a page, on a specified size and rank each of them* It would allow the user to sort based on name, date, number of rankings and current ranking* It would include a simple option (clicking) to enlarge the picture, and include a link to the detail page on commons.

Does that make sense? Lodewijk 2012/6/21 aude aude.wiki@gmail.com

On Thu, Jun 21, 2012 at 12:21 PM, Lodewijk lodewijk@effeietsanders.org wrote:

Hi all, is there anyone who would be willing to help set up a jury process tool that can be used in multiple countries? I'm looking for a software tool that makes juring more easy than going through commons categories. Would be happy to brainstorm if someone is willing to work it out.

We have a technical volunteer "intern" helping us in DC and this is something he wants to work on. I'm not exactly sure how the process worked last year, but we're thinking either integrating something into the WordPress site so that people can browse photos (pulled from Commons, like InstantCommons) and rate them. A public voting phase could help narrow the selection for the jury, which could also make use of the tool.

Thoughts? suggestions? brilliant ideas? Cheers,Katie

Thanks! Lodewijk

_______________________________________________

Wiki Loves Monuments mailing list

WikiLovesMonuments@lists.wikimedia.org

https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments

http://www.wikilovesmonuments.eu

Bas vb, 21/06/2012 19:59:

Last year I scrolled to the categories (looking 200 pictures in 10 seconds), every picture that I liked or jumped out was then opened in a new tab by me, and if I liked the picture I copied the link and added it to a page with wikistyle pictures on big size (700px) this page I watched in preview edit mode and this way narrowed down the selection.

I think a basic but strong to tool to speed this up very much is to skip the copy the link and place it somewhere else part of that and make that a simple mouseclick.

Such a tool exists already, it's cat-a-lot with a (temporary) sub- or parent category. You said it shouldn't involve saving on Commons, but I don't think transparency here is a problem: rather, given the number of images and the high chances of mistakes, it could be an asset; more likely, nobody will bother to follow closely.

Nemo

On Thu, Jun 21, 2012 at 8:59 PM, Bas vb wrote:

No voter will watch all images (or maybe not even a big part), so I think a public voting system will only work with lists up to a maximum of 500 pictures. Watching 10.000 pictures (and some will have 50k) just takes quite some time (1-2 days).

Last year I scrolled to the categories (looking 200 pictures in 10 seconds), every picture that I liked or jumped out was then opened in a new tab by me, and if I liked the picture I copied the link and added it to a page with wikistyle pictures on big size (700px) this page I watched in preview edit mode and this way narrowed down the selection.

Something like that is what we used for the jury pre-screening, we had the jury members to look at the category pages as thumbnails and make a selection from there. The links to individual image pages were collected and we produced the final voting selection with a reasonable amount of images. More people doing this, and the selection will be balanced.

I think a basic but strong to tool to speed this up very much is to skip the copy the link and place it somewhere else part of that and make that a simple mouseclick.

I had the people making the selection copy the image URL from the address bar copy and paste it into a text file, send it when ready. A web app can be a helpful replacement for that, but keep in mind one aspect: jury members may NOT be regular Wikipedia contributors and NOT have an account, so it have to 1. work without login and 2. keep their work together somehow.

so you start with a huge category, you go throught that page by page (200 pictures a time), and select those that you like (the add pictures to list mode). These will then show in a list (where you can pick the pixel size), there you have a delete mode to delete pictures not good enough. The only thing is, no saving on Commons should be involved because participants/everybody shouldn't see what the juries select (at least not to soon).

Oh, I forgot the most interesting part.

These tools are being migrated to Wordpress, so I believe pretty soon we will have WP plugins for all our tools.

Best, -NT

Em 24-06-2012 21:06, Nuno Tavares escreveu:

Greetings all,

We had such a tool running on our website. Briefly:

• We create tokens for each juri;
• We randomly assign images for each token (each will get TOTAL/TOKENS

images approx.), putting them on a "bucket level 1";

• Then, the token is sent to the juri person, which will use it to

browse through the gallery of his assigned photos, and the selection process begins.

• The selection process consists in 2 passes:

-- The juri person either "promotes" interesting images to "bucket 2" or "demotes" uninteresting images to "bucket 0" (to mark them as viewed). -- Most probably, in the end the juri person has selected too much images for bucket 2, so the last step is to demote images from bucket 2 to 1 again.

In the end, the bucket 2 should have the number we established (50) for him to have present on the juri presencial meeting.

The tool was further extended for helping during the juri presencial meeting: people gather somewhere, and the tool merges the 50 selected photos from each, and then a voting mechanism is due: each juri present will assign a classification.

At the end, the selection list is downloaded (for mobility) and ordered to clear out exequo's (each juri will change his vote according to the discussion).

You can have a "status" from last year, here:

http://wlm2011.ufp.wikimedia.pt/tools/juri/status.php

And I believe you can try things (this is a "running copy" of the old site):

http://wlm2011.ufp.wikimedia.pt/tools/juri/?token=a5e88407-be35-11e1-a7d1-ea...

As for the presencial meeting' extensions, I'll have to dig where did I put them .

-NT

Em 22-06-2012 07:11, Nicu Buculei escreveu:

...

On Thu, Jun 21, 2012 at 8:59 PM, Bas vb wrote:

...

No voter will watch all images (or maybe not even a big part), so I think a public voting system will only work with lists up to a maximum of 500 pictures. Watching 10.000 pictures (and some will have 50k) just takes quite some time (1-2 days).

Last year I scrolled to the categories (looking 200 pictures in 10 seconds), every picture that I liked or jumped out was then opened in a new tab by me, and if I liked the picture I copied the link and added it to a page with wikistyle pictures on big size (700px) this page I watched in preview edit mode and this way narrowed down the selection.

Something like that is what we used for the jury pre-screening, we had the jury members to look at the category pages as thumbnails and make a selection from there. The links to individual image pages were collected and we produced the final voting selection with a reasonable amount of images. More people doing this, and the selection will be balanced.

...

I think a basic but strong to tool to speed this up very much is to skip the copy the link and place it somewhere else part of that and make that a simple mouseclick.

I had the people making the selection copy the image URL from the address bar copy and paste it into a text file, send it when ready. A web app can be a helpful replacement for that, but keep in mind one aspect: jury members may NOT be regular Wikipedia contributors and NOT have an account, so it have to 1. work without login and 2. keep their work together somehow.

...

so you start with a huge category, you go throught that page by page (200 pictures a time), and select those that you like (the add pictures to list mode). These will then show in a list (where you can pick the pixel size), there you have a delete mode to delete pictures not good enough. The only thing is, no saving on Commons should be involved because participants/everybody shouldn't see what the juries select (at least not to soon).

---

Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.eu

Platonides,

Well, honestly, because no one cared to do it for mediawiki :-)

Alas, it's not a CMS per se (nor WP, though..), but usability in WP is far better than MW's.

The tools should be easy to port (it would be good to do it as Extensions), if you care to do it..

-NT

Em 24-06-2012 21:29, Platonides escreveu:

On 24/06/12 22:10, Nuno Tavares wrote:

...

Oh, I forgot the most interesting part.

These tools are being migrated to Wordpress, so I believe pretty soon we will have WP plugins for all our tools.

Best, -NT

Why wordpress? Wouldn't mediawiki integration be better?

---

Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.eu

Andre,

In that case, the image-to-token assignment phase can repeat images for multiple tokens, that's possible as of now, and it is really up to you.

In fact, one of the tokens (those I showed below) was for Wikimedia Portugal, which got the whole lot of images, while the token was shared between our mailing-list (with a gentlemen agreement that noone would demote images already placed to bucket 2). That was the "Wikimedia Portugal"' selection, and it was obviously undefended on the presencial meeting.

-NT

Em 24-06-2012 21:30, Andre Koopal escreveu:

Hi Nuno,

I principle splitting the pictures over the jury-members is not wrong, but that should be a decision of the jury. If you have a mixed jury, it makes more sense that everybody goes over all pictures. So the dividing over the jury members should be more flexible. I can also imagine to assign each picture random to 2 jury members.

For the rest this indeeds sounds what I was describing, I probably was inspired :-)

Regards,

Andre

On Sun, Jun 24, 2012 at 09:06:21PM +0100, Nuno Tavares wrote:

...

Greetings all,

We had such a tool running on our website. Briefly:

• We create tokens for each juri;
• We randomly assign images for each token (each will get

TOTAL/TOKENS images approx.), putting them on a "bucket level 1";

• Then, the token is sent to the juri person, which will use it to

browse through the gallery of his assigned photos, and the selection process begins.

• The selection process consists in 2 passes:

-- The juri person either "promotes" interesting images to "bucket 2" or "demotes" uninteresting images to "bucket 0" (to mark them as viewed). -- Most probably, in the end the juri person has selected too much images for bucket 2, so the last step is to demote images from bucket 2 to 1 again.

In the end, the bucket 2 should have the number we established (50) for him to have present on the juri presencial meeting.

The tool was further extended for helping during the juri presencial meeting: people gather somewhere, and the tool merges the 50 selected photos from each, and then a voting mechanism is due: each juri present will assign a classification.

At the end, the selection list is downloaded (for mobility) and ordered to clear out exequo's (each juri will change his vote according to the discussion).

You can have a "status" from last year, here:

http://wlm2011.ufp.wikimedia.pt/tools/juri/status.php

And I believe you can try things (this is a "running copy" of the old site):

http://wlm2011.ufp.wikimedia.pt/tools/juri/?token=a5e88407-be35-11e1-a7d1-ea...

As for the presencial meeting' extensions, I'll have to dig where did I put them .

-NT

Em 22-06-2012 07:11, Nicu Buculei escreveu:

...

On Thu, Jun 21, 2012 at 8:59 PM, Bas vb wrote:

...

No voter will watch all images (or maybe not even a big part), so I think a public voting system will only work with lists up to a maximum of 500 pictures. Watching 10.000 pictures (and some will have 50k) just takes quite some time (1-2 days).

Last year I scrolled to the categories (looking 200 pictures in 10 seconds), every picture that I liked or jumped out was then opened in a new tab by me, and if I liked the picture I copied the link and added it to a page with wikistyle pictures on big size (700px) this page I watched in preview edit mode and this way narrowed down the selection.

Something like that is what we used for the jury pre-screening, we had the jury members to look at the category pages as thumbnails and make a selection from there. The links to individual image pages were collected and we produced the final voting selection with a reasonable amount of images. More people doing this, and the selection will be balanced.

...

I think a basic but strong to tool to speed this up very much is to skip the copy the link and place it somewhere else part of that and make that a simple mouseclick.

I had the people making the selection copy the image URL from the address bar copy and paste it into a text file, send it when ready. A web app can be a helpful replacement for that, but keep in mind one aspect: jury members may NOT be regular Wikipedia contributors and NOT have an account, so it have to 1. work without login and 2. keep their work together somehow.

...

so you start with a huge category, you go throught that page by page (200 pictures a time), and select those that you like (the add pictures to list mode). These will then show in a list (where you can pick the pixel size), there you have a delete mode to delete pictures not good enough. The only thing is, no saving on Commons should be involved because participants/everybody shouldn't see what the juries select (at least not to soon).

---

Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.eu

---

Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.eu

Hi Sylvain,

Sorry for not replying sooner, I missed your message...

The images were shown live, fetched directly from Commons, which means the juri would need Internet connection. Only for the presential meeting, as I tend to be extremely aware of [[Murphy's law]], I put a script to download the pre-selection (triage of the juri) so we wouldn't depend on the Internet - the voting mechanism was running on my laptop.

The full list was fetched from the statistics module of the API, IIRC: http://toolserver.org/~ntavares/patrimonio/api/

The random assigments were performed in October 1st, yes.

Best, -NT

Em 20-07-2012 10:48, Sylvain Boissel escreveu:

Hi Nuno,

Your tool looks really interesting !

How do the images got into the tool ? Do you provide a full list on October 1st ? Or are the new pictures assigned to jury members on the fly as soon as they are posted to Commons ?

Best regards, Sylvain.

2012/6/24 Nuno Tavares <nuno.tavares@wikimedia.pt mailto:nuno.tavares@wikimedia.pt>

Greetings all,

We had such a tool running on our website. Briefly:

* We create tokens for each juri;
* We randomly assign images for each token (each will get
TOTAL/TOKENS images approx.), putting them on a "bucket level 1";
* Then, the token is sent to the juri person, which will use it to
browse through the gallery of his assigned photos, and the selection
process begins.

* The selection process consists in 2 passes:
-- The juri person either "promotes" interesting images to "bucket
2" or "demotes" uninteresting images to "bucket 0" (to mark them as
viewed).
-- Most probably, in the end the juri person has selected too much
images for bucket 2, so the last step is to demote images from
bucket 2 to 1 again.

In the end, the bucket 2 should have the number we established (50)
for him to have present on the juri presencial meeting.

The tool was further extended for helping during the juri presencial
meeting: people gather somewhere, and the tool merges the 50
selected photos from each, and then a voting mechanism is due: each
juri present will assign a classification.

At the end, the selection list is downloaded (for mobility) and
ordered to clear out exequo's (each juri will change his vote
according to the discussion).

You can have a "status" from last year, here:

http://wlm2011.ufp.wikimedia.__pt/tools/juri/status.php
<http://wlm2011.ufp.wikimedia.pt/tools/juri/status.php>

And I believe you can try things (this is a "running copy" of the
old site):

http://wlm2011.ufp.wikimedia.__pt/tools/juri/?token=a5e88407-__be35-11e1-a7d1-eae4baf4981c
<http://wlm2011.ufp.wikimedia.pt/tools/juri/?token=a5e88407-be35-11e1-a7d1-eae4baf4981c>


As for the presencial meeting' extensions, I'll have to dig where
did I put them .


-NT


Em 22-06-2012 07:11, Nicu Buculei escreveu:

    On Thu, Jun 21, 2012 at 8:59 PM, Bas vb  wrote:

        No voter will watch all images (or maybe not even a big
        part), so I think a
        public voting system will only work with lists up to a
        maximum of 500
        pictures. Watching 10.000 pictures (and some will have 50k)
        just takes quite
        some time (1-2 days).

        Last year I scrolled to the categories (looking 200 pictures
        in 10 seconds),
        every picture that I liked or jumped out was then opened in
        a new tab by me,
        and if I liked the picture I copied the link and added it to
        a page with
        wikistyle pictures on big size (700px) this page I watched
        in preview edit
        mode and this way narrowed down the selection.


    Something like that is what we used for the jury pre-screening,
    we had
    the jury members to look at the category pages as thumbnails and
    make
    a selection from there. The links to individual image pages were
    collected and we produced the final voting selection with a
    reasonable
    amount of images. More people doing this, and the selection will be
    balanced.

        I think a basic but strong to tool to speed this up very
        much is to skip the
        copy the link and place it somewhere else part of that and
        make that a
        simple mouseclick.


    I had the people making the selection copy the image URL from the
    address bar copy and paste it into a text file, send it when
    ready. A
    web app can be a helpful replacement for that, but keep in mind one
    aspect: jury members may NOT be regular Wikipedia contributors
    and NOT
    have an account, so it have to 1. work without login and 2. keep
    their
    work together somehow.

        so you start with a huge category, you go throught that page
        by page (200
        pictures a time), and select those that you like (the add
        pictures to list
        mode). These will then show in a list (where you can pick
        the pixel size),
        there you have a delete mode to delete pictures not good
        enough. The only
        thing is, no saving on Commons should be involved because
        participants/everybody shouldn't see what the juries select
        (at least not to
        soon).



_________________________________________________
Wiki Loves Monuments mailing list
WikiLovesMonuments@lists.__wikimedia.org
<mailto:WikiLovesMonuments@lists.wikimedia.org>
https://lists.wikimedia.org/__mailman/listinfo/__wikilovesmonuments
<https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments>
http://www.wikilovesmonuments.__eu <http://www.wikilovesmonuments.eu>

-- Sylvain Boissel Chargé de mission communauté et technologie de Wikimédia France tél 07.62.93.42.02 - email sylvain.boissel@wikimedia.fr mailto:sylvain.boissel@wikimedia.fr - twitter @sboissel https://twitter.com/#!/sboissel

/Imaginez un monde où chaque personne sur la planète aurait librement accès à la totalité du savoir humain. C'est notre engagement. Aidez Wikimedia France à en faire une réalité https://dons.wikimedia.fr./ www.wikimedia.fr http://www.wikimedia.fr/

---

Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org

On Sun, Aug 5, 2012 at 8:07 PM, Lodewijk lodewijk@effeietsanders.org wrote:

Bumping this thread up, as I'm getting more and more questions about jury tools.

Is someone able to take the work that Nuno has done already, and rework it into a tool that allows easy juring without distributing them over jury members? Also, a question I got several times is to allow infinite jury members, basically to make a community round easier.

Best,

Lodewijk

IMHO you would need dedication from such juries to review at least N images, with N being a big number, usually either the full image list or a big share of them. Fair ranking of many juries with small votes in non-trivial cases looks hard, so although support for "infinite jury members" should be simple (with the juries looking at many photos), I don't think it would solve making a community round.

On Tue, Aug 7, 2012 at 1:01 PM, Lodewijk lodewijk@effeietsanders.org wrote:

My idea of making the number of jury members in a particular round infinite, was that we could hand out a jury token to each community member easily, and let them process 100-1000 images. If enough people do that, and if the distribution is either random or based on the pictures that have the least votes, that should give a somewhat (not perfect) workflow.

Hope that makes sense,

Lodewijk

I stand on my point that it would be hard. :-) Not really in the tool supporting so many "juries", which would be simple, but in processing that. You could easily augment the data points by storing which images where viewed by a single user and if it was +1, -1, or skipped. We could also assume that all members are honest wikimedians and nobody is trying to game to contest. You end up with a pool of images ranked (eg. 1-10) by 1,000 different users. How do you get the top-10/100/500 images? The image some gave 10 points to, would barely have received 5 by others, and viceversa...

On Tue, Aug 7, 2012 at 6:57 PM, Lodewijk lodewijk@effeietsanders.org wrote:

Yeah, it won't be easy :) But with big numbers it might be the only way. One possible way is to only allow two scores: 1 or 0 (and skip). And then take the average instead of the sum. That way you should be able to get the most wonderful images on top - we don't need to rank everything, but should only make sure that the top-25 images are within the selection of 500.

Lodewijk

1 or 0 would be worse. You have ranked 980 photos and now find out the greatest one that shadows all the previous ones, would you just give it a 1 like hundreds before? I think you should offer 1-to-10 scores, freeform even, allowing for a 12 or 5.768 if the user so wishes. Then you can normalize them before agregating. Sum vs average is probably similar. If you really have infinite reviews for each image, it should be about right (each image would have been served to the same porcentage of inclusionists as deletionists), but I don't think we would have an infinite so big. :-)

Hi guys,

I'm not really sure if we're still talking WMPT "voting" tool, I'm assuming it is.

My comments go inline.

Em 07-08-2012 20:55, maarten deneckere escreveu:

Hey all,

my opinion, based on the 7000 images last year in Belgium.

1/ some preprocessing. Remove images that are too small, have no emailaddress linked to their account, they cannot participate, ...

I think I forgot to mention... I actually did a script to filter out some images that didn't match a minimum quality. I found that they were not many, though, so I'm not sure if I even used it or not.

2/ User generated voting. Two types of votes:

• *failed images* (we need to define and explain this very well!) Three "-1" mean the image is removed from the contest. This cleans up the list of images, so people voting at the end will see less images then the first ones. This greatly improves the voting.

This step was actually done by the community: we had a group of volunteers "validating" directly in Commons with a Javascript tool which images were "accepted" to the contest (visual quality issues like blurring, distortion, etc would require too much time to develop).

• *favourite images of the voter*. Every selected image is giving 1/N points (N being the number of votes giving by this users*). People who select a limited amount of images will impact one image more then somebody who selects almost all images.

That's why I think it's important to set a "maximum" number of images to leave in Bucket 2 (consider this bucket as the voter's "finalists"), and leave that sorting responsibility to the "voter".

Bear in mind that I find "voter" and "voting" misleading. Voting is typically assigned to "election" (the winner is who gets more votes) and that's not really the case in Phase 1 (like described before) which is simply a "pre-selection".

Regards, -NT

3/ The final selection happens by the judges (you cannot replace this by user voting). They only see the best 100-200 images.

Regards, Maarten

• for the mathematicians, I would keep 10 < N < 100, to avoid strange

results when people only select one image, and to give more credit to people selecting more than 100 images.

2012/8/7 Lodewijk <lodewijk@effeietsanders.org mailto:lodewijk@effeietsanders.org>

Yeah, it won't be easy :) But with big numbers it might be the only
way. One possible way is to only allow two scores: 1 or 0 (and
skip). And then take the average instead of the sum. That way you
should be able to get the most wonderful images on top - we don't
need to rank everything, but should only make sure that the top-25
images are within the selection of 500.

Lodewijk


2012/8/7 Platonides <platonides@gmail.com <mailto:platonides@gmail.com>>

    On Tue, Aug 7, 2012 at 1:01 PM, Lodewijk
    <lodewijk@effeietsanders.org
    <mailto:lodewijk@effeietsanders.org>> wrote:
     > My idea of making the number of jury members in a particular
    round infinite,
     > was that we could hand out a jury token to each community
    member easily, and
     > let them process 100-1000 images. If enough people do that,
    and if the
     > distribution is either random or based on the pictures that
    have the least
     > votes, that should give a somewhat (not perfect) workflow.
     >
     > Hope that makes sense,
     >
     > Lodewijk

    I stand on my point that it would be hard. :-)
    Not really in the tool supporting so many "juries", which would be
    simple, but in processing that.
    You could easily augment the data points by storing which images
    where
    viewed by a single user and if it was +1, -1, or skipped. We could
    also assume that all members are honest wikimedians and nobody is
    trying to game to contest.
    You end up with a pool of images ranked (eg. 1-10) by 1,000
    different users.
    How do you get the top-10/100/500 images?
    The image some gave 10 points to, would barely have received 5 by
    others, and viceversa...

    _______________________________________________
    Wiki Loves Monuments mailing list
    WikiLovesMonuments@lists.wikimedia.org
    <mailto:WikiLovesMonuments@lists.wikimedia.org>
    https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
    http://www.wikilovesmonuments.org



_______________________________________________
Wiki Loves Monuments mailing list
WikiLovesMonuments@lists.wikimedia.org
<mailto:WikiLovesMonuments@lists.wikimedia.org>
https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
http://www.wikilovesmonuments.org

---

Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org

Of course if you only use this tool as a means to get the best 1% of the pictures... it doesn't have to be as precise. Just have the final selection in a second round, with zero scores - then vote again for those or have a discussion on the mailing list.

Personally, I think every country can decide its own process - so lets make the tool somewhat dynamic, but not shoot down major options. Last year Russia had a public voting, and it seems they were happy with it. I'm no big fan of it, but who am I to stop it? :)

Lodewijk

2012/6/22 Platonides platonides@gmail.com

On 21/06/12 18:29, aude wrote:

...

We have a technical volunteer "intern" helping us in DC and this is something he wants to work on.

I'm not exactly sure how the process worked last year, but we're thinking either integrating something into the WordPress site so that people can browse photos (pulled from Commons, like InstantCommons) and rate them. A public voting phase could help narrow the selection for the jury, which could also make use of the tool.

Thoughts? suggestions? brilliant ideas?

Each instance used its own system. Nuno made a web application for voting, from which you could start. There is also a toolserver tool which facilitates downloading all the files from a category. Last year we provided zips with all the images which our jury downloaded. Then they chose those they deemed the best, which went to a second round. If you're joining the jury in one room (with multiple computers), it may be simpler if they are provided directly a copy of the images in an external drive. Remember that not all your jury members will be tech-savvy.

If I were designing such system, I'd make it an interface where the jury would go giving a mark from 1 to 10 to each photo. But not actually restricted to that. So after 200 photos, it could give a 12 if needed (instead of recalculating all previous votes), or even provide marks with decimal points. The interface itself would be just a (zoomable) gallery of the photos that he didn't review yet (plus auxiliar pages, to view the best ranked by you, change a vote, etc.).

What would be interesting is that they could be using it from the first day, so instead of reviewing all files after the competition closes, they could keep up with the upload rate. As far as a 10 given on Sep 1st is the same as one given on 30th Sep (which is easy by things like moving the bar higher up to a 14), it'd be equivalent. You then fetch the N most ranked from each member to next round, so different scales aren't a problem.

As for a public voting, I don't think it would work. You would need each person to review a significant number of files, otherwise the noise given by each different reviewer (a 10 by me could be an 8 by you). And you won't be getting volunteers to review thousands of photos. Only the jury will do that, because they agreed to. (Obviously, anyone is able to volunteer to be jury. We were discussing on wlm-iberconf ml giving a jury for another country and getting one).

What I had thought as a possibility for involving the public was to allow it to choose a number of photos that pass to the next round (just as each jury does), thus ensuring they get attention. But that won't the jury task of having to view all of them.

Regards

---

Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.eu