Hi all,
If you are a Wiki Loves Monuments organizer, jury coordinator in the
national level, or a jury member, this email can contain crucial
information for you. Everyone else, you can safely stop here. :)
This email contains some best practices on how to run your local jury
process with regards to criteria, jury membership and process. Finally, it
includes some instructions on how to choose your jury tool.
==Timeline==
You should plan to start the jury process as soon as the competition
ends in your country, this is October 1 for many of you. :) The
international team expects to receive the top 10 photos of your
country no later than October 31 [0-1]. This is a strict deadline due
to international jury process timeline.
==Jury coordinator==
Make sure your local team has one person in charge of jury
coordinations. This person will need to make sure your jury team is in
place and that your jury process starts and ends on time.
==Judging criteria==
While every national competition can choose the judging criteria based
on the specific needs of the country, the international team
recommends the following three judging criteria to be considered in
the country-level jury processes: technical quality, originality, and
usefulness of the photo for Wikipedia. If you are interested to learn
more about these criteria, please
read more about the judging criteria [2].
==Local jury set-up==
National competitions typically have a jury with at least 3 members (the
more photos, the more jury members). The local jury set-up is usually
defined based on the judging criteria you will choose to go with. The
international team recommends that, depending on the number of photos
you expect to have by the end of the contest, you have 1-3 Wikimedians
(maybe those with quality/featured images on Commons), 1-3 people who
are familiar or are experts in the heritage of your country
(especially the monuments), and 1-3 people who are professional or
(quality) amateur photographers. Of course, all jury members should be
excluded from winning prizes awarded by the jury. The national jury
can then nominate maximum 10 photos per country for the international
finale.
Putting the jury team together is local team's responsibility and we
are happy to help you where we can. If you have a hard time finding a
jury member for your team and you are looking for specific
qualifications, please email me off-list. We can't guarantee that we
will find someone for you, but we can guarantee that we will try.
==Jury process==
The jury process on the national level is defined by each country. The
international team recommends the following process:
Round 1: Yes/No or rating
The goal of this round is to lower the number of competing photos to
400-700. If your total number of pictures is less than 500, you can skip
this round.
Jury members are asked to vote yes/no for each photo they are shown
and are requested to stick to a maximum number of ‘yes’ votes (for
example, 500). Based on these votes, a set of 400-700 photos is
selected for the next round. If you have many photos, you may have to
repeat this round one more time to reduce the number of photos in two
steps, especially if you go with Yes/No round (as opposed to rating).
Round 2: Rating
The goal of this round is to come to a selection of the top-50 images.
Jury members are asked to rate/score each image with 1-5 stars. Based
on the average from their votes, the top-50 is selected for the next
round. If there are many pictures with similar scores, the coordinator
can choose to select a top-40, top-60, etc.
Round 2.5: Now you have a list of ~50 photos that will need to go
through the final ranking process. As a jury coordinator, you can take
a few steps here to make your life easier later on:
* Check the license of these photos and make sure they're the correct licenses.
* Check upload time, uploader ID, etc. to make sure you don't spot
something that is against your local rules.
* Do backward checking to make sure to a reasonable extent that the
photo uploaded is the work of the uploader. There are different ways
to do this. [3]
* Show the photos to your jury and ask them if any of them thinks a
photo should be excluded from the final round. Note that you have a
jury with a diverse background and there are rare cases in which one
juror sees major issues with a photo that others don't spot. Giving
your jurors a chance to deliberate and discuss in case a photo needs
to be removed can be helpful.
If you or your jurors spot any issue that results in a photo to be
excluded from the next round, create a Yes/No round with the 50 photos
and remove photos that should be removed by a No vote. Note that these
kind of exclusions are rare.
Round 3: Live meeting or ranking
The goal of this round is to arrive at a final ranking and winning
pictures. If geographically feasible, the jury can meet in person. The
jury tool supports an alternative method: ranking. Each jury member is
asked to rank their favorite 20 photos in order. Based on this
ranking, points are awarded to each photo (20 for the number 1, 19 for
number 2, etc). The total number of points determines the final
result. Jury members are also asked to give a reason for selecting
their top images. You can use this latter information to explain why
the winner was selected, an information that can be useful for your
jury report and press releases.
==Jury report==
We highly recommend that you document every step of your jury process:
how many photos entered each round, how many jurors, how many jurors
voted on each photo, the logic behind any exclusion, etc. This
documentation should ideally be shared with your audience when you
announce the winners of your competition. Remember, jury process is a
very important in Wiki Loves Monuments. Having a sound and transparent
jury process is important to build trust with your current and future
participants.
==Jury tool==
There are quite a few tools available for you to use for your jury
process. [4] The international team develops and maintains Montage [5]
and we recommend that you use this tool unless you have already used
another tool in the past and would like to continue using it. Below
you can find more information about Montage.
==Montage==
The next version of Montage will go live some time before October 1.
In this new version, we have created features and workflows based on
the feedback that we have collected from the users of the tool in
different campaigns: Wiki Loves Monuments, Earth, Africa, and Folk.
There are two features of the tool that are worth pointing out: The
tool is designed and developed in a way that can accommodate the
recommended jury process explained above (yes/no, rating, ranking
features are supported). The developer team will also guarantee to
provide timely support (within 24 hours) starting October 1.
If you're a national jury coordinator or a jury member and want to
test the current version of the tool, please leave a note on the
tool’s discussion page [6]. You should feel free to leave
questions/comments about the tool in the same page. If you are
familiar with GitHub, you can also create issues on GitHub [7] while
testing the tool. Please note that the developers will continue adding
features and smoothing the workflow until the end of this month when
the major release happens. :)
Please sign up for Montage if you haven't already done so and know you
want to use it. [8]
Questions and comments are welcome.
Best,
Lily, on behalf of the international team
[0] For the countries that have a Flickr photowalk, we may need to
coordinate with you for your jury process to start up to a day or two
later than October 1 as we will need time to move photos from Flickr
to Commons (Some walks are organized on September 30). We will work
with you to find a timeline that works with your jury process in this
case.
[1] Unless we have already agreed for a different date with your country.
[2]
http://www.wikilovesmonuments.org/judging-criteria/
[3]
https://ctrlq.org/google/images/
[4]
https://commons.wikimedia.org/wiki/Commons:Jury_tools
[5]
https://commons.wikimedia.org/wiki/Commons:Montage
[6]
https://commons.wikimedia.org/wiki/Commons_talk:Montage
[7]
https://github.com/hatnote/montage/issues
[8]
https://commons.wikimedia.org/wiki/Commons_talk:Montage#Wiki_Loves_Monument…
--
User: LilyOfTheWest