Hi all,
If you are a
Wiki Loves Monuments organizer, jury coordinator in the
national
level, or a jury member, this email can contain crucial
information
for you. Everyone else, you can safely stop here. :)
This email
contains some best practices on how to run your local jury
process
with regards to criteria, jury membership and process. Finally, it
includes
some instructions on how to choose your jury tool.
==Timeline==
You
should plan to start the jury process as soon as the competition
ends
in your country, this is October 1 for many of you. :) The
international
team expects to receive the top 10 photos of your
country no later
than October 31 [0-1]. This is a strict deadline due
to international
jury process timeline.
==Jury coordinator==
Make sure
your local team has one person in charge of jury
coordinations. This
person will need to make sure your jury team is in
place and that
your jury process starts and ends on time.
==Judging criteria==
While
every national competition can choose the judging criteria based
on
the specific needs of the country, the international team
recommends
the following three judging criteria to be considered in
the
country-level jury processes: technical quality, originality, and
usefulness
of the photo for Wikipedia. If you are interested to learn
more
about these criteria, please
read more about the judging criteria
[2].
==Local jury set-up==
National competitions typically
have a jury with at least 3 members (the
more photos, the more jury
members). The local jury set-up is usually
defined based on the
judging criteria you will choose to go with. The
international team
recommends that, depending on the number of photos
you expect to have
by the end of the contest, you have 1-3 Wikimedians
(maybe those
with quality/featured images on Commons), 1-3 people who
are familiar
or are experts in the heritage of your country
(especially the
monuments), and 1-3 people who are professional or
(quality) amateur
photographers. Of course, all jury members should be
excluded from
winning prizes awarded by the jury. The national jury
can then
nominate maximum 10 photos per country for the international
finale.
Putting
the jury team together is local team's responsibility and we
are
happy to help you where we can. If you have a hard time finding a
jury
member for your team and you are looking for specific
qualifications,
please email me off-list. We can't guarantee that we
will find
someone for you, but we can guarantee that we will try.
==Jury
process==
The jury process on the national level is defined by
each country. The
international team recommends the following
process:
Round 1: Yes/No or rating
The goal of this round
is to lower the number of competing photos to
400-700. If your total
number of pictures is less than 500, you can skip
this round.
Jury
members are asked to vote yes/no for each photo they are shown
and
are requested to stick to a maximum number of ‘yes’ votes (for
example,
500). Based on these votes, a set of 400-700 photos is
selected for
the next round. If you have many photos, you may have to
repeat this
round one more time to reduce the number of photos in two
steps,
especially if you go with Yes/No round (as opposed to rating).
Round
2: Rating
The goal of this round is to come to a selection of
the top-50 images.
Jury members are asked to rate/score each image
with 1-5 stars. Based
on the average from their votes, the top-50 is
selected for the next
round. If there are many pictures with similar
scores, the coordinator
can choose to select a top-40, top-60, etc.
Round
2.5: Now you have a list of ~50 photos that will need to go
through
the final ranking process. As a jury coordinator, you can take
a few
steps here to make your life easier later on:
* Check the license of
these photos and make sure they're the correct licenses.
* Check
upload time, uploader ID, etc. to make sure you don't spot
something
that is against your local rules.
* Do backward checking to make sure
to a reasonable extent that the
photo uploaded is the work of the
uploader. There are different ways
to do this. [3]
* Show the
photos to your jury and ask them if any of them thinks a
photo should
be excluded from the final round. Note that you have a
jury with a
diverse background and there are rare cases in which one
juror sees
major issues with a photo that others don't spot. Giving
your jurors a
chance to deliberate and discuss in case a photo needs
to be removed
can be helpful.
If you or your jurors spot any issue that
results in a photo to be
excluded from the next round, create a
Yes/No round with the 50 photos
and remove photos that should be
removed by a No vote. Note that these
kind of exclusions are rare.
Round
3: Live meeting or ranking
The goal of this round is to arrive
at a final ranking and winning
pictures. If geographically feasible,
the jury can meet in person. The
jury tool supports an alternative
method: ranking. Each jury member is
asked to rank their favorite 20
photos in order. Based on this
ranking, points are awarded to each
photo (20 for the number 1, 19 for
number 2, etc). The total number
of points determines the final
result. Jury members are also asked to
give a reason for selecting
their top images. You can use this
latter information to explain why
the winner was selected, an
information that can be useful for your
jury report and press
releases.
==Jury report==
We highly recommend that you
document every step of your jury process:
how many photos entered
each round, how many jurors, how many jurors
voted on each photo, the
logic behind any exclusion, etc. This
documentation should ideally
be shared with your audience when you
announce the winners of your
competition. Remember, jury process is a
very important in Wiki Loves
Monuments. Having a sound and transparent
jury process is important
to build trust with your current and future
participants.
==Jury
tool==
There are quite a few tools available for you to use for
your jury
process. [4] The international team develops and maintains
Montage [5]
and we recommend that you use this tool unless you have
already used
another tool in the past and would like to continue
using it. Below
you can find more information about Montage.
==Montage==
The
next version of Montage will go live some time before October 1.
In
this new version, we have created features and workflows based on
the
feedback that we have collected from the users of the tool in
different
campaigns: Wiki Loves Monuments, Earth, Africa, and Folk.
There
are two features of the tool that are worth pointing out: The
tool is
designed and developed in a way that can accommodate the
recommended
jury process explained above (yes/no, rating, ranking
features are
supported). The developer team will also guarantee to
provide timely
support (within 24 hours) starting October 1.
If you're a
national jury coordinator or a jury member and want to
test the
current version of the tool, please leave a note on the
tool’s
discussion page [6]. You should feel free to leave
questions/comments
about the tool in the same page. If you are
familiar with GitHub,
you can also create issues on GitHub [7] while
testing the tool.
Please note that the developers will continue adding
features and
smoothing the workflow until the end of this month when
the major
release happens. :)
Please sign up for Montage if you haven't
already done so and know you
want to use it. [8]
Questions and
comments are welcome.
Best,
Lily, on behalf of the
international team
[0] For the countries that have a Flickr
photowalk, we may need to
coordinate with you for your jury process
to start up to a day or two
later than October 1 as we will need time
to move photos from Flickr
to Commons (Some walks are organized on
September 30). We will work
with you to find a timeline that works
with your jury process in this
case.
[1] Unless we have already
agreed for a different date with your country.
[2]
http://www.wikilovesmonuments.org/judging-criteria/[3]
https://ctrlq.org/google/images/[4]
https://commons.wikimedia.org/wiki/Commons:Jury_tools[5]
https://commons.wikimedia.org/wiki/Commons:Montage[6]
https://commons.wikimedia.org/wiki/Commons_talk:Montage[7]
https://github.com/hatnote/montage/issues[8]
https://commons.wikimedia.org/wiki/Commons_talk:Montage#Wiki_Loves_Monuments_2017