New subject: Jury criteria, membership, process, and tools for WLM 2016

26 Sep 2016

Hi all,

If you are a WLM organizer, jury coordinator in the national level, or a
jury member, this email can contain crucial information for you. Everyone
else, you can safely stop here. :)

This email contains some best practices on how to run your local jury
process with regards to criteria, jury membership and process. Finally, it
includes some instructions on how to choose your jury tool.

Timeline

You should plan to start the jury process as soon as the competition ends
in your country, this is October 1 for many of you. :) The international
team expects to receive the top 10 photos of your country no later than
October 25.

Judging criteria

While every national competition can choose the judging criteria based on
the specific needs of the country, the international team recommends the
following three judging criteria to be considered in the country-level jury
processes: technical quality, originality, and usefulness of the photo for
Wikipedia. If you are interested to learn more about these criteria, please
read more about the judging criteria
<http://www.wikilovesmonuments.org/judging-criteria/> that the
international jury will be using as explained in this blogpost by Lodewijk.

Local jury set-up

National competitions typically have a jury with at least 3 members (the
more photos, the more jury members). The local jury set-up is usually
defined based on the judging criteria you will choose to go with. The
international team recommends that, depending on the number of photos you
expect to have by the end of the contest, you have 1-3 Wikimedians (maybe
those with quality/featured images on Commons), 1-3 people who are familiar
or are experts in the heritage of your country (especially the monuments),
and 1-3 people who are professional or (quality) amateur photographers. Of
course, all jury members should be excluded from winning prizes awarded by
the jury.  The national jury can then nominate maximum 10 photos per
country for the international finale.

Jury process

The jury process on the national level is defined by each country. The
international team recommends the following process:

Round 1: Yes/No or rating

The goal of this round is to lower the number of competing photos to
400-700. If your total number of pictures is less than 500, you can skip
this round.

Jury members are asked to vote yes/no for each photo they are shown and are
requested to stick to a maximum number of ‘yes’ votes (for example, 500).
Based on these votes, a set of 400-700 photos is selected for the next
round. If you have many photos, you may have to do this round in two steps.

Round 2: Rating

The goal of this round is to come to a selection of the top-50 images. Jury
members are asked to rate/score each image with 1-5 stars. Based on the
average from their votes, the top-50 is selected for the next round. If
there are many pictures with similar scores, the coordinator can choose to
select a top-40, top-60, etc.

Round 3: Live meeting or ranking

The goal of this round is to arrive at a final ranking and winning
pictures. If geographically feasible, the jury can meet in person. The jury
tool supports an alternative method: ranking. Each jury member is asked to
rank their favorite 20 photos in order. Based on this ranking, points are
awarded to each photo (20 for the number 1, 19 for number 2, etc). The
total number of points determines the final result. Jury members are also
asked to give a reason for selecting their top images. You can use this
latter information to explain explain why the winner was selected, an
information that can be useful for your jury report and press releases.

Jury tool

Unless you have a very small number of submissions (less than 100), it's
usually efficient for your jury to use one of the available tools
<https://commons.wikimedia.org/wiki/Commons:Jury_tools> for judging the
photos. The international team would like to bring your attention to two
tools: ‘Montage’ and ‘WLX’. Below you can find more information about these
two tools. We suggest that you look into these tools now, and make your
choice soon.

Montage

Montage is the new tool that has been developed this year by volunteers and
per the request by the WLM international team. There are two features of
the tool that are worth pointing out: The tool is designed and developed in
a way that can accommodate the recommended jury process explained above
(yes/no, rating, ranking features are supported). The developer team will
also guarantee to provide timely support (within 24 hours) starting October
1.

If you're a national jury coordinator or a jury member and want to test the
current version of the tool, please leave a note on the tool’s discussion
page <https://commons.wikimedia.org/wiki/Commons_talk:Montage>. If you want
to have a 1:1 conversation with one of the developers, please leave a
message using the same link. You should feel free to leave
questions/comments about the tool in the same page. If you are familiar
with GitHub, you can also create issues on GitHub
<https://github.com/hatnote/montage/issues>. Please note that the
developers will continue adding features and smoothing the workflow until
October 1, what you will test now will give you a sense of the basics you
should expect. :)

What is essential for you to be able to start the jury process on October 1
using Montage is the Wikimedia usernames of your jury members (if they
don't have an account, please ask them to open one). If you decide to go
with Montage, please gather this information sooner rather than later.

WLX

You can learn more about the tool here
<https://commons.wikimedia.org/wiki/Commons:WLX_Jury_Tool>. The tool has
been used by many countries as part of WLM and WLE contests in the past
years. If you're interested to use WLX, please contact Ilya (WLX's tool
developer) in the tool's talk page
<https://commons.wikimedia.org/wiki/Commons_talk:WLX_Jury_Tool>.

If you have any questions or comments, please don't hesitate to ask them
here or off-list.

Best,

Leila

-- 
User: LilyOfTheWest