Hi all,

If you are a WLM organizer, jury coordinator in the national level, or a jury member, this email can contain crucial information for you. Everyone else, you can safely stop here. :)

This email contains some best practices on how to run your local jury process with regards to criteria, jury membership and process. Finally, it includes some instructions on how to choose your jury tool.

Timeline

You should plan to start the jury process as soon as the competition ends in your country, this is October 1 for many of you. :) The international team expects to receive the top 10 photos of your country no later than October 25.

Judging criteria

While every national competition can choose the judging criteria based on the specific needs of the country, the international team recommends the following three judging criteria to be considered in the country-level jury processes: technical quality, originality, and usefulness of the photo for Wikipedia. If you are interested to learn more about these criteria, please read more about the judging criteria http://www.wikilovesmonuments.org/judging-criteria/ that the international jury will be using as explained in this blogpost by Lodewijk.

Local jury set-up

National competitions typically have a jury with at least 3 members (the more photos, the more jury members). The local jury set-up is usually defined based on the judging criteria you will choose to go with. The international team recommends that, depending on the number of photos you expect to have by the end of the contest, you have 1-3 Wikimedians (maybe those with quality/featured images on Commons), 1-3 people who are familiar or are experts in the heritage of your country (especially the monuments), and 1-3 people who are professional or (quality) amateur photographers. Of course, all jury members should be excluded from winning prizes awarded by the jury. The national jury can then nominate maximum 10 photos per country for the international finale.

Jury process

The jury process on the national level is defined by each country. The international team recommends the following process:

Round 1: Yes/No or rating

The goal of this round is to lower the number of competing photos to 400-700. If your total number of pictures is less than 500, you can skip this round.

Jury members are asked to vote yes/no for each photo they are shown and are requested to stick to a maximum number of ‘yes’ votes (for example, 500). Based on these votes, a set of 400-700 photos is selected for the next round. If you have many photos, you may have to do this round in two steps.

Round 2: Rating

The goal of this round is to come to a selection of the top-50 images. Jury members are asked to rate/score each image with 1-5 stars. Based on the average from their votes, the top-50 is selected for the next round. If there are many pictures with similar scores, the coordinator can choose to select a top-40, top-60, etc.

Round 3: Live meeting or ranking

The goal of this round is to arrive at a final ranking and winning pictures. If geographically feasible, the jury can meet in person. The jury tool supports an alternative method: ranking. Each jury member is asked to rank their favorite 20 photos in order. Based on this ranking, points are awarded to each photo (20 for the number 1, 19 for number 2, etc). The total number of points determines the final result. Jury members are also asked to give a reason for selecting their top images. You can use this latter information to explain explain why the winner was selected, an information that can be useful for your jury report and press releases.

Jury tool

Unless you have a very small number of submissions (less than 100), it's usually efficient for your jury to use one of the available tools https://commons.wikimedia.org/wiki/Commons:Jury_tools for judging the photos. The international team would like to bring your attention to two tools: ‘Montage’ and ‘WLX’. Below you can find more information about these two tools. We suggest that you look into these tools now, and make your choice soon.

Montage

Montage is the new tool that has been developed this year by volunteers and per the request by the WLM international team. There are two features of the tool that are worth pointing out: The tool is designed and developed in a way that can accommodate the recommended jury process explained above (yes/no, rating, ranking features are supported). The developer team will also guarantee to provide timely support (within 24 hours) starting October 1.

If you're a national jury coordinator or a jury member and want to test the current version of the tool, please leave a note on the tool’s discussion page https://commons.wikimedia.org/wiki/Commons_talk:Montage. If you want to have a 1:1 conversation with one of the developers, please leave a message using the same link. You should feel free to leave questions/comments about the tool in the same page. If you are familiar with GitHub, you can also create issues on GitHub https://github.com/hatnote/montage/issues. Please note that the developers will continue adding features and smoothing the workflow until October 1, what you will test now will give you a sense of the basics you should expect. :)

What is essential for you to be able to start the jury process on October 1 using Montage is the Wikimedia usernames of your jury members (if they don't have an account, please ask them to open one). If you decide to go with Montage, please gather this information sooner rather than later.

WLX

You can learn more about the tool here https://commons.wikimedia.org/wiki/Commons:WLX_Jury_Tool. The tool has been used by many countries as part of WLM and WLE contests in the past years. If you're interested to use WLX, please contact Ilya (WLX's tool developer) in the tool's talk page https://commons.wikimedia.org/wiki/Commons_talk:WLX_Jury_Tool.

If you have any questions or comments, please don't hesitate to ask them here or off-list.

Best,

Leila