Thanks, that is helpful to be aware of.
Montage is indeed not designed as a communication platform between jurors. I think the assumption is that national organizers are better able to organize that separately. I know that the international team set up a mailing list for that, which works OK (This is mostly used between rounds). That is something the national team could also do, I suspect. In some situations another medium (chat, call, etc) may be more appropriate. I'll make sure to pass on your thought to the development team though.
A balance between the different criteria is a returning concern among organizers, and some recommendations from what works well, may be helpful. Maybe other national teams figured this out?
I think it would be especially helpful to set up some specific constructive recommendations for how to organize a national process. I know that Laura from the US team started working on a recommended workflow during the international team meeting, and hope that will be a good starting point for that conversation.
Warmly, Lodewijk
On Sat, Mar 7, 2020 at 8:50 PM Shyamal Lakshminarayanan lshyamal@gmail.com wrote:
[I am not on the list, but feel free to forward]
Just to add, this should not be reduced completely to a software requirements issue - there is also a need for pre-discussion and consensus on the aims, the process to be followed, and documentation of these to ensure clarity.
There is also the point about the software - we had decided that high weightage was to be given to Encyclopaedic value of the image which was also judged on effort and research inputs of the photographer and not just on photographic merit - however some judges were looking only at the photography angle - on the other hand I had decided that any image of places with a very large number of images already on Commons (of places like Qutb Minar or Taj Mahal ) needed to be down-valued regardless of HDR effects or other photographic enhancement attempts. Consistency of user contribution (ie not just here for the contest) also seemed like values to reward. So essentially we were judging along multiple dimensions - some looking at photo quality, some looking at usability, and there were also some grounds for disqualification (lacking EXIF, looking like copyvios, contributor incommunicado ) and so on that need to be handled carefully before just looking at a sorted top 20 from four judges and then looking at the combined top rank . Also the judges were kept out during this final sorting stage and were only revealed the final results (with surprise/shock results).
best wishes Shyamal
On Sun, Mar 8, 2020 at 9:55 AM Bodhisattwa Mandal < bodhisattwa.rgkmc@gmail.com> wrote:
Hi,
[cc'ing Shyamal]
Yes, there were some disappointments from Shyamal's side as a jury member. The following one was his main point of concern.
Shyamal had an expectation that through Montage, while selecting photographs, he would have an option to communicate with other jury members to discuss his points and to know about other member's judgements. While not able to do so, he at the end of the contest, surprisingly noticed that one photograph which was given low rank by him, came to first 10 photographs as it got higher ranking by other jury members as per their judgements. He felt that Montage still needs more improvements so that jury members can communicate with each other and deliver effective judgements.
This was the primary concern which was beyond our capacity to handle as a national organizer team. This can be taken care of by the international team and Montage developers. This year, fron our side, we tried to organize online calls among our jury members, but for some reason, that didn't materialize.
Regards, Bodhisattwa
On Sun, 8 Mar 2020, 09:30 effe iets anders, effeietsanders@gmail.com wrote:
Hi Bodhisattwa,
Thanks for sharing. Could you provide a bit more background information? The LP seems to be mostly a list of things to think about when accepting the position. Was this a judge that was particularly disappointed by how the process was ran? What were the primary pain points? Did the team manage to improve on them for the future?
Thanks!
Lodewijk
On Sat, Mar 7, 2020 at 7:32 PM Bodhisattwa Mandal < bodhisattwa.rgkmc@gmail.com> wrote:
Hi,
Our WLM in India 2018 jury member User:Shyamal wrote a learning pattern on meta.
Here is the link - https://meta.wikimedia.org/wiki/Learning_patterns/Being_a_judge
Regards, Bodhisattwa _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Hi Lodewijk,
Its easy to arrange an online call by national organizers but not always, the jury members want to communicate with each other through chats or calls due to various reasons. If 1 of the 3 members does not want to join calls or does not want to communicate through other private channels, then the objective of communication fails. That happened this year, as we could not arrange calls for jury members for this reason.
Thats why, it is necessary to give jury members power to communicate with each other through Montage or to check each other's point of judgements, so that communication does not depend on the will of organizing team or jury team members.
Regards Bodhisattwa
On Sun, 8 Mar 2020 at 10:27, effe iets anders effeietsanders@gmail.com wrote:
Thanks, that is helpful to be aware of.
Montage is indeed not designed as a communication platform between jurors. I think the assumption is that national organizers are better able to organize that separately. I know that the international team set up a mailing list for that, which works OK (This is mostly used between rounds). That is something the national team could also do, I suspect. In some situations another medium (chat, call, etc) may be more appropriate. I'll make sure to pass on your thought to the development team though.
A balance between the different criteria is a returning concern among organizers, and some recommendations from what works well, may be helpful. Maybe other national teams figured this out?
I think it would be especially helpful to set up some specific constructive recommendations for how to organize a national process. I know that Laura from the US team started working on a recommended workflow during the international team meeting, and hope that will be a good starting point for that conversation.
Warmly, Lodewijk
On Sat, Mar 7, 2020 at 8:50 PM Shyamal Lakshminarayanan < lshyamal@gmail.com> wrote:
[I am not on the list, but feel free to forward]
Just to add, this should not be reduced completely to a software requirements issue - there is also a need for pre-discussion and consensus on the aims, the process to be followed, and documentation of these to ensure clarity.
There is also the point about the software - we had decided that high weightage was to be given to Encyclopaedic value of the image which was also judged on effort and research inputs of the photographer and not just on photographic merit - however some judges were looking only at the photography angle - on the other hand I had decided that any image of places with a very large number of images already on Commons (of places like Qutb Minar or Taj Mahal ) needed to be down-valued regardless of HDR effects or other photographic enhancement attempts. Consistency of user contribution (ie not just here for the contest) also seemed like values to reward. So essentially we were judging along multiple dimensions - some looking at photo quality, some looking at usability, and there were also some grounds for disqualification (lacking EXIF, looking like copyvios, contributor incommunicado ) and so on that need to be handled carefully before just looking at a sorted top 20 from four judges and then looking at the combined top rank . Also the judges were kept out during this final sorting stage and were only revealed the final results (with surprise/shock results).
best wishes Shyamal
On Sun, Mar 8, 2020 at 9:55 AM Bodhisattwa Mandal < bodhisattwa.rgkmc@gmail.com> wrote:
Hi,
[cc'ing Shyamal]
Yes, there were some disappointments from Shyamal's side as a jury member. The following one was his main point of concern.
Shyamal had an expectation that through Montage, while selecting photographs, he would have an option to communicate with other jury members to discuss his points and to know about other member's judgements. While not able to do so, he at the end of the contest, surprisingly noticed that one photograph which was given low rank by him, came to first 10 photographs as it got higher ranking by other jury members as per their judgements. He felt that Montage still needs more improvements so that jury members can communicate with each other and deliver effective judgements.
This was the primary concern which was beyond our capacity to handle as a national organizer team. This can be taken care of by the international team and Montage developers. This year, fron our side, we tried to organize online calls among our jury members, but for some reason, that didn't materialize.
Regards, Bodhisattwa
On Sun, 8 Mar 2020, 09:30 effe iets anders, effeietsanders@gmail.com wrote:
Hi Bodhisattwa,
Thanks for sharing. Could you provide a bit more background information? The LP seems to be mostly a list of things to think about when accepting the position. Was this a judge that was particularly disappointed by how the process was ran? What were the primary pain points? Did the team manage to improve on them for the future?
Thanks!
Lodewijk
On Sat, Mar 7, 2020 at 7:32 PM Bodhisattwa Mandal < bodhisattwa.rgkmc@gmail.com> wrote:
Hi,
Our WLM in India 2018 jury member User:Shyamal wrote a learning pattern on meta.
Here is the link - https://meta.wikimedia.org/wiki/Learning_patterns/Being_a_judge
Regards, Bodhisattwa _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Dear All,
we were indeed using the mailing list (consistently since I believe 2014), and a discussion of every photograph of the top 10, as well as before that a discussion of what we leave in the short list, is a necessary part of a workflow. We never had jury members who declined to respond (with a couple of exceptions when a jury member would let us know well in advance that they would not be available during a certain time interval), and though we typically do not agree with each other the discussions are always constructive and productive.
Best Yaroslav
On Sun, Mar 8, 2020 at 6:12 AM Bodhisattwa Mandal < bodhisattwa.rgkmc@gmail.com> wrote:
Hi Lodewijk,
Its easy to arrange an online call by national organizers but not always, the jury members want to communicate with each other through chats or calls due to various reasons. If 1 of the 3 members does not want to join calls or does not want to communicate through other private channels, then the objective of communication fails. That happened this year, as we could not arrange calls for jury members for this reason.
Thats why, it is necessary to give jury members power to communicate with each other through Montage or to check each other's point of judgements, so that communication does not depend on the will of organizing team or jury team members.
Regards Bodhisattwa
On Sun, 8 Mar 2020 at 10:27, effe iets anders effeietsanders@gmail.com wrote:
Thanks, that is helpful to be aware of.
Montage is indeed not designed as a communication platform between jurors. I think the assumption is that national organizers are better able to organize that separately. I know that the international team set up a mailing list for that, which works OK (This is mostly used between rounds). That is something the national team could also do, I suspect. In some situations another medium (chat, call, etc) may be more appropriate. I'll make sure to pass on your thought to the development team though.
A balance between the different criteria is a returning concern among organizers, and some recommendations from what works well, may be helpful. Maybe other national teams figured this out?
I think it would be especially helpful to set up some specific constructive recommendations for how to organize a national process. I know that Laura from the US team started working on a recommended workflow during the international team meeting, and hope that will be a good starting point for that conversation.
Warmly, Lodewijk
On Sat, Mar 7, 2020 at 8:50 PM Shyamal Lakshminarayanan < lshyamal@gmail.com> wrote:
[I am not on the list, but feel free to forward]
Just to add, this should not be reduced completely to a software requirements issue - there is also a need for pre-discussion and consensus on the aims, the process to be followed, and documentation of these to ensure clarity.
There is also the point about the software - we had decided that high weightage was to be given to Encyclopaedic value of the image which was also judged on effort and research inputs of the photographer and not just on photographic merit - however some judges were looking only at the photography angle - on the other hand I had decided that any image of places with a very large number of images already on Commons (of places like Qutb Minar or Taj Mahal ) needed to be down-valued regardless of HDR effects or other photographic enhancement attempts. Consistency of user contribution (ie not just here for the contest) also seemed like values to reward. So essentially we were judging along multiple dimensions - some looking at photo quality, some looking at usability, and there were also some grounds for disqualification (lacking EXIF, looking like copyvios, contributor incommunicado ) and so on that need to be handled carefully before just looking at a sorted top 20 from four judges and then looking at the combined top rank . Also the judges were kept out during this final sorting stage and were only revealed the final results (with surprise/shock results).
best wishes Shyamal
On Sun, Mar 8, 2020 at 9:55 AM Bodhisattwa Mandal < bodhisattwa.rgkmc@gmail.com> wrote:
Hi,
[cc'ing Shyamal]
Yes, there were some disappointments from Shyamal's side as a jury member. The following one was his main point of concern.
Shyamal had an expectation that through Montage, while selecting photographs, he would have an option to communicate with other jury members to discuss his points and to know about other member's judgements. While not able to do so, he at the end of the contest, surprisingly noticed that one photograph which was given low rank by him, came to first 10 photographs as it got higher ranking by other jury members as per their judgements. He felt that Montage still needs more improvements so that jury members can communicate with each other and deliver effective judgements.
This was the primary concern which was beyond our capacity to handle as a national organizer team. This can be taken care of by the international team and Montage developers. This year, fron our side, we tried to organize online calls among our jury members, but for some reason, that didn't materialize.
Regards, Bodhisattwa
On Sun, 8 Mar 2020, 09:30 effe iets anders, effeietsanders@gmail.com wrote:
Hi Bodhisattwa,
Thanks for sharing. Could you provide a bit more background information? The LP seems to be mostly a list of things to think about when accepting the position. Was this a judge that was particularly disappointed by how the process was ran? What were the primary pain points? Did the team manage to improve on them for the future?
Thanks!
Lodewijk
On Sat, Mar 7, 2020 at 7:32 PM Bodhisattwa Mandal < bodhisattwa.rgkmc@gmail.com> wrote:
Hi,
Our WLM in India 2018 jury member User:Shyamal wrote a learning pattern on meta.
Here is the link - https://meta.wikimedia.org/wiki/Learning_patterns/Being_a_judge
Regards, Bodhisattwa _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Hello Bodhisattwa
I'm not convinced Montage should be *the* communications platform for the jurors. Actually, I think that having each juror point the images independently at first is a good thing, as it avoids potential bias based on other jurors points. Seems that your national competition used a direct output of the top-10 images as rated by every juror combined, rather than e.g. take the resulting top-20 and then have the jurors discuss the merits of each one to make them into a top-10.
Finally, I am a bit concerned about your scenario of a jury member which «does not want to join calls or does not want to communicate through other private channels» and «so that communication does not depend on the will of organizing team or jury team members.». If a jury member reject to communicate with other jury members, that's a social problem, not a technical one. You could make it easier to find all the tools needed (such as adding a link to the mailing list from montage), and it can be quite hard to get a suitable time to get everyone online. In that case asynchronous communication (like mailing lists) may be needed. But ultimately, your jury members should be willing to communicate with each other. And the expectation of that they should do so, and may need to perhaps join X online meetings, should be clear from the start.
What makes you think that someone who refused communicating with other members would however be willing to communicate with them if done through montage?
Cheers
wikilovesmonuments@lists.wikimedia.org