We had very bad experience with using the WLX jury tool this year. I described it at length here: https://commons.wikimedia.org/wiki/Commons_talk:WLX_Jury_Tool
Several possible solutions are also mentioned on that page. I would like to hear your opinion about this situation. Could anyone help Ilya with developing and maintaining the jury tool, and make sure that it works at it should work?
If we are planning to meet a very sharp deadline of October 25 for WLM photos, the jury tool should work absolutely smoothly, and user support has to be very prompt. Or it would be good to know already now that we can't rely on the jury tool, and alternative solutions should be searched for.
Sincerely, Alexander
Hi Alexander,
Thank you for sharing your experience. We will review your comments in the WLM international team and will get back to you by 2016-08-03. In terms of the timelines, there is a general understanding in the team that we'd like to have a fully functional jury tool before WLM kicks off on Sep. 1. Otherwise, we will have many sleepless nights and create many more of those for others. It's our high priority to avoid that. :)
Best, Leila, on behalf of WLM international team
On Sun, Jul 31, 2016 at 6:55 AM, Alexander Tsirlin altsirlin@gmail.com wrote:
We had very bad experience with using the WLX jury tool this year. I described it at length here: https://commons.wikimedia.org/wiki/Commons_talk:WLX_Jury_Tool
Several possible solutions are also mentioned on that page. I would like to hear your opinion about this situation. Could anyone help Ilya with developing and maintaining the jury tool, and make sure that it works at it should work?
If we are planning to meet a very sharp deadline of October 25 for WLM photos, the jury tool should work absolutely smoothly, and user support has to be very prompt. Or it would be good to know already now that we can't rely on the jury tool, and alternative solutions should be searched for.
Sincerely, Alexander
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Hi Alexander, I used Ilya's tool this year for first time and I was limited by some functionalities, but I survived.
In my opinion a lot of problems are generated by operational problems (server availability and delays) which are really minor problems in my opinion.
I used other tools in the past and had similar problems because the hosting solution did not assure the scalability and the resources were limited.
I really suggest to focus in a singles solution instead of looking in alternatives.
Any software, even if good, can have problem if hosted in a limited hosting provider.
FYI I started anytime a trial round to check the configuration and afterwards a valid round as soon I was sure to set correctly any configuration. This is a workaround but a workaround is needed in any critical mission.
If a selection is run in the last days, we are in this case in a mission critical situation, and the tool cannot assure to solve any kind of problems.
I will be sure that Ilya will set a tool and will assure a good level of service.
Kind regards
In data 31/Lug/2016 15.55.23, Alexander Tsirlin ha scritto:
We had very bad experience with using the WLX jury tool this year. I described it at length here: https://commons.wikimedia.org/wiki/Commons_talk:WLX_Jury_Tool
Several possible solutions are also mentioned on that page. I would like to hear your opinion about this situation. Could anyone help Ilya with developing and maintaining the jury tool, and make sure that it works at it should work?
If we are planning to meet a very sharp deadline of October 25 for WLM photos, the jury tool should work absolutely smoothly, and user support has to be very prompt. Or it would be good to know already now that we can't rely on the jury tool, and alternative solutions should be searched for.
Sincerely, Alexander
Wikilovesearth mailing list Wikilovesearth@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesearth
As in every previous year, I am offering my tool from 2012 (bug fixes were made but it's essentially the same) for anyone's use. It was used by many countries in 2012 and by a few after that, and while I heard of some issues, AFAIK no one encountered any crippling problems (actual known bugs have been fixed anyway). I am also open to feature requests, except changing the two-level system which would be difficult.
If you are interested, please talk to me before the competition so there is ample time to set up the tool. I can probably set it up on Labs but I hate using Labs so if you have your own LAMP server or don't mind using ours, that would be preferable. I might also upload it to Github sometime.
—Yan.
On Sun, Jul 31, 2016 at 5:33 PM, Ilario Valdelli < ilario.valdelli@wikimedia.ch> wrote:
Hi Alexander, I used Ilya's tool this year for first time and I was limited by some functionalities, but I survived.
In my opinion a lot of problems are generated by operational problems (server availability and delays) which are really minor problems in my opinion.
I used other tools in the past and had similar problems because the hosting solution did not assure the scalability and the resources were limited.
I really suggest to focus in a singles solution instead of looking in alternatives.
Any software, even if good, can have problem if hosted in a limited hosting provider.
FYI I started anytime a trial round to check the configuration and afterwards a valid round as soon I was sure to set correctly any configuration. This is a workaround but a workaround is needed in any critical mission.
If a selection is run in the last days, we are in this case in a mission critical situation, and the tool cannot assure to solve any kind of problems.
I will be sure that Ilya will set a tool and will assure a good level of service.
Kind regards
In data 31/Lug/2016 15.55.23, Alexander Tsirlin ha scritto:
We had very bad experience with using the WLX jury tool this year. I described it at length here: https://commons.wikimedia.org/wiki/Commons_talk:WLX_Jury_Tool
Several possible solutions are also mentioned on that page. I would like to hear your opinion about this situation. Could anyone help Ilya with developing and maintaining the jury tool, and make sure that it works at it should work?
If we are planning to meet a very sharp deadline of October 25 for WLM photos, the jury tool should work absolutely smoothly, and user support has to be very prompt. Or it would be good to know already now that we can't rely on the jury tool, and alternative solutions should be searched for.
Sincerely, Alexander
Wikilovesearth mailing list Wikilovesearth@lists.wikimedia.org <#m_8448359304454196217_> https://lists.wikimedia.org/mailman/listinfo/wikilovesearth
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
I attempted to use the tools on a number of occassions but eventually gave up as I couldnt get the dtat past round 1 ended up bringing all our jurours into one place then just adding and subtracting a category on commons until we had down to final set
On 31 July 2016 at 22:33, Ilario Valdelli ilario.valdelli@wikimedia.ch wrote:
Hi Alexander, I used Ilya's tool this year for first time and I was limited by some functionalities, but I survived.
In my opinion a lot of problems are generated by operational problems (server availability and delays) which are really minor problems in my opinion.
I used other tools in the past and had similar problems because the hosting solution did not assure the scalability and the resources were limited.
I really suggest to focus in a singles solution instead of looking in alternatives.
Any software, even if good, can have problem if hosted in a limited hosting provider.
FYI I started anytime a trial round to check the configuration and afterwards a valid round as soon I was sure to set correctly any configuration. This is a workaround but a workaround is needed in any critical mission.
If a selection is run in the last days, we are in this case in a mission critical situation, and the tool cannot assure to solve any kind of problems.
I will be sure that Ilya will set a tool and will assure a good level of service.
Kind regards
In data 31/Lug/2016 15.55.23, Alexander Tsirlin ha scritto:
We had very bad experience with using the WLX jury tool this year. I described it at length here: https://commons.wikimedia.org/wiki/Commons_talk:WLX_Jury_Tool
Several possible solutions are also mentioned on that page. I would like to hear your opinion about this situation. Could anyone help Ilya with developing and maintaining the jury tool, and make sure that it works at it should work?
If we are planning to meet a very sharp deadline of October 25 for WLM photos, the jury tool should work absolutely smoothly, and user support has to be very prompt. Or it would be good to know already now that we can't rely on the jury tool, and alternative solutions should be searched for.
Sincerely, Alexander
Wikilovesearth mailing list Wikilovesearth@lists.wikimedia.org <#m_-4230169113316713078_> https://lists.wikimedia.org/mailman/listinfo/wikilovesearth
Wikilovesearth mailing list Wikilovesearth@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesearth
Hi Alexander and others, :)
*What I did in the past 2+ days* I reviewed your comments carefully and talked with a subset of the WLM international team including Ilya about the issues you have raised. I also reached out to few people outside of the team, to hear their thoughts and experiences, and here I am proposing the next steps.
*Next steps* * Ilya will be looking to onboard one person to work with him on the code-base. This person will also create the redundancy that you mentioned in your notes earlier. Ideally, between Ilya and this person, the response time will be "same-day" except for non-critical requests. (If you know Scala or you know Java and you are passionate to get to the Scala world, please contact Ilya, and cc me.)
* Ilya will give it a try to move the code-base to Tools this week. This will make sure that the service will have a standard up-time we can count on. There are three tasks associated with this effort: T141910, T124902, T141908. You can find them at https://phabricator.wikimedia.org/project/view/2005/. Ilya and I will review the tasks at the end of this week and re-assess the time they require.
* Given the time crunch, Ilya and I discussed pursuing a plan B or an alternative tool that the international team can work on while Ilya is busy fixing bugs and improving WLX in the coming weeks. Based on this conversation, I have started reaching out to the owners of the 5 other jury tools. In the coming days, my attention will be focused on assessing where these tools are at the moment, and figuring out if offering one of them as an alternative makes sense. We also discussed the option of developing another tool. Basically, all options are on the table with plan b at the moment.
* I will do my best to send updates to wikilovesmonuments public list about the progress with regards to the jury tool. If you don't hear from me and you want to hear more, please ask on the list, and I will get back to you there. :)
*Help needed* We have a short time before the start of the contest, and I want us to feel confident that we will have a solid tool for everyone when we start on Sep. 1. We cannot do this alone though: * If we are to adapt any of the current tools or create a new tool, we are under a time crunch. If you can help with usability testing of the tool we will work on under time constraints, please let me know off-list. * We also may need help for further developing one of the current tools, or creating a new one. If you have time to help with that under the current time constraints, please email me off-list with information about the languages you are comfortable with and maybe a GitHub repository of (a sample of) your past work.
And last but not least, I want to assure you that coming up with a solution for the jury tool(s) that we can use more smoothly during this year's WLM contest has a high priority for the international team. You have our commitments here. :)
Best, Leila, on behalf of the WLM international team
On Sun, Jul 31, 2016 at 7:22 AM, Lily lilyofthewest.wikimedia@gmail.com wrote:
Hi Alexander,
Thank you for sharing your experience. We will review your comments in the WLM international team and will get back to you by 2016-08-03. In terms of the timelines, there is a general understanding in the team that we'd like to have a fully functional jury tool before WLM kicks off on Sep. 1. Otherwise, we will have many sleepless nights and create many more of those for others. It's our high priority to avoid that. :)
Best, Leila, on behalf of WLM international team
On Sun, Jul 31, 2016 at 6:55 AM, Alexander Tsirlin altsirlin@gmail.com wrote:
We had very bad experience with using the WLX jury tool this year. I described it at length here: https://commons.wikimedia.org/wiki/Commons_talk:WLX_Jury_Tool
Several possible solutions are also mentioned on that page. I would like to hear your opinion about this situation. Could anyone help Ilya with developing and maintaining the jury tool, and make sure that it works at it should work?
If we are planning to meet a very sharp deadline of October 25 for WLM photos, the jury tool should work absolutely smoothly, and user support has to be very prompt. Or it would be good to know already now that we can't rely on the jury tool, and alternative solutions should be searched for.
Sincerely, Alexander
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
-- User: LilyOfTheWest
I'll add on * Creating next round - sorry for latency. This is already implemented. On July 16 as Alexander mentioned himself * Tool unavailability ** I installed monit service to reload jury tool and mysql when it goes down. Will have to wait and see if this helps. Before this I tried upstart and systemd configurations that also have respawn option. Unfortunately they worked on Ubuntu/Debian/Centos 7, but not on Centos 6 that is on WMUA's server, so I have to look for other tool and used monit for this. ** Tool is already deployed on Tool Labs - https://tools.wmflabs.org/wlxjury/auth * [T141908 https://phabricator.wikimedia.org/T141908] Restriction that users have contest assigned to them and cannot be added to another - will be fixed by end of the week. * [T141909 https://phabricator.wikimedia.org/T141909] Complete export from the jury tool. It is actually available in the form of exporting each juror rating or selection separately and overall rating for rating rounds. What is suggested is to have overall number of jurors that selected an image in the selection round and (Leila's suggestion) one cross table with images as rows, jurors as columns and rating/selections as cell. This is a trivial change as everything is already loaded in statistics module, I'll just to have to output these specific fields. Also will be done by end of the week.
Regarding alternate tool. Overall I think it's always good to have alternative and understand what we have. I know at least two tools which can to be good -
- Israel jury tool by User:Ynhockey https://commons.wikimedia.org/wiki/User:Ynhockey (his own claim) - Germany jury tool - http://tools.wmflabs.org/jury/ (Germany has lots of images and usually select winners fast, so I hope their tool is good)
Unfortunately I have not seen them, and their sources are not published.
Regarding developing of some completely new tool. I don't think it is a serious option. It will be very effort consuming and most certainly will not provide a better tool than already existing ones. At least for this years WLM.
Regards, Ilya
On Wed, Aug 3, 2016 at 9:16 AM, Lily lilyofthewest.wikimedia@gmail.com wrote:
Hi Alexander and others, :)
*What I did in the past 2+ days* I reviewed your comments carefully and talked with a subset of the WLM international team including Ilya about the issues you have raised. I also reached out to few people outside of the team, to hear their thoughts and experiences, and here I am proposing the next steps.
*Next steps*
- Ilya will be looking to onboard one person to work with him on the
code-base. This person will also create the redundancy that you mentioned in your notes earlier. Ideally, between Ilya and this person, the response time will be "same-day" except for non-critical requests. (If you know Scala or you know Java and you are passionate to get to the Scala world, please contact Ilya, and cc me.)
- Ilya will give it a try to move the code-base to Tools this week. This
will make sure that the service will have a standard up-time we can count on. There are three tasks associated with this effort: T141910, T124902, T141908. You can find them at https://phabricator.wikimedia.org/project/view/2005/. Ilya and I will review the tasks at the end of this week and re-assess the time they require.
- Given the time crunch, Ilya and I discussed pursuing a plan B or an
alternative tool that the international team can work on while Ilya is busy fixing bugs and improving WLX in the coming weeks. Based on this conversation, I have started reaching out to the owners of the 5 other jury tools. In the coming days, my attention will be focused on assessing where these tools are at the moment, and figuring out if offering one of them as an alternative makes sense. We also discussed the option of developing another tool. Basically, all options are on the table with plan b at the moment.
- I will do my best to send updates to wikilovesmonuments public list
about the progress with regards to the jury tool. If you don't hear from me and you want to hear more, please ask on the list, and I will get back to you there. :)
*Help needed* We have a short time before the start of the contest, and I want us to feel confident that we will have a solid tool for everyone when we start on Sep. 1. We cannot do this alone though:
- If we are to adapt any of the current tools or create a new tool, we are
under a time crunch. If you can help with usability testing of the tool we will work on under time constraints, please let me know off-list.
- We also may need help for further developing one of the current tools,
or creating a new one. If you have time to help with that under the current time constraints, please email me off-list with information about the languages you are comfortable with and maybe a GitHub repository of (a sample of) your past work.
And last but not least, I want to assure you that coming up with a solution for the jury tool(s) that we can use more smoothly during this year's WLM contest has a high priority for the international team. You have our commitments here. :)
Best, Leila, on behalf of the WLM international team
On Sun, Jul 31, 2016 at 7:22 AM, Lily lilyofthewest.wikimedia@gmail.com wrote:
Hi Alexander,
Thank you for sharing your experience. We will review your comments in the WLM international team and will get back to you by 2016-08-03. In terms of the timelines, there is a general understanding in the team that we'd like to have a fully functional jury tool before WLM kicks off on Sep. 1. Otherwise, we will have many sleepless nights and create many more of those for others. It's our high priority to avoid that. :)
Best, Leila, on behalf of WLM international team
On Sun, Jul 31, 2016 at 6:55 AM, Alexander Tsirlin altsirlin@gmail.com wrote:
We had very bad experience with using the WLX jury tool this year. I described it at length here: https://commons.wikimedia.org/wiki/Commons_talk:WLX_Jury_Tool
Several possible solutions are also mentioned on that page. I would like to hear your opinion about this situation. Could anyone help Ilya with developing and maintaining the jury tool, and make sure that it works at it should work?
If we are planning to meet a very sharp deadline of October 25 for WLM photos, the jury tool should work absolutely smoothly, and user support has to be very prompt. Or it would be good to know already now that we can't rely on the jury tool, and alternative solutions should be searched for.
Sincerely, Alexander
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
-- User: LilyOfTheWest
-- User: LilyOfTheWest
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Dear Ilya, Leila, and others,
Thank you for your response. Unfortunately, I am not programmer, and I can't help you with the development. I only know that Tools had severe problems in the past, and it may not be a very reliable solution because, if something happens to the server, you won't have direct control and won't be able to solve the problem quickly. I had enough trouble when map scripts for Wikivoyage were running there.
I have spoken to our jury members, and their request is two-fold. First, the jury tool should be fully operational and stable before the contest and not 1.5 months after its end. Second, it is highly desirable to start the grading already in September, because processing 20,000 photos in 25 days would be difficult.
When I am writing about alternatives, I also mean that the grading can be organized with a very simple system of galleries and grading sheets that lacks the nice graphical interface, but works reliably and directly on Commons. And it perfectly fits our needs mentioned above. After our recent WLE experience I seriously think that such a primitive system will save us a lot of time and nerves. However, let's wait until the end of August and see where we are.
From my side, I can test the system in the end of August and write my comments about its usability. It will also help me to decide whether we use WLX jury tool for WLM this year.
Sincerely, Alexander
PS. Thanks for submitting the Phabricator tickets!
On 03.08.2016 11:38, Ilya Korniyko wrote:
I'll add on
- Creating next round - sorry for latency. This is already
implemented. On July 16 as Alexander mentioned himself
- Tool unavailability
** I installed monit service to reload jury tool and mysql when it goes down. Will have to wait and see if this helps. Before this I tried upstart and systemd configurations that also have respawn option. Unfortunately they worked on Ubuntu/Debian/Centos 7, but not on Centos 6 that is on WMUA's server, so I have to look for other tool and used monit for this. ** Tool is already deployed on Tool Labs - https://tools.wmflabs.org/wlxjury/auth
- [T141908 https://phabricator.wikimedia.org/T141908] Restriction
that users have contest assigned to them and cannot be added to another - will be fixed by end of the week.
- [T141909 https://phabricator.wikimedia.org/T141909] Complete
export from the jury tool. It is actually available in the form of exporting each juror rating or selection separately and overall rating for rating rounds. What is suggested is to have overall number of jurors that selected an image in the selection round and (Leila's suggestion) one cross table with images as rows, jurors as columns and rating/selections as cell. This is a trivial change as everything is already loaded in statistics module, I'll just to have to output these specific fields. Also will be done by end of the week.
Regarding alternate tool. Overall I think it's always good to have alternative and understand what we have. I know at least two tools which can to be good -
- Israel jury tool by User:Ynhockey https://commons.wikimedia.org/wiki/User:Ynhockey (his own claim)
- Germany jury tool - http://tools.wmflabs.org/jury/ (Germany has lots of images and usually select winners fast, so I hope their tool is good)
Unfortunately I have not seen them, and their sources are not published.
Regarding developing of some completely new tool. I don't think it is a serious option. It will be very effort consuming and most certainly will not provide a better tool than already existing ones. At least for this years WLM.
Regards, Ilya
On Wed, Aug 3, 2016 at 9:16 AM, Lily <lilyofthewest.wikimedia@gmail.com mailto:lilyofthewest.wikimedia@gmail.com> wrote:
Hi Alexander and others, :) *What I did in the past 2+ days* I reviewed your comments carefully and talked with a subset of the WLM international team including Ilya about the issues you have raised. I also reached out to few people outside of the team, to hear their thoughts and experiences, and here I am proposing the next steps. *Next steps* * Ilya will be looking to onboard one person to work with him on the code-base. This person will also create the redundancy that you mentioned in your notes earlier. Ideally, between Ilya and this person, the response time will be "same-day" except for non-critical requests. (If you know Scala or you know Java and you are passionate to get to the Scala world, please contact Ilya, and cc me.) * Ilya will give it a try to move the code-base to Tools this week. This will make sure that the service will have a standard up-time we can count on. There are three tasks associated with this effort: T141910, T124902, T141908. You can find them at https://phabricator.wikimedia.org/project/view/2005/. Ilya and I will review the tasks at the end of this week and re-assess the time they require. * Given the time crunch, Ilya and I discussed pursuing a plan B or an alternative tool that the international team can work on while Ilya is busy fixing bugs and improving WLX in the coming weeks. Based on this conversation, I have started reaching out to the owners of the 5 other jury tools. In the coming days, my attention will be focused on assessing where these tools are at the moment, and figuring out if offering one of them as an alternative makes sense. We also discussed the option of developing another tool. Basically, all options are on the table with plan b at the moment. * I will do my best to send updates to wikilovesmonuments public list about the progress with regards to the jury tool. If you don't hear from me and you want to hear more, please ask on the list, and I will get back to you there. :) *Help needed* We have a short time before the start of the contest, and I want us to feel confident that we will have a solid tool for everyone when we start on Sep. 1. We cannot do this alone though: * If we are to adapt any of the current tools or create a new tool, we are under a time crunch. If you can help with usability testing of the tool we will work on under time constraints, please let me know off-list. * We also may need help for further developing one of the current tools, or creating a new one. If you have time to help with that under the current time constraints, please email me off-list with information about the languages you are comfortable with and maybe a GitHub repository of (a sample of) your past work. And last but not least, I want to assure you that coming up with a solution for the jury tool(s) that we can use more smoothly during this year's WLM contest has a high priority for the international team. You have our commitments here. :) Best, Leila, on behalf of the WLM international team On Sun, Jul 31, 2016 at 7:22 AM, Lily <lilyofthewest.wikimedia@gmail.com <mailto:lilyofthewest.wikimedia@gmail.com>> wrote: Hi Alexander, Thank you for sharing your experience. We will review your comments in the WLM international team and will get back to you by 2016-08-03. In terms of the timelines, there is a general understanding in the team that we'd like to have a fully functional jury tool before WLM kicks off on Sep. 1. Otherwise, we will have many sleepless nights and create many more of those for others. It's our high priority to avoid that. :) Best, Leila, on behalf of WLM international team On Sun, Jul 31, 2016 at 6:55 AM, Alexander Tsirlin <altsirlin@gmail.com <mailto:altsirlin@gmail.com>> wrote: We had very bad experience with using the WLX jury tool this year. I described it at length here: https://commons.wikimedia.org/wiki/Commons_talk:WLX_Jury_Tool Several possible solutions are also mentioned on that page. I would like to hear your opinion about this situation. Could anyone help Ilya with developing and maintaining the jury tool, and make sure that it works at it should work? If we are planning to meet a very sharp deadline of October 25 for WLM photos, the jury tool should work absolutely smoothly, and user support has to be very prompt. Or it would be good to know already now that we can't rely on the jury tool, and alternative solutions should be searched for. Sincerely, Alexander _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org <mailto:WikiLovesMonuments@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org -- User: LilyOfTheWest -- User: LilyOfTheWest _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org <mailto:WikiLovesMonuments@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org-- You received this message because you are subscribed to the Google Groups "WLX Russia organizers" group. To unsubscribe from this group and stop receiving emails from it, send an email to wlx-russia+unsubscribe@googlegroups.com mailto:wlx-russia+unsubscribe@googlegroups.com. To post to this group, send email to wlx-russia@googlegroups.com mailto:wlx-russia@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/wlx-russia/CAN8OKLZdymV7xTTJj-8X9BaLxticDy... https://groups.google.com/d/msgid/wlx-russia/CAN8OKLZdymV7xTTJj-8X9BaLxticDys7geY1wG0dTgxFS9GYpw%40mail.gmail.com?utm_medium=email&utm_source=footer. For more options, visit https://groups.google.com/d/optout.
+ Yuvi
On Wed, Aug 3, 2016 at 3:31 AM, Alexander Tsirlin altsirlin@gmail.com wrote:
Dear Ilya, Leila, and others,
Thank you for your response. Unfortunately, I am not programmer, and I can't help you with the development. I only know that Tools had severe problems in the past, and it may not be a very reliable solution because, if something happens to the server, you won't have direct control and won't be able to solve the problem quickly. I had enough trouble when map scripts for Wikivoyage were running there.
Yuvi, what kind of guarantee can we have for Tools' availability?
I have spoken to our jury members, and their request is two-fold. First, the jury tool should be fully operational and stable before the contest and not 1.5 months after its end. Second, it is highly desirable to start the grading already in September, because processing 20,000 photos in 25 days would be difficult.
Our goal is to have a functioning tool before the start of the contest. This being said, I only can commit to a fully functioning jury tool availability by October 1, not before that (again, we will do our best to have the tool ready earlier, I just can't guarantee it at this point). The assumption is that countries do not start the jury process until all photos are in (though I understand that some countries start earlier).
When I am writing about alternatives, I also mean that the grading can be organized with a very simple system of galleries and grading sheets that lacks the nice graphical interface, but works reliably and directly on Commons. And it perfectly fits our needs mentioned above. After our recent WLE experience I seriously think that such a primitive system will save us a lot of time and nerves. However, let's wait until the end of August and see where we are.
Yes, I understand. I can comment more on such a system after I finish this round of researching all the tools available.
From my side, I can test the system in the end of August and write my comments about its usability. It will also help me to decide whether we use WLX jury tool for WLM this year.
Perfect. Thank you! :)
Leila
Sincerely, Alexander
PS. Thanks for submitting the Phabricator tickets!
On 03.08.2016 11:38, Ilya Korniyko wrote:
I'll add on
- Creating next round - sorry for latency. This is already implemented. On July
16 as Alexander mentioned himself
- Tool unavailability
** I installed monit service to reload jury tool and mysql when it goes down. Will have to wait and see if this helps. Before this I tried upstart and systemd configurations that also have respawn option. Unfortunately they worked on Ubuntu/Debian/Centos 7, but not on Centos 6 that is on WMUA's server, so I have to look for other tool and used monit for this. ** Tool is already deployed on Tool Labs - https://tools.wmflabs.org/wlxjury/auth https://tools.wmflabs.org/wlxjury/auth
- [T141908 https://phabricator.wikimedia.org/T141908] Restriction that
users have contest assigned to them and cannot be added to another - will be fixed by end of the week.
- [T141909 https://phabricator.wikimedia.org/T141909] Complete export
from the jury tool. It is actually available in the form of exporting each juror rating or selection separately and overall rating for rating rounds. What is suggested is to have overall number of jurors that selected an image in the selection round and (Leila's suggestion) one cross table with images as rows, jurors as columns and rating/selections as cell. This is a trivial change as everything is already loaded in statistics module, I'll just to have to output these specific fields. Also will be done by end of the week.
Regarding alternate tool. Overall I think it's always good to have alternative and understand what we have. I know at least two tools which can to be good -
- Israel jury tool by User:Ynhockey
https://commons.wikimedia.org/wiki/User:Ynhockey (his own claim)
- Germany jury tool - http://tools.wmflabs.org/jury/
http://tools.wmflabs.org/jury/ (Germany has lots of images and usually select winners fast, so I hope their tool is good)
Unfortunately I have not seen them, and their sources are not published.
Regarding developing of some completely new tool. I don't think it is a serious option. It will be very effort consuming and most certainly will not provide a better tool than already existing ones. At least for this years WLM.
Regards, Ilya
On Wed, Aug 3, 2016 at 9:16 AM, Lily lilyofthewest.wikimedia@gmail.com wrote:
Hi Alexander and others, :)
*What I did in the past 2+ days* I reviewed your comments carefully and talked with a subset of the WLM international team including Ilya about the issues you have raised. I also reached out to few people outside of the team, to hear their thoughts and experiences, and here I am proposing the next steps.
*Next steps*
- Ilya will be looking to onboard one person to work with him on the
code-base. This person will also create the redundancy that you mentioned in your notes earlier. Ideally, between Ilya and this person, the response time will be "same-day" except for non-critical requests. (If you know Scala or you know Java and you are passionate to get to the Scala world, please contact Ilya, and cc me.)
- Ilya will give it a try to move the code-base to Tools this week. This
will make sure that the service will have a standard up-time we can count on. There are three tasks associated with this effort: T141910, T124902, T141908. You can find them at https://phabricator.wikimedia.org/project/view/2005/. Ilya and I will review the tasks at the end of this week and re-assess the time they require.
- Given the time crunch, Ilya and I discussed pursuing a plan B or an
alternative tool that the international team can work on while Ilya is busy fixing bugs and improving WLX in the coming weeks. Based on this conversation, I have started reaching out to the owners of the 5 other jury tools. In the coming days, my attention will be focused on assessing where these tools are at the moment, and figuring out if offering one of them as an alternative makes sense. We also discussed the option of developing another tool. Basically, all options are on the table with plan b at the moment.
- I will do my best to send updates to wikilovesmonuments public list
about the progress with regards to the jury tool. If you don't hear from me and you want to hear more, please ask on the list, and I will get back to you there. :)
*Help needed* We have a short time before the start of the contest, and I want us to feel confident that we will have a solid tool for everyone when we start on Sep. 1. We cannot do this alone though:
- If we are to adapt any of the current tools or create a new tool, we
are under a time crunch. If you can help with usability testing of the tool we will work on under time constraints, please let me know off-list.
- We also may need help for further developing one of the current tools,
or creating a new one. If you have time to help with that under the current time constraints, please email me off-list with information about the languages you are comfortable with and maybe a GitHub repository of (a sample of) your past work.
And last but not least, I want to assure you that coming up with a solution for the jury tool(s) that we can use more smoothly during this year's WLM contest has a high priority for the international team. You have our commitments here. :)
Best, Leila, on behalf of the WLM international team
On Sun, Jul 31, 2016 at 7:22 AM, Lily < lilyofthewest.wikimedia@gmail.comlilyofthewest.wikimedia@gmail.com> wrote:
Hi Alexander,
Thank you for sharing your experience. We will review your comments in the WLM international team and will get back to you by 2016-08-03. In terms of the timelines, there is a general understanding in the team that we'd like to have a fully functional jury tool before WLM kicks off on Sep. 1. Otherwise, we will have many sleepless nights and create many more of those for others. It's our high priority to avoid that. :)
Best, Leila, on behalf of WLM international team
On Sun, Jul 31, 2016 at 6:55 AM, Alexander Tsirlin altsirlin@gmail.com wrote:
We had very bad experience with using the WLX jury tool this year. I described it at length here: https://commons.wikimedia.org/wiki/Commons_talk:WLX_Jury_Tool
Several possible solutions are also mentioned on that page. I would like to hear your opinion about this situation. Could anyone help Ilya with developing and maintaining the jury tool, and make sure that it works at it should work?
If we are planning to meet a very sharp deadline of October 25 for WLM photos, the jury tool should work absolutely smoothly, and user support has to be very prompt. Or it would be good to know already now that we can't rely on the jury tool, and alternative solutions should be searched for.
Sincerely, Alexander
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
-- User: LilyOfTheWest
-- User: LilyOfTheWest
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
-- You received this message because you are subscribed to the Google Groups "WLX Russia organizers" group. To unsubscribe from this group and stop receiving emails from it, send an email to wlx-russia+unsubscribe@googlegroups.com. To post to this group, send email to wlx-russia@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/wlx-russia/CAN8OKLZdymV7xTTJj-8X9BaLxticDys7geY1wG0dTgxFS9GYpw%40mail.gmail.com?utm_medium=email&utm_source=footer https://groups.google.com/d/msgid/wlx-russia/CAN8OKLZdymV7xTTJj-8X9BaLxticDy... . For more options, visit https://groups.google.com/d/optout.
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
On Wed, Aug 3, 2016 at 9:12 PM, Lily lilyofthewest.wikimedia@gmail.com wrote:
- Yuvi
On Wed, Aug 3, 2016 at 3:31 AM, Alexander Tsirlin altsirlin@gmail.com wrote:
Dear Ilya, Leila, and others,
Thank you for your response. Unfortunately, I am not programmer, and I can't help you with the development. I only know that Tools had severe problems in the past, and it may not be a very reliable solution because, if something happens to the server, you won't have direct control and won't be able to solve the problem quickly. I had enough trouble when map scripts for Wikivoyage were running there.
Yuvi, what kind of guarantee can we have for Tools' availability?
It has definitely been more stable than in the past by quite a margin, I think. However, we do have some maintenance planned (not announced yet) for end of Aug / beginning of September related to NFS that could cause some instabilities. In light of that, I think the following is a reasonable course of action to follow:
1. Identify critical 'must have' tools (not too many!) 2. Set them up in a labs instances that doesn't have any NFS (I can help do this) 3. When we do the planned maintenance, re-route the tools to hit the labs instances setup in (2) (I can help do this a well).
Now all we need is a *super* minimal (maybe 2?) of tools that should be considered absolutely essential for September (I expect everyting to be stable by October), so we can setup redundancies for that tool.
Sounds good. We will let you know the list of couple of tools we need extra support with in the next 10 days.
As a side note, as long as the maintenance schedule does not collide with the last days of the contest or jury process (the end of September and October), we know ahead of time about the downtime, and we're talking about few hours of unavailability, WLM will be fine. It will of course be very helpful to have the absolutely essential tools up at all times. :)
Thanks for offering help.
Leila
On Aug 6, 2016 10:12 PM, "Yuvi Panda" yuvipanda@gmail.com wrote:
On Wed, Aug 3, 2016 at 9:12 PM, Lily lilyofthewest.wikimedia@gmail.com wrote:
- Yuvi
On Wed, Aug 3, 2016 at 3:31 AM, Alexander Tsirlin altsirlin@gmail.com wrote:
Dear Ilya, Leila, and others,
Thank you for your response. Unfortunately, I am not programmer, and I can't help you with the development. I only know that Tools had severe problems in the past, and it may not be a very reliable solution
because, if
something happens to the server, you won't have direct control and
won't be
able to solve the problem quickly. I had enough trouble when map
scripts for
Wikivoyage were running there.
Yuvi, what kind of guarantee can we have for Tools' availability?
It has definitely been more stable than in the past by quite a margin, I think. However, we do have some maintenance planned (not announced yet) for end of Aug / beginning of September related to NFS that could cause some instabilities. In light of that, I think the following is a reasonable course of action to follow:
- Identify critical 'must have' tools (not too many!)
- Set them up in a labs instances that doesn't have any NFS (I can
help do this) 3. When we do the planned maintenance, re-route the tools to hit the labs instances setup in (2) (I can help do this a well).
Now all we need is a *super* minimal (maybe 2?) of tools that should be considered absolutely essential for September (I expect everyting to be stable by October), so we can setup redundancies for that tool.
-- Yuvi Panda T http://yuvi.in/blog
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
My tool went down after 3 days on Labs https://www.statuscake.com/App/AllStatus.php?tid=1334098 I think it will require https://phabricator.wikimedia.org/T142164 Build replacement for the webservice toolschecker test On WMUA's server I think I solved this with the monit service, the was no downtime in 5 days according to statuscake checks (each 5 minutes)
On Sun, Aug 7, 2016 at 8:11 AM, Yuvi Panda yuvipanda@gmail.com wrote:
On Wed, Aug 3, 2016 at 9:12 PM, Lily lilyofthewest.wikimedia@gmail.com wrote:
- Yuvi
On Wed, Aug 3, 2016 at 3:31 AM, Alexander Tsirlin altsirlin@gmail.com wrote:
Dear Ilya, Leila, and others,
Thank you for your response. Unfortunately, I am not programmer, and I can't help you with the development. I only know that Tools had severe problems in the past, and it may not be a very reliable solution
because, if
something happens to the server, you won't have direct control and
won't be
able to solve the problem quickly. I had enough trouble when map
scripts for
Wikivoyage were running there.
Yuvi, what kind of guarantee can we have for Tools' availability?
It has definitely been more stable than in the past by quite a margin, I think. However, we do have some maintenance planned (not announced yet) for end of Aug / beginning of September related to NFS that could cause some instabilities. In light of that, I think the following is a reasonable course of action to follow:
- Identify critical 'must have' tools (not too many!)
- Set them up in a labs instances that doesn't have any NFS (I can
help do this) 3. When we do the planned maintenance, re-route the tools to hit the labs instances setup in (2) (I can help do this a well).
Now all we need is a *super* minimal (maybe 2?) of tools that should be considered absolutely essential for September (I expect everyting to be stable by October), so we can setup redundancies for that tool.
-- Yuvi Panda T http://yuvi.in/blog
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
On Sun, Aug 7, 2016 at 11:22 AM, Ilya Korniyko intracer@gmail.com wrote:
My tool went down after 3 days on Labs https://www.statuscake.com/App/AllStatus.php?tid=1334098
Can not access that link, requires a log in?
I think it will require https://phabricator.wikimedia.org/T142164 Build replacement for the webservice toolschecker test
This has nothing to do with stability of user tools on tool labs at all, only with the ways we are monitoring the overall system in place to keep webservices up.
On Mon, Aug 8, 2016 at 7:50 AM, Yuvi Panda yuvipanda@gmail.com wrote:
On Sun, Aug 7, 2016 at 11:22 AM, Ilya Korniyko intracer@gmail.com wrote:
My tool went down after 3 days on Labs https://www.statuscake.com/App/AllStatus.php?tid=1334098
Can not access that link, requires a log in?
Down From: 2016-08-06 19:41:19 (check interval is 5 minutes) To: 2016-08-06 20:09:45 (when I logged in and restarted)
I think it will require https://phabricator.wikimedia.org/T142164 Build replacement for the webservice toolschecker test
This has nothing to do with stability of user tools on tool labs at all, only with the ways we are monitoring the overall system in place to keep webservices up.
-- Yuvi Panda T http://yuvi.in/blog
On Sun, Aug 7, 2016 at 10:01 PM, Ilya Korniyko intracer@gmail.com wrote:
Down From: 2016-08-06 19:41:19 (check interval is 5 minutes) To: 2016-08-06 20:09:45 (when I logged in and restarted)
Please report it if it happens again - yours is the first scala tool running on tools as a webservice, and I've no idea what could be causing it :)
wikilovesmonuments@lists.wikimedia.org