Hello Erik, At Tuesday 25 September 2012 22:24:33 DaB. wrote:
Hi all,
let me weigh in with a few initial comments, and I'll ask the Labs folks to participate here and on Meta as well with regard to technical questions.
The initial focus for Labs has been to provide functionality that toolserver doesn't - get root on a VM or set of VMs to install/test arbitrary software/services, and get it ready for production deployment.
It is nice to have root on a (virtual) machine, but I doubt most tools need it. I would also bet that most tool-authors should not have root-access and should not be able to install software. I guess what most people and the WMF and also WMDE does not understand is: Most tool-authors are no system-ops or have a Master of Informatic Science. Most are amateur programmers who have fun to code a tool and do not maintain it once it is done (they also not document it). The tools they are write a slow and need way too much resources. They have no backup of their ssh-keys and do not extend their accounts in time. And all this is ok. Because (Nosy and) I am there to give them a infrastructure they can use, kick them a little bit if their tools misbehavior or really need an update, extend their accounts and add new ssh-keys. Some times this job sucks, but most times it is a good feeling to know that other people use these tools and it helps the wikimedia- projects. I very doubt that ANY sysop at WMF would do that (can you imagine Domas helping an user to create its first ssh-key (and the second shortly after because the user commit the private key instead of the public)?). I'm very sure that I have users on the TS that are able to use Wikilabs (like Merlissimo), but for most users it will be too complex.
Toolserver is in fact hosted by the Wikimedia Foundation today, in our Amsterdam data-center. We provide space, power and racks for the toolserver cluster, at a cost of about $65,000/year to WMF according to our Director of TechOps. We also maintain the database replication on our end which enables tools to function.
It is true that WMF hosts the TS and I am thankful for this. But these costs will not go away when the toolserver dies and everyone move to labs – because than you have to pay for Labs. If "we created a mysql-user in the past which took 5 minute and never touched it again" is "maintain the database replication on our end " then it's also true. The ssh-tunnel to get the data is maintained by our side and most times the wmf-sysops are not able to even announce to us when the mysql-master changed (we will notice some hours later when our replag increase) or when they create a new wiki (we will learn weeks later when an user searches for an wiki in our database and can not find it).
We can't provide the same level of service for the toolserver infrastructure as we do for core operations, and it makes no sense for a chapter to build out the required staffing and expertise to do so (set up/maintain all or some of the aforementioned functions). Even with slightly increased investment, toolserver would always suffer from being second or third tier infrastructure.
It made sense for years and worked.
- We're not comfortable hosting the toolserver infrastructure as-is.
There are too many idiosyncratic aspects of its configuration; it has its own wiki, its own (closed source) version control system, its own (closed source) issue tracker. There are hacks like TUSC that we want to replace with better systems/services (e.g. OpenID/OAuth).
Because we use JIRA instead of Bugzilla and Fisheye (which is not a version control system BTW) instead of Gerrit the Toolserver has to die? Are you kidding? We get these system for free because River asked nicely, but if that is really such a problem we can move away from them. And I was at the tech-meeting in Berlin 3 or 4 years ago where some wmf-techs told me, that OpenID would be "next year" – as we can all see the WMF has other things to do because there is no OpenID yet. To be clear: We WILL use OpenID when it is availablem but we can not use what is not there!
So, what's next?
What will happen next: I will request a change of the budget for WMDE at the general member-meeting of the WMDE. If my change is accepted there will be some money to extend the toolserver in 2013 so it can run again properly. Somewhen in 2013 Wikilabs will be moved to late 2014 and the TS has to run for another year. If my change is not accepted, I will retire from the TS at 30. December 2012 and it is not longer my beer what will happen. Somewhen in January the TS will collapse, because of non-maintenance, a wild-running tool or a security problem no-one are there to fix. Later this year Wikilabs will be moved to late 2014. There will be no working plattform for at least two years.
There is also still the possibility that WMDE comes to senses and changes the budget-plan of their own, giving the toolserver the little money it needs.
Good night, DaB.