Dear toolserver crowd, especially dear WMDE members among you!
tldr; I am asking the WMDE members committed to the toolserver: What can be a satisfactory state of the server given this strange situation of transition?
A few thoughts about the current state of the toolserver. This e-mail is born because I'm not satisfied: I took over the management for this project at WMDE three months ago. The toolserver's performance is not better than before (that's my perception). And all I am doing is reacting to things - though I would like to be in a more active position. This e-mail is also directed to the WMDE members whom we promised to improve the toolserver's performance. (For transparence, this mail is on the public list in English.)
I would like to make some of my thoughts transparent:
First of all, I'm in an awkward situation with contradictory tasks: On the one hand, I am trying to do things to keep the toolserver running. On the other hand my job is to set up the roadmap for its abolishment. The first means to argue for investments into hardware. This entails to talk about rack space and additional possibilities for power supply, both of which are at their limits in the data centre. The second means to propose the date of shutting down the server which is so close that new investments seem a funny idea. Do we want new contracts with a data centre we are about to leave next year? Also, getting hardware there is a lenghty process as the data centre is far away and physically maintained by others (WMF staff) who don't live next door to it either. (To clarify: It's *not* that WMDE refuses to spend the money. I am simply wondering if and how my work makes sense.)
Secondly: I think that my task is fuzzy: that "the toolserver runs (better)". What is that supposed to mean? When is this goal attained? Has this ever been put in a way that it becomes an accomplishable task? I refuse to struggle with undefined tasks that I'll never get done in a satisfying way. I miss a plan! Some might say this is a community-driven project that doesn't work like this. It develops the way it develops and planning is impossible. But I have spent quite some time thinking about questions like these:
* Has there ever been a systematic planning of resources for the toolserver? * Is there any agreement about growth and limits of the toolserver? Does it grow with growing demands? I think I implicitly understand the toolserver is supposed to grow infinitely. This makes it hard to stabilize the server: We only learn about its growth when the admins inform us that the system hits its technical limits. (They did inform us, but the future of the project was unclear already then.)
Thirdly, I have been thinking about how to go on. * Looking at my own roadmap I think this is the moment to stop the toolserver from growing. We might rather want to talk about strategies to improve performance by systematically cleaning up what has moved to Tool Labs. Or by cleaning up the remains of about 300 inactive accounts to free space. * When does the toolserver no longer accept new accounts? Tool Labs shall be ready in weeks. Once it's there, I think it would be fair to send new people to Tool Labs right away.
To cut a long story short: I am not sure what you expect me to do in this situation. Please clarify! I don't want to deal with a task without a clear task.
Best, Silke
Hello, At Saturday 27 April 2013 17:19:29 DaB. wrote:
To cut a long story short: I am not sure what you expect me to do in this situation. Please clarify! I don't want to deal with a task without a clear task.
as a WMDE-member and toolserver-admin I would like to answer you.
I expect you to represent the toolserver against your boss, Pavel. And I expect that you do it in favor of the toolserver and get us as much support (in money, man-power and other things) as possible. I also expect you that you can differ between the part of your job that should help the toolserver and the part of your job that has the goal to destroy the toolserver; I do expect you to not do propaganda for Labs or ToolLabs (neutral information is ok).
The goal of the toolserver is to help the Wikimedia-projects with tools. For this it provides a stable place for tool-authors for hosting their tools (before the toolserver tool-author had to host their stuff themself). There is no limit for this, because the Wikimedia-Projects have no limit too and continue to grow. The tool-authors expect that the toolserver is (more or less) stable, free, fast, maintained and that they will get help if needed. They do not like downtimes, replags, changing of rules and to document their stuff. Somewhere in the middle is the toolserver (we are free, somewhere stable, not very fast, more or less maintained, offer help if possible, have downtimes, much replag from time to time and change our rules sparely). Your job is to help to move the toolserver nearer to the expectations of the users with buying hardware for the toolserver (for fastness and against replag) and hire admins (for offering help and do maintenance). Every investment in the toolserver is a good investment because it helps the Wikimedia-projects. And it doesn't matter for how long the toolserver will exists – if you hire a person for 1 year that's more expensive than 1 server and after the year you have the hardware as a bonus.
And no, there was never a plan how the toolserver should grow; we always played by ear used how little money we had. In a ideal world WMDE would had read the toolserver-mailinglist and, if the complements of tool-authors increased, asked the admins how the cluster can be extended (in a VERY ideal world WMDE would had read the Wikipedia looking for complements of tool- USERS). But this is not a ideal world – but WMDE wasn't able to help even if the admins ASKED themself. So why creating a long-time-plan if WMDE is not able to fix even short-time-problems?
To conclude this mail I come back to "exceptions" in the form what I expect from WMDE: Nothing. I do not think anymore that WMDE has the goal to support the toolserver. All WMDE cares for is to move the tools (with or without tool- authors) to Labs as fast as possible, investing as few as it can in the toolserver, and shutdown the toolserver as early as possible.
Sincerely, DaB.
P.S: And I understand that it is just your job.
Op 27-4-2013 18:00, DaB. schreef:
Hello, At Saturday 27 April 2013 17:19:29 DaB. wrote:
To cut a long story short: I am not sure what you expect me to do in this situation. Please clarify! I don't want to deal with a task without a clear task.
as a WMDE-member and toolserver-admin I would like to answer you.
I expect you to represent the toolserver against your boss, Pavel. And I expect that you do it in favor of the toolserver and get us as much support (in money, man-power and other things) as possible.
Hear hear DaB! We're in a classic IT situation: The "old" system (in this case the Toolserver) is falling apart and the "new" system (labs) is far from ready. Let's not make the age old mistake of trying to move faster to the new system. I tried labs and I hate it, if someone tries to force it on my now I'll just shut down my tools and leave. I'm afraid I wouldn't be the only one.
Maarten
On 04/28/2013 04:42 PM, Maarten Dammers wrote:
I tried labs and I hate it, if someone tries to force it on my now I'll just shut down my tools and leave.
Even more useful would be a list of actual issues you've encountered, so that we may address them in the future.
That said, "far from ready" is neither fair nor very accurate; unless your tool requires access to the database dumps - which are coming - the tools project /is/ ready and quite functional; and there are already several tools running there.
-- Marc
Hello, At Monday 29 April 2013 01:11:46 DaB. wrote:
On 04/28/2013 04:42 PM, Maarten Dammers wrote:
I tried labs and I hate it, if someone tries to force it on my now I'll just shut down my tools and leave.
Even more useful would be a list of actual issues you've encountered, so that we may address them in the future.
Even more useful would have been a "We improved since last time, please come and check again".
That said, "far from ready" is neither fair nor very accurate; unless your tool requires access to the database dumps - which are coming - the tools project /is/ ready and quite functional; and there are already several tools running there.
Until now I never found a tool-labs-URL in the wild. Is there a list of what you are hosting?
-- Marc
Sincerely, DaB.
On 04/28/2013 07:15 PM, DaB. wrote:
Even more useful would have been a "We improved since last time, please come and check again".
Well, I /do/ take some pains to post regular updates, on labs-l and here as well. :-) Watch for emails from me (generally on Mondays) on those lists.
Also, [1] and [2] are good sources of information and generally kept pretty well up-to-date. I remember Silke and Sumanah having pointed at both at regular interval.
Until now I never found a tool-labs-URL in the wild. Is there a list of what you are hosting?
http://tools.wmflabs.org/ displays a list of publicly visible tools, I know there are a couple more being set up that aren't yet ready for prime time.
-- Marc
[1] http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_fea...
[2] http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Help
Hi Marc,
Op 29-4-2013 1:22, Marc A. Pelletier schreef:
On 04/28/2013 07:15 PM, DaB. wrote:
Even more useful would have been a "We improved since last time, please come and check again".
Well, I /do/ take some pains to post regular updates, on labs-l and here as well. :-) Watch for emails from me (generally on Mondays) on those lists.
My point was the *now* part. I'm confident that labs will become awesome, stable and much better than the Toolserver is right now in some point in the future, but it isn't right now. Moving to early gives bad user experience and gives disappointed users who turn their back on it (like me).
Maarten
On 04/29/2013 03:12 AM, Maarten Dammers wrote:
Moving to early gives bad user experience and gives disappointed users who turn their back on it (like me).
That may well be the case; I have been careful to invite early adopters who had tools that had full support for now (i.e. do not require DB replication). Current feedback is mostly "It's a bit more complicated to set up, but rock solid."
That said, I still have no idea what bad user experience you may have had, or what functionality you needed that was not available to you. Unless you are a bit more explicit about what the issues /were/, it's very difficult to address them.
-- Marc
On Fri, Apr 26, 2013 at 4:25 AM, Silke Meyer silke.meyer@wikimedia.de wrote:
- When does the toolserver no longer accept new accounts? Tool Labs
shall be ready in weeks. Once it's there, I think it would be fair to send new people to Tool Labs right away.
OH? They finally have the cross db-join situation sorted?
Which can be argued is one of the main features of the TS.
Silke Meyer silke.meyer@wikimedia.de wrote:
Dear toolserver crowd, especially dear WMDE members among you!
tldr; I am asking the WMDE members committed to the toolserver: What can be a satisfactory state of the server given this strange situation of transition?
[...]
I would start with fixing the current state. For example, every few days, apparently the LDAP server crashes. This sucks big time, as many (all?) other services like the mail system depend on them and get stuck. Nagios/tsnag shows various issues as well. They need to be addressed and are not.
While searching for another topic, I found no claims outside of the Toolserver universe that long-running read queries on slaves increase the replication lag per se, yet we do this. This needs looking into.
Any discussion of enhancements to the Toolserver is hindered by the lack of transparency that surrounds it: Nagios com- plains of too little disk space, yet Nosy recently added some. So do we need more disks, or do we have enough disk space, but it's just sitting idle?
How much does additional hardware and its installation cost? Is it € 1.000,-? € 10.000,-? € 100.000,-? We (WMDE) spend a lot of money on crap, so a reasonable investment that im- proves quality of service is well justified (and was dis- cussed ad nausea and *approved* by the general assembly). But for example Sebastian was tasked to investigate what servers to purchase so they could be reused by WMF after the Toolserver shutdown. I haven't seen any followup to this. Any investment in hardware also should keep in mind that the foundation gives away decommissioned servers for *free* *every* year just because new servers reduce their costs, so we shouldn't be too shy about it.
Tim
Hello, If I think positive, I have the hope that we can move relatively shortly the 20% of tools to labs that are producing 80% of our load. So than we should come back in a stable situation on toolserver with acceptabel performance. This "stable time" we can use than to run tools until they are obsolete or moved to labs.
For now you should analyse the system and predict what we will need in such or an other scenario. It can be that we need so or so some new database servers, some more storage or whatever. If you need place and power in the rig, replace old stuff.
In the moment it can also be helpful to be hard and kill some bots/tools with a relative bad cost-benefit-relation (Perhaps every user can also check this for it's own, because it's hard to decide from the outside.). A statistic with the loads of tools would be nice for this.
The actual situation of Toolserver is a shame for Wikimedia Germany! One actual example is the zoom-viewer[1], it's a tool that is "in production" on commons. Yesterday I want to use it, but it was unusable slow. I was so frustrated, but after months with tons of these problems on the server I feel that it makes no sense to fight against the situation, if WMDE decide to invest nothing. It's nearly impossible to develop on this server in the moment, because you get no felling for the performance of your changes in simple request can take 30sec or so. Again, only a shame!
For the long term I (as WMDE member) would support Toolserver to win flexibility and in-dependency. I learn that decisions from WMF can take months or years. Projects sponsert by WMDE (like my "Multilingual Maps"[2]) die perhaps because there is no server available.
WMDE has money and it's relatively cheap to invest money into techniques compared to pay persons in high-wages Germany. I don't believe that we lose only one donor if we spend money for a server system.
It would be nice if someone could explain me why it should be useful and secure for free knowledge to collect everything on one system. Yes, I understand the financial aspect but I also know the fire in the Library of Alexandria and the Nazi book burnings.
With more than one system in the world the developers can decide where the best environment and best support is for there work. This option will motivate all to increase the quality of service. With only one system we have to live with some decisions and also WMFlabs will become old over the years and better systems will come.
Greetings Tim alias Kolossos
[1] http://toolserver.org/~dschwen/iip/wip.php?f=Dresden-Neumarkt-Kulti-2013-04.... [2] http://wiki.openstreetmap.org/wiki/Multilingual_maps_Wikipedia_project
toolserver-l@lists.wikimedia.org