On Tue, Sep 25, 2012 at 8:27 AM, ralf@ark.in-berlin.de wrote:

, I cloned my applet page on github as backup

also who is backing up github? please share important urls. thanks mike

On Tue, Sep 25, 2012 at 11:10:13AM +0200, Platonides wrote:

On 25/09/12 00:51, DaB. wrote:

...

Hello all,

in these days WMDE (the chapter that finance the toolserver) is discussing the budget for the next year (2013); you can find it at [1]. At the moment there is no money for new toolserver-hardware in this budget and the CEO Pavel Richter is unwilling to change this ([2] in german) – because he fears that there will be a Wikilabs in 2014. It is not possible for me to run the toolserver for another year with the current hardware – you all know why. For this reason I will request a change of the budget at the general meeting at November, so there will be a vote about. If this vote should fail (and we get no money for new hardware), I am going to retire from my job as root at 30. December 2012. I'm not longer able to tolerant the behavior of the german chapter and the WMF in matter of the toolserver; I do this for free and for fun, and it is not longer fun.

Sincerely, DaB.

So, out of fear that there may be a better alternative in two years time, he decides to stop supporting it now?

I know labs, and I'm not sure it will really be a better alternative. Currently, it isn't. The toolserver is much more reliable and flexible. I don't think labs will "win" in the near future, either. Although it might change in two years. Specially if no attention is payed to the toolserver for that time.

I see two big problems:

1. If labs really becomes the "perfect tool hosting" in 2014, What

happens before we reach that? "Yes, your tool doesn't work, migrate to labs next year"

2. We risk ending up with no good alternative at that time. labs is

still not good enough, toolserver has degraded so much it's unusable.

Not to mention the migration cost that would need to be payed by tool authors if forced to move. Although perhaps he doesn't care.

I think we should help DaB to make a case why toolserver is important. We should list the projects depending on toolserver and how, and all the other things that are important. What I can quickly think of:

- Wiki Loves Monuments: All the tooling around this really need toolserver. - Commons: commons has a lot of toolserver tools more or less integrated in the interface, and there are supporttools running. - GLAM: a lot of glam-related work is done on toolservers - small tools for wiki support: all the bots archiving talkpages, create administrative pages automatically etc, etc ... - steward support: There are a lot of tools (also used generally) that really make cross-wiki abuse detection and other stewardwork much easier.

With a document like that we can make a stronger case for WMDE to reserve budget, or have them apply for a separate grant to the foundation.

Regards,

Andre

...

P.S: If you are in a board of a chapter that gives money to WMDE for the toolserver: Make sure that it will be spend for hardware.

[1] http://meta.wikimedia.org/wiki/Wikimedia_Deutschland/2013_annual_plan_draft/... [2] meta.wikimedia.org/wiki/Talk:Wikimedia_Deutschland/2013_annual_plan_draft/de#Toolserver

---

Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

---

Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

On Tue, Sep 25, 2012 at 02:15:35PM +0200, Delphine Ménard wrote:

Hi Andre,

As Pavel states, we are also looking at the idea of maybe making improvements that can then be handed over to Labs. Note that this might be complicated, as this would mean buying hardware in Europe that would then somehow have to be administrated from the US, so we are not sure what such a scenario might look like and if it is feasible at all.

Just as a very quick comment to this one, the foundation already has hardware in europe, administrated from the US, even in the same datacentre as toolserver is now, so those structures do exist.

Regards,

Andre

On 25 September 2012 14:01, DaB. WP@daniel.baur4.info wrote:

You are right in 1 thing: WMDE has no plan to abandon the toolserver, it has already abandon it. In 2012 there was no investment in new server and I very doubt that there will be any new servers this year. You just finance the "running", meaning that WMDE pay replacement parts if something breaks and you pay the loan of Nosy. But that is not enough: The load is growing. The toolserver has more users like in 2011, more tools like in 2011, and more people use this tools than in 2011; but there is no "more toolserver". That was bearable in 2012, but it will definitively not possible in 2013 because we have exceeded our possibilities.

Am I correct in thinking that the WMF gives WMDE a grant for the running of the toolserver, or was this just a one time thing back in 2009 ( http://meta.wikimedia.org/wiki/Grants:WM_DE/Improve_toolserver_reliability)?

If it's regular, does anyone know the details of it (is it annual, how much is it, is there a breakdown of how it is expected to be spent etc)?

Actually, is there a page with a list of all the financial (and in-kind) support that is provided to the Toolserver, by the WMF, chapters and individuals?

THO

On 25 September 2012 14:15, Ariel T. Glenn ariel@wikimedia.org wrote:

It might be helpful to put together a list of functions that the toolserver supports but that labs currently does not; such a list could serve as a basis for talks with the WMF. Perhaps the labs folks could makes some guesses at when those functions would be available and stable there, which would give everyone a better idea about how long the transition would realistically take.

If I am not mistaken, one of the big items is the ability to run expensive db queries without impacting production. I don't believe this is possible from labs right now, and I'm not sure what their plans are for that.

Ariel

p.s. this post is by me as a former toolserver user, having nothing to do with my status as a wmf staff member etc.

There is a partial list at http://www.mediawiki.org/wiki/Wikimedia_Labs/Toolserver_features_wanted.

According to the milestones at http://www.mediawiki.org/wiki/Wikimedia_Engineering/2012-13_Goals#Milestones..., we should be expecting database replication from production and user databases in January-March 2013.

Delphine and Ariel have already showed the approach to follow. https://www.mediawiki.org/wiki/Wikimedia_Labs/Toolserver_features_wanted_in_Tool_Labs is indeed very lacking; it should be expanded and each item moved to bugzilla. Tool Labs is currently not even on the radars, as far as I understood; some schedule is needed from the WMF.

Given the tone of this thread, I also want to point out that WMDE doesn't seem to have any fault to me here: it was the WMF's (sudden) decision, a year or two ago, to "replace" (aka kill) the Toolserver rather than helping it (after the one and only little grant in 2009) or letting other chapters help it. It obviously doesn't make sense to invest much on the Toolserver while the WMF is spending millions to duplicate it. Again, a document like the one Delphine requested would also help WMF understand what's needed to make the transition less traumatic: they need a clear plan and it's their responsibility to coordinate better with the service the state to be replacing, including financial aid. The Funds Dissemination Committee doesn't have any role about the part of WMF's budget which includes Labs, but they'd supposedly do something about such a problem if they want their work to mean anything IMHO.

Nemo

Am 25.09.2012 20:48, schrieb Erik Moeller:

Toolserver is in fact hosted by the Wikimedia Foundation today, in our Amsterdam data-center. [..] We also maintain the database replication on our end which enables tools to function.

I don't know all internal systems, but i think by maintaining you mean: grant access to mysql binlogs, traffic costs and sometimes creating a dump.

Why can WMF not administrate the hole database replication servers for toolserver users in short-term if WMDE should not spend money on this anymore? Setting up new replication servers at production system is done quite often. Adding views for hiding private data and adding access control based on toolserver ldap should be possible. The rest of the toolserver infrastructure won't be touched by this change.

Currently the replication of database cluster of s3/s6/s7 (all on server hyacinth) is lagging for more than an hour, performance is very low and complex queries are taking 10 times longer than normal, so that some of my queries can not finish within maximum allowed runtime (which brakes some of my tools since about five days). To solve this problem new hardware is need. I as toolserver user don't care if support comes from WMDE or WMF as long as this problem is fixed. I am the one with the oversized user talk page because other authors asked me why my tools are not working.

Merlissimo

Hello Erik, At Tuesday 25 September 2012 22:24:33 DaB. wrote:

Hi all,

let me weigh in with a few initial comments, and I'll ask the Labs folks to participate here and on Meta as well with regard to technical questions.

The initial focus for Labs has been to provide functionality that toolserver doesn't - get root on a VM or set of VMs to install/test arbitrary software/services, and get it ready for production deployment.

It is nice to have root on a (virtual) machine, but I doubt most tools need it. I would also bet that most tool-authors should not have root-access and should not be able to install software. I guess what most people and the WMF and also WMDE does not understand is: Most tool-authors are no system-ops or have a Master of Informatic Science. Most are amateur programmers who have fun to code a tool and do not maintain it once it is done (they also not document it). The tools they are write a slow and need way too much resources. They have no backup of their ssh-keys and do not extend their accounts in time. And all this is ok. Because (Nosy and) I am there to give them a infrastructure they can use, kick them a little bit if their tools misbehavior or really need an update, extend their accounts and add new ssh-keys. Some times this job sucks, but most times it is a good feeling to know that other people use these tools and it helps the wikimedia- projects. I very doubt that ANY sysop at WMF would do that (can you imagine Domas helping an user to create its first ssh-key (and the second shortly after because the user commit the private key instead of the public)?). I'm very sure that I have users on the TS that are able to use Wikilabs (like Merlissimo), but for most users it will be too complex.

Toolserver is in fact hosted by the Wikimedia Foundation today, in our Amsterdam data-center. We provide space, power and racks for the toolserver cluster, at a cost of about $65,000/year to WMF according to our Director of TechOps. We also maintain the database replication on our end which enables tools to function.

It is true that WMF hosts the TS and I am thankful for this. But these costs will not go away when the toolserver dies and everyone move to labs – because than you have to pay for Labs. If "we created a mysql-user in the past which took 5 minute and never touched it again" is "maintain the database replication on our end " then it's also true. The ssh-tunnel to get the data is maintained by our side and most times the wmf-sysops are not able to even announce to us when the mysql-master changed (we will notice some hours later when our replag increase) or when they create a new wiki (we will learn weeks later when an user searches for an wiki in our database and can not find it).

We can't provide the same level of service for the toolserver infrastructure as we do for core operations, and it makes no sense for a chapter to build out the required staffing and expertise to do so (set up/maintain all or some of the aforementioned functions). Even with slightly increased investment, toolserver would always suffer from being second or third tier infrastructure.

It made sense for years and worked.

2. We're not comfortable hosting the toolserver infrastructure as-is.

There are too many idiosyncratic aspects of its configuration; it has its own wiki, its own (closed source) version control system, its own (closed source) issue tracker. There are hacks like TUSC that we want to replace with better systems/services (e.g. OpenID/OAuth).

Because we use JIRA instead of Bugzilla and Fisheye (which is not a version control system BTW) instead of Gerrit the Toolserver has to die? Are you kidding? We get these system for free because River asked nicely, but if that is really such a problem we can move away from them. And I was at the tech-meeting in Berlin 3 or 4 years ago where some wmf-techs told me, that OpenID would be "next year" – as we can all see the WMF has other things to do because there is no OpenID yet. To be clear: We WILL use OpenID when it is availablem but we can not use what is not there!

So, what's next?

What will happen next: I will request a change of the budget for WMDE at the general member-meeting of the WMDE. If my change is accepted there will be some money to extend the toolserver in 2013 so it can run again properly. Somewhen in 2013 Wikilabs will be moved to late 2014 and the TS has to run for another year. If my change is not accepted, I will retire from the TS at 30. December 2012 and it is not longer my beer what will happen. Somewhen in January the TS will collapse, because of non-maintenance, a wild-running tool or a security problem no-one are there to fix. Later this year Wikilabs will be moved to late 2014. There will be no working plattform for at least two years.

There is also still the possibility that WMDE comes to senses and changes the budget-plan of their own, giving the toolserver the little money it needs.

Good night, DaB.

Hello, At Wednesday 26 September 2012 11:54:20 DaB. wrote:

Access to the wmf wiki replicated dbs is public across the entire wmflabs network so that's a given within the toollabs project as well.

so the entire labs-cluster use the same database-servers? Do you know that we have users that run queries that took hours/days to complete (because there are complex, for academic research or just because there is bug in the query)? And while I read about user-databases in another mail: While the toolserver has a dedicated sql-server for that (which REALLY needs replacement/extension), many user-databases are on the replicated-db-servers (users LOVES joins); some are hundreds of MB big (you can imagine how many rows they have). Does labs support that too?

Sincerely, DaB.

On 25/09/12 20:48, Erik Moeller wrote:

1. WMF is a technology organization. Hosting the core infrastructure

for Wikimedia projects is very much what we do. This includes data center operation, monitoring and backups, software deployments, software/service upgrades, code versioning infrastructure, bug tracking infrastructure, additional support systems and services (like this mailing list), etc.

Toolserver is in fact hosted by the Wikimedia Foundation today, in our Amsterdam data-center. We provide space, power and racks for the toolserver cluster, at a cost of about $65,000/year to WMF according to our Director of TechOps.

Something we should all be grateful for.

We also maintain the database replication on our end which enables tools to function.

A replication which WMF is already doing for itself, adding an extra slave hosted by a trusted party doesn't make a techical difference. While WMF gains a backup server (at some point the toolserver was the only externally hosted backup). It has also inadvertedly dropped the binlogs needed by TS when doing server cleanup.

So while very welcome, the WMF side of the db replication is not the most important piece of the toolserver (having a db copy still makes the TS better than any other alternative available at the time, though). The value is probably in how WMDE was able to open that database and many fruitful tools emerged.

We can't provide the same level of service for the toolserver infrastructure as we do for core operations, and it makes no sense for a chapter to build out the required staffing and expertise to do so (set up/maintain all or some of the aforementioned functions). Even with slightly increased investment, toolserver would always suffer from being second or third tier infrastructure.

labs is also a second class citizen.

Moreover, it is explicitely stated not to be for production-like level.

What will happen if a really successful tool reaches to a point where it de-facto needs it ? (eg. a WLM tool, tools to Request to get an Account Created, or Request an Unblock appeal...)

2. We're not comfortable hosting the toolserver infrastructure as-is.

There are too many idiosyncratic aspects of its configuration;

it has its own wiki,

Since when is that a problem? To take an obvious example, labs also began by making its own wiki, instead of incorporating itself into an existing one.

its own (closed source) version control system,

The vcs isn't closed source, it's just plain subversion.

The *repository viewer* is closed source, although with a an Open Source license. I don't know if people really use it too much.

its own (closed source) issue tracker.

I'm not a fan of jira, but this is the silliest reason to drop the toolserver :)

There are hacks like TUSC that we want to replace with better systems/services (e.g. OpenID/OAuth).

OpenID nor OAuth are currently available. If they were, TUSC wouldn't have been developed in the first time. When they appear, they will -slowly at first- take over TUSC, as it should. This ball is at WMF side.

Not to mention, if these «idiosyncratic aspects» were really a problem, they could easily be changed.

Chapters are autonomous organizations, and it's WM-DE's call how much / whether it wants to continue to invest in infrastructure of any kind (and the decision of funding bodies like the FDC to accept or reject that proposition). However, for our part, we will not continue to support the current arrangement (DB replication, hosting in our data-center, etc.) indefinitely.

This sounds as forcing them to go this route.

The timeline we've discussed with Wikimedia Germany is roughly as follows:

• Wind down new account creation on toolserver by Q2 of 2013 calendar year
• Decommission toolserver by December 2013

WMF can't commit to providing technical support for tool transition (there are too many tools), so if there's any area where I think it would make sense to ramp up investments on WM-DE's part, it's in engineering capacity to support tool developers in porting tools to Labs.

So the tool authors are "on their own", while forced to move out. What is the WMF *offering*?

That said, there may be a need for emergency purchases/investments to keep TS in a usable state until December 2013 (and perhaps allow for some buffer room beyond that). That's not our call to make.

But you are refraining WMDE from investing to it now. Would for instance WMF be willing to lend a few servers to the toolserver until December 2013?

On 25/09/12 23:38, Federico Leva (Nemo) wrote:

Indeed the first answers on https://www.mediawiki.org/wiki/Wikimedia_Labs/Toolserver_features_wanted_in_Tool_Labs seem to indicate that all tools and services currently on Toolserver will just be trashed and people forced to reinvent them from scratch because a completely different approach is thought to be better.

Nemo

Both approaches have its merits. The WMF could have copied the toolserver recipe much more easily than gooing the route of labs. They were brave pursuing the VM based system, and it allows solving different problems. But both of them have their use. It's not a case where one should "win" the other, or the toolserver is "bad" because it is currently hosted by WMDE instead of WMF.

Organizations trying to defeat one another will end up with one big loser: the community.

Hello, At Wednesday 26 September 2012 00:40:24 DaB. wrote:

+1. If people really love the Toolserver way of doing things, they are more than welcome to re-create the environment as a Labs project.

I doubt that you understand the toolserver. The toolserver-CLUSTER is make of 18 different servers. Some run Solaris, some run Debian, some are (big) database-servers, some are userland-servers, some are web-servers, some are HA-nodes and some are aux-servers (forgetting such details like virtual machines, network-devices and external storage). It's a grown infrastructure and you can not just move it to labs or rebuild it there as a simple virtual machine or cloud-instance. Even if I would begin this very instance I doubt I would be finish in December 2013; and it would need the same 18 servers (or better: more).

Sincerely, DaB.

P.S: There is no OSM in the 18 servers. OSM is again completely difference (Postgres vs. mysql for example).

Hello, At Wednesday 26 September 2012 12:07:17 DaB. wrote:

...

Even if I would begin this very instance I doubt I would be finish in December 2013; and it would need the same 18 servers (or better: more).

The database servers (replicated and user-databases) are going to be provided by the Labs infrastructure. That's the majority of Toolserver's servers. We have shared storage provided already.

Let me count: -2 userland-server (linux) -1 userland-server (solaris) -2 web-servers (solaris) -1 web-serever (linux) (for testing) -2 HA-nodes -2 aux-servers -1 server for the roots -1 user-db-server -5 replicated db-servers

6 vs 11 – clearly the majority ;-)

The capacity of Labs is far greater than Toolserver's. We have a cluster in the pmtpa and eqiad datacenters. Each cluster has 7 compute nodes for virtual machines, 2 database servers for replicas, 1 database server for user databases and 4 storage nodes for shared storage access.

So you have 2 database servers for 7 wmf-clusters? You are poorer than we are.

It's definitely possible to re-create Toolserver inside of Labs. I'm not suggesting that's what should be done for a migration, but it's an option.

So who is helping me to install solaris? Who is helping me to port our self- written software over? Who is helping me to change the self-written scripts? Who is helping me to convert our jira-bugs into your bugzilla? Who will update all the wiki-pages?

The toolserver is not just a simple Debian with a puppet-system running; its much more complex. And while I'm sure that I could re-build something similar in labs in a great time-window, it would just be something similar, not the same; many tools would break (if you have time, take a look in our JIRA for things that are not working on Debian, but on work on Solaris).

Sometimes I have the feeling WMF and WMDE thinks that we can shutdown the TS on a Friday, rsync the stuff at the weekend, power-up labs at Monday and everything will work….

• Ryan

Sincerely, DaB.

On 26/09/12 21:20, Ryan Lane wrote:

Labs is an infrastructure for building infrastructure. It isn't a Toolserver replacement.

• Ryan

Yet at high level it has been treated as such replacement. We could also replace the Wikimedia servers with turing machines.

Toolserver's way of doing things is very individualized. It doesn't promote collaboration, and it leads to tools disappearing when a user's account is disabled.

Wrong. Do you know MMP https://wiki.toolserver.org/view/MMP ? Toolserver already allows collaboration as you describe it. What you seem to be saying is that you want to *force* people to collaborate: are you sure this works? I'd rather suggest to investigate what works or not in the MMP (very simple) model and why people don't use it more in Toolserver, so that you can build on that experience and understand better the users' needs.

Nemo

Yes, I know about MMP. It was introduced much later, after people were already very used to doing things in an individualized manner. From what I'm told MMPs aren't used much.

So you think that even new users (I believe many users arrived after MMP were introduced) are not using MMP due to bad old habits of the other users? I've had a hard time trying to convince some users to use a MMP; I think it's not trivial to understand what models actually work and are liked.

Toolserver's model is fundamentally different. It's based on an old concept of shared hosting. Labs is built on a model more like a VPS (really more like EC2). Due to that, it's possible to give users far more rights. MMPs aren't a very elegant solution. I'd prefer not to force an inelegant solution into a system that allows much more elegant approaches.

I'm not a technical person but yes, I can probably understand that MMP is not very elegant. But, do you think users are not using MMP because they're not elegant?

Nemo

2012/9/26 Platonides platonides@gmail.com:

On 26/09/12 08:27, Federico Leva (Nemo) wrote:

...

...

Yes, I know about MMP. It was introduced much later, after people were already very used to doing things in an individualized manner. From what I'm told MMPs aren't used much.

So you think that even new users (I believe many users arrived after MMP were introduced) are not using MMP due to bad old habits of the other users? I've had a hard time trying to convince some users to use a MMP; I think it's not trivial to understand what models actually work and are liked.

The reason is as simple as being much more easier, run mkdir and start coding vs open a jira request to get a new MMP for a project which may or may not be interesting (heh, most tools were probably born after coding for a couple of hours after having a good idea) and which nobody is wanting to maintain with you, probably.

It is not hard to start coding in your own workspace and move code from your newly created directory to a MMP. A MMP doesn't necessarely have to be maintained by multiple users. While it's convenient having someone in the project that can jump in in case your tool breaks badly and you are not around to fix it, it is not by any means necessary.

Ryan wrote:

...

Toolserver's model is fundamentally different. It's based on an old concept of shared hosting. Labs is built on a model more like a VPS (really more like EC2). Due to that, it's possible to give users far more rights.

labs model didn't exist when the toolserver started. This route was the only 'normal' one to follow, specially with a single server. It even looks too new for labs right now.

...

If you guys want to build the exact same Toolserver environment as a Labs project, go for it. I have a good feeling you'll start doing things differently when you see the affordances given by having more rights, though.

I have a few ideas on how to improve it on webtools, based on toolserver experience, but without big changes.

...

MMPs aren't a very elegant solution. I'd prefer not to force an inelegant solution into a system that allows much more elegant approaches.

Actually, I'm unsure on how to replicate MMP accounts there. It shouldn't need to manually create them everywhere. The elegant solution is probably to have those accounts in LDAP... Any ideas?

---

Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

Another point to consider is that for some projects there is considerable interdependence (this may not be the norm). My WikiMiniAtlas for example depends on

1. the OSM database mirror being available 2. the WIWOSM project (although I could proxy that from the TS) 3. Dispensers coordinate extraction database (GHEL)

These dependencies will increase the viscosity of the migration process. In theory database connections could be ssh tunneled although

1. that may not be a very stable or fast solution 2. someone/somewhat is blocking ssh traffic from labs to the toolserver cluster (other ssh connections work fine) what is going on here?!

Dschwen

On 26/09/12 20:55, Ryan Lane wrote:

They can be created in LDAP by making a labsconsole account. Additionally, unless the account needs to directly log in via ssh, there's no request process needed for the user to be used.

Of course right now there isn't open registration in labsconsole. We're waiting on some Jenkins changes for that to happen. Should be any day now.

Alternatively, the user could be created as a system account via puppet.

• Ryan

I thought the jenkins changes were ready, after Antoine wrote:

I have finished the preparation work in Jenkins. It will be deployed as soon as we agree on the new workflow, which I still have to write down.

On Wed, Sep 26, 2012 at 01:59:51AM -0400, Ryan Lane wrote:

Of course, as I mentioned before, there's nothing forcing this new model at all. If you guys want to build the exact same Toolserver environment as a Labs project, go for it. I have a good feeling you'll start doing things differently when you see the affordances given by having more rights, though. Either path chosen, we'll help you with infrastructure issues along the way.

• Ryan

Hi Ryan,

This statement has been made before, but then you put all responsibility with the community, while the foundation can benefit as well. What people want and has appeared usefull, is a *supported* environment like toolserver where people can just get an account, run bots to support a wiki, experiment with simple tools, if needed work and support a tool together. This can then be a breeding bed for tools, and if needed brought a stage up, be documented and pacakged, so it can be rolled out as a wmf supported tool.

That the toolserver environment is supported is imho an essential. While volunteer admins can certainly help and do a lot of work, including helping non-unix persons to startup, if there are really problems with the environment, it better guaranties there is somebody available to acct.

Regards,

Andre Koopal

I think the vast majority of the people on the Toolserver don't need a lot of that extra "beefy" stuff. To run our tools, all we need is:

* A database providing replication from the Foundation's databases. * A web server capable of running PHP, CGI, and other web scripts. * A linux or solaris machine that we can run bots on. * A platform on which all of this runs reliably.

If any of us needed a Mediawiki installation, we'd have set up our own site for it years ago. Sure, I can see you wanting to make options like that available to developers, particularly those building extensions to MW, but what the Toolserver community - and really, all of the Wikimedia projects - need right now is something that meets those four bullet points. With the Toolserver down as often as it's up lately, and Labs not ready to provide any of that support until December 2013 (not to mention no assistance moving there), either a) the Toolserver needs funding so it can get back to working order, even if it means no further accounts or projects can be created on it, or b) Labs needs to get that stuff running much sooner.

And frankly, I don't see why WMDE should be footing the bill for all the transitioning work when the Foundation isn't providing them any support to do so while setting up a competing system at the same time.

---- User:Hersfold hersfoldwiki@gmail.com

On 9/25/2012 9:30 PM, Erik Moeller wrote:

On Tue, Sep 25, 2012 at 2:27 PM, Platonides platonides@gmail.com wrote:

Hi Platonides.

...

labs is also a second class citizen.

Moreover, it is explicitely stated not to be for production-like level.

What will happen if a really successful tool reaches to a point where it de-facto needs it ? (eg. a WLM tool, tools to Request to get an Account Created, or Request an Unblock appeal...)

Labs doesn't take options away in those cases -- it's just designed to make it a lot more straightforward to take a tool and integrate it well, if that's the best path to take for a particular tool.

It opens up additional possibilities:

1. It makes it easy to set up a MediaWiki instance (we're working on a

standard VM config for this to make it really straightforward) so you can prototype an extension, if that's a direction you want to take for a particular tool. E.g. some of the above sound like they should be ideally done as special pages.

I buy into some of the premise stated here (that a lot of tool developers really don't _want_ to do MediaWiki development), but I also think it's our job to make it as easy as possible to take software down the path that makes the most sense. In contrast, large web apps like MediaWiki are explicitly prohibited from being made publicly available per https://wiki.toolserver.org/view/Rules

2. It makes it possible to prototype a standalone service, puppetize

it, and prepare it for production deployment on allocated hardware.

For tools that aren't mission-critical and that are only lightly integrated, I don't view it as an issue for a tool to run at tools.wmflabs.org/some/cool/tool (or wherever it's going to live) indefinitely. (Ryan may want to chip in on that.)

...

So the tool authors are "on their own", while forced to move out. What is the WMF *offering*?

We're offering free access to very beefy infrastructure for Wikimedia-relevant purposes. We're in this for the long haul and intend to develop Labs into the best environment for testing/staging, bottom-up experimentation and tool development that we can possibly build.

We're not able to provide hands-on support for porting stuff over, but as Ryan suggested, it may be feasible for interested folks to build an environment in Labs that makes transition easier for toolserver users. The Labs team (which includes volunteers) is generally very responsive and helpful, within reason.

...

Would for instance WMF be willing to lend a few servers to the toolserver until December 2013?

WMDE has to decide what level of transitional support is appropriate and budget for it. As for support, we've kicked around the possibility of them purchasing some additional hardware and us buying it from them later and using it for other purposes, but we'd have to carefully work out whether that's feasible in practice.

Erik

On Tue, Sep 25, 2012 at 9:41 PM, Hersfold hersfoldwiki@gmail.com wrote:

I think the vast majority of the people on the Toolserver don't need a lot of that extra "beefy" stuff. To run our tools, all we need is:

• A database providing replication from the Foundation's databases.
• A web server capable of running PHP, CGI, and other web scripts.
• A linux or solaris machine that we can run bots on.
• A platform on which all of this runs reliably.

Excluding the database replication, all of these things are possible now.

If any of us needed a Mediawiki installation, we'd have set up our own site for it years ago. Sure, I can see you wanting to make options like that available to developers, particularly those building extensions to MW, but what the Toolserver community - and really, all of the Wikimedia projects - need right now is something that meets those four bullet points. With the Toolserver down as often as it's up lately, and Labs not ready to provide any of that support until December 2013 (not to mention no assistance moving there), either a) the Toolserver needs funding so it can get back to working order, even if it means no further accounts or projects can be created on it, or b) Labs needs to get that stuff running much sooner.

Database replication should be done in the next 2-3 months. It may be done sooner, but that's our current goal date. Brave community members can start building any other missing infrastructure.

And frankly, I don't see why WMDE should be footing the bill for all the transitioning work when the Foundation isn't providing them any support to do so while setting up a competing system at the same time.

I've never built Labs to be a competing system. Labs should have similar functionality, but I find the toolserver portions of Labs to be a small part of its function. My biggest goal with integrating Toolserver features is to bring our Toolserver community more closely together with our developer community. They are fairly disjoint right now.

WMDE offered to help with the transition work. The decision to move the community from Toolserver to Labs was made jointly between WMDE and WMF. It's not possible for WMF to handle building the Labs infrastructure and to transition the tools. We don't have enough Toolserver experience to do so properly anyway.

Toolserver was mostly built by the community from the start. Even WMDE working the transition won't be enough. We need people like you to help transition. My team will help you as much as needed, and will gladly answer any questions you have.

- Ryan

Pavel Richter wrote:

Wikimedia Deutschland is committed to run and maintain the Toolserver as

long as

Wikilabs are not in position to offer the same level of service as the

Toolserver. This

committment includes the purchase of new hardware, where necessary to

keep the

servers running. As in the past years, I hope for support by other

entities in this project,

both financially and with administration of the Toolserver.

@DaB: how much of an annual budget do you think the toolserver requires?

On Tue, Sep 25, 2012, Delphine Ménard notafishz@gmail.com wrote:

we can't at this stage enter in expensive improvements, because Labs is,

in the short to mid-run, destined to replace the "toolserver as you know it"

completely. It falls under the attributions of the Foundation, as hosting provider

of the Wikimedia Projects, to provide authors, developers and contributors

with the technical tools they need to make their work easier.

Why is this? The Foundation has always tried to provide some technical tools, but not all; it is not an exclusive job of the Foundation, and the toolserver in particular has often (quite often!) provided support that was not available anywhere else. It's good that more tools are being developed and maintained. But in my opinion we need more entities, not fewer, providing this sort of support.

And Ryan has said elsewhere that Wikimedia Labs is not intended to replace the toolserver. So why is WM-DE considering dropping the toolserver? And why is the WMF considering not providing db replication for it? I thought the goal was to make that easier, not harder.

SJ

On Sat, Sep 29, 2012 at 9:31 PM, Krinkle krinklemail@gmail.com wrote:

On Sep 29, 2012, at 9:41 PM, Samuel Klein meta.sj@gmail.com wrote:

...

And why is the WMF considering not providing db replication for it?

[citation needed]

I think you misunderstood.

Clearly.

SJ

On Sep 30, 2012, at 2:21 PM, Platonides platonides@gmail.com wrote:

On 30/09/12 03:31, Krinkle wrote:

...

On Sep 29, 2012, at 9:41 PM, Samuel Klein meta.sj@gmail.com wrote:

...

And why is the WMF considering not providing db replication for it?

[citation needed]

I think you misunderstood.

-- Krinkle

Written by Erik Moeller the 25th Sep:

...

Chapters are autonomous organizations, and it's WM-DE's call how much / whether it wants to continue to invest in infrastructure of any kind (and the decision of funding bodies like the FDC to accept or reject that proposition). However, for our part, we will not continue to support the current arrangement (DB replication, hosting in our data-center, etc.) indefinitely.

He is writing with his "WMF hat", the above 'we' refers to the WMF. So no, there's no misunderstanding: The WMF has threaten to stop providing the db replication. That would be a really bad move to do, of course (and a gratuitous one, since there aren't technical or financial reasons for doing that). And I hope they won't do that.

• http://lists.wikimedia.org/pipermail/toolserver-l/2012-September/005294.html

Ah, I understand the confusion now.

If I understand correctly Samuel was talking about the (future) labs environment, not the Toolserver. The above citation from Erik, however, is about the Toolserver (not Labs).

Erik is saying here that WMF will (at some unspecified point in the future) stop providing the services it is currently providing for Toolserver (such as the space in their datacenter and arranging db replication). It does not mean they will (or are considering to) stop providing db replication in general (just not to Toolserver).

Maybe Samuel already knew that and he was talking about Toolserver as well, in that case: never mind, sorry about the confusion. I just wanted to make sure you know that there are confirmed and scheduled plans for Wikimedia Labs to have a live db replication arranged between the labs cluster and the wmf production cluster. As far as I know there are no considerations to not provide this, quite the contrary.

-- Krinkle

On Sun, Sep 30, 2012 at 2:32 PM, Erik Moeller erik@wikimedia.org wrote:

On Sun, Sep 30, 2012 at 10:24 AM, Krinkle krinklemail@gmail.com wrote:

...

I just wanted to make sure you know that there are confirmed and scheduled plans for Wikimedia Labs to have a live db replication arranged between the labs cluster and the wmf production cluster.

That's correct.

The main difficulty at the moment is that the plans for WMF Labs don't seem to include database replication *in a form that makes it a useful, direct replacement for toolserver*. This is a subtle point that is easy for non-technical people to miss when they hear that replication will be available. The toolserver database replicas are useful because there are there, because they are easy to use, and because users can directly join their own databases against them to perform more complicated analysis and data gathering than would be efficient otherwise.

Reading about the plans for labs in the past few days, I have seen the following claims:

* "User databases will not be able to be joined against replicated databases." The reasoning behind this seems to be a misunderstanding of the role of "application logic". For Mediwiki itself, which works with only a few articles at a time, joins can be efficiently simulated within Mediawiki itself, or by making changes to the database schema on the live servers. For a toolserver application that works with millions of articles in a single query, it is silly to essentially re-implement a SQL engine in the application logic - joins of this size, which may require filesorts, should be done at the database server level, not at the application level. That's why we use a database server in the first place.

* "User databases will not be backed up directly, and users will have to back them up manually". This is again a huge step backwards in reliability, as most users don't have the time or experience to do reliable backups of their own databases. The utterly predicable outcome will be that one or more highly-used services will fail and have no backup.

* "User home directories will somehow be deprecated." A key function of the toolserver is "data analysis": users can simply run queries against the replicated database, process the results, and use them to plan things on the wikis. There is no "application" or "project" for this - it is essentially ad hoc manual work. This kind of data analysis could be done from a database dump, but then the data is far out of date. It can be done using api.pm on the live wiki, but that is prohibitively inefficient for queries that have to consider millions of articles.

Looking at the discussions about labs in the past few days, it seems clear to me that labs will be useful for some projects, particularly for developing Mediawiki extensions. But the plans seem to make it needlessly difficult for most of the things that the toolserver is currently used for. The current plans seem to be intentionally preventing toolserver users from simply migrating their tools to labs; the result will be a great leap backwards when/if the toolserver is taken offline. There is some ideological purity in that, but it will result in a huge loss of functionality for the actual wikis, which rely on the existing toolserver in many ways for normal, on-wiki functionality. For example, there are many links in the interface on enwiki to toolserver tools.

I do think it is silly for a WMF chapter to run the toolserver when it is really a vital part of the infrastructure for the live wiki projects. But the right solution is for the WMF to offer a convenient replacement that will not require an unreasonable amount of effort for migration.

- Carl

On Mon, Oct 1, 2012 at 8:12 AM, Platonides platonides@gmail.com wrote:

Note that it's false that MediaWiki doesn't perform joins. There are many of them. The statement about that was (probably) talking about cross-db joins.

Yes, that's right, I was talking about joins between user databases and the replicated databases containing the live wiki data. The quote I am thinking of was posted by Ryan Lane to toolserver-l on Sep 26. It said, "We currently have no plans for having the user databases on the same servers as the replicated databases. Direct joins will not be possible, so tools will need to be modified.".

As you have pointed out in a previous message, the reason behind many cross-db joins on toolserver is that users are unable to create tables within the replicated databases. That's expected - the replicated databases need to match the live ones - but the natural solution is to have user databases on the same servers.

This also leads to the problem you pointed out:

Even for MediaWiki extensions, not having user tables can be hard.

Yes, I agree. At [1], there is currently this claim about cross-db joins on labs: "Unlikely, that logic should be handled within the application. It's impossible to shared data otherwise, as well as the extra overhead on the database servers which are effectively a shared component. DamianZaremba (talk) 11:13, 27 September 2012 (UTC)" I am not sure what "within the application" means, but if someone is writing a Mediawiki extension and needs another table or needs a schema change, it is not obvious to me right now how they would be able to do a join between a user table for their extension and the replicated tables from the live projects. And those joins would certainly be "within the application".

But I expect that few toolserver users were/are working on mediawiki extensions. For non-extension projects that have a lot of data, using a database server is the only reasonable solution. Just as a point of data, the "logging" table for the WP 1.0 project on toolserver, which has records for article assessments on enwiki, has about 46 million rows in a user database.

- Carl

1: http://www.mediawiki.org/wiki/Wikimedia_Labs/Toolserver_features_wanted_in_T...

---

From: Samuel Klein meta.sj@gmail.com To: Wikimedia Toolserver toolserver-l@lists.wikimedia.org Sent: Saturday, September 29, 2012 10:41 PM Subject: Re: [Toolserver-l] Future of the toolserver

On Tue, Sep 25, 2012, Delphine Ménard notafishz@gmail.com wrote:

we can't at this stage enter in expensive improvements, because Labs is,  in the short to mid-run, destined to replace the "toolserver as you know it"

...

completely. It falls under the attributions of the Foundation, as hosting provider

of the Wikimedia Projects, to provide authors, developers and contributors with the technical tools they need to make their work easier.

...

Why is this?  The Foundation has always tried to provide some technical tools, but not all; it is not an exclusive job of the Foundation, and the toolserver in particular has often (quite often!) provided support that was not available anywhere else.   It's good that more tools are being developed and maintained.  But in my opinion we need more entities, not fewer, providing this sort of support.

And Ryan has said elsewhere that Wikimedia Labs is not intended to replace the toolserver.   So why is WM-DE considering dropping the toolserver?   And why is the WMF considering not providing db replication for it?  I thought the goal was to make that easier, not harder.

Samuel, you're obviously not on the same page as the executive staff at WMF (see Erik's email). Following this thread, it's becoming more and more clear that before any serious discussion of the toolserver's future, the WMF needs to get it's priorities straight. Do you or do you not want to replace the toolserver? If you do, it seems like a sensible thing to do to keep as much of the configuration as possible.

I've been looking at the labs in the last few days and it seems to me that it's architecture is overkill for doing simple things. I think more and more people will choose to go with their own hosting for robots instead of using the labs. This probably means money spent by the WMF in traffic, since those users will be using the api instead of accessing the database directly.

On Sun, Sep 30, 2012 at 8:09 AM, Samuel Klein meta.sj@gmail.com wrote:

I am not writing as a WMF trustee, but in my personal capacity as a community member interested in the toolserver. This is a fairly operational discussion, and not something discussed by the board; I only know about it because I read this list.

Thanks for clarifying that.

Erik

The Account Request process and the Unblock Request process on the English Wikipedia also run on Toolserver. Account requests is interesting, because the en.wikipedia community recently decided not to move this process back on-wiki: https://en.wikipedia.org/wiki/Wikipedia_talk:ACC#Account_Request_extension.2...

Not sure if it matters, but my web-based tools are located at https://github.com/Matthewrbowker/toolserver and my bots will be located at https://github.com/Matthewrbowker/toolserverBots .

Matthew Bowker

On Sep 25, 2012, at 4:45 AM, Andre Koopal andre@molens.org wrote:

On Tue, Sep 25, 2012 at 11:10:13AM +0200, Platonides wrote:

...

On 25/09/12 00:51, DaB. wrote:

...

Hello all,

in these days WMDE (the chapter that finance the toolserver) is discussing the budget for the next year (2013); you can find it at [1]. At the moment there is no money for new toolserver-hardware in this budget and the CEO Pavel Richter is unwilling to change this ([2] in german) – because he fears that there will be a Wikilabs in 2014. It is not possible for me to run the toolserver for another year with the current hardware – you all know why. For this reason I will request a change of the budget at the general meeting at November, so there will be a vote about. If this vote should fail (and we get no money for new hardware), I am going to retire from my job as root at 30. December 2012. I'm not longer able to tolerant the behavior of the german chapter and the WMF in matter of the toolserver; I do this for free and for fun, and it is not longer fun.

Sincerely, DaB.

So, out of fear that there may be a better alternative in two years time, he decides to stop supporting it now?

I know labs, and I'm not sure it will really be a better alternative. Currently, it isn't. The toolserver is much more reliable and flexible. I don't think labs will "win" in the near future, either. Although it might change in two years. Specially if no attention is payed to the toolserver for that time.

I see two big problems:

1. If labs really becomes the "perfect tool hosting" in 2014, What

happens before we reach that? "Yes, your tool doesn't work, migrate to labs next year"

2. We risk ending up with no good alternative at that time. labs is

still not good enough, toolserver has degraded so much it's unusable.

Not to mention the migration cost that would need to be payed by tool authors if forced to move. Although perhaps he doesn't care.

I think we should help DaB to make a case why toolserver is important. We should list the projects depending on toolserver and how, and all the other things that are important. What I can quickly think of:

• Wiki Loves Monuments: All the tooling around this really need toolserver.
• Commons: commons has a lot of toolserver tools more or less integrated

in the interface, and there are supporttools running.

• GLAM: a lot of glam-related work is done on toolservers
• small tools for wiki support: all the bots archiving talkpages, create

administrative pages automatically etc, etc ...

• steward support: There are a lot of tools (also used generally) that really

make cross-wiki abuse detection and other stewardwork much easier.

With a document like that we can make a stronger case for WMDE to reserve budget, or have them apply for a separate grant to the foundation.

Regards,

Andre

...

...

P.S: If you are in a board of a chapter that gives money to WMDE for the toolserver: Make sure that it will be spend for hardware.

[1] http://meta.wikimedia.org/wiki/Wikimedia_Deutschland/2013_annual_plan_draft/... [2] meta.wikimedia.org/wiki/Talk:Wikimedia_Deutschland/2013_annual_plan_draft/de#Toolserver

---

Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

---

Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

---

Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

Hello Pavel, At Wednesday 26 September 2012 12:31:52 DaB. wrote:

With regards to new hardware: Our committment here dos not mean that we will simply replace broken parts only; it also means that we will buy new stuff, so that the Toolserver can handle increased demand. We do have the ressources to do so, and the money is available within our annual budget.

so you told me last year and how many servers did we bought this year (quick help: less than 1)? All new hardware (with the exception of a simple switch and some memory) that was installed this year was bought in 2011. And I told WMDE again and again that we are short on hardware. Why is it so hard to just add a position to your budget-plan

Toolserver-Hardware xx,000€

? If we will not need all of it in 2013: fine. I will not point at the number and demand that all will be spend. The TS will not need much money (2 or 3 new database-server would be enough), compared with other projects (like Wikidata, Render oder the community-project-budget) or the personal-cost (and you know I ALWAYS said that we need personal and it is ok to spend money there!) the position would be very small. I just need to be sure that there is money at all.

Sincerely, DaB.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 06.10.2012 11:06, Dr. Trigon wrote:

So essentially MY BOTS DO NOT RUN ANYMORE BECAUSE OF THE TS OVERLOAD for at least 1 week!! I am up to stop my work here in wikipedia (de, en, commons, meta, ...) because I do not have the time and ressources to keep my bots running...

https://jira.toolserver.org/browse/TS-1534

should help as well... Thanks to everybody involved in helping me to come to that clue!!!

Greetings

On Wed, Sep 26, 2012 at 10:37:11AM +0200, Pavel Richter wrote:

Hi Andre,

2012/9/26 Andre Koopal andre@molens.org:

...

Just replying to the original mail here.

While the discussion what the labs project should look like is usefull, the immediate problem at hand is a bit forgotten, that toolserver might end early next year if there is less support.

Toolserver will not end early next year. Period.

Hi Pavel,

With all due respect, and I do believe your intentions are good, but if you can't convince DaB about this and he does quit, toolserver will break and stop. Although Nosy is still there, afaik she is still not up to date about all the details around toolserver, so there will be a lot of knowledge lost effectively ending toolserver.

Regards,

Andre