On Tue, Sep 25, 2012 at 12:51:50AM +0200, DaB. wrote: I support your decision and am looking forward to a solution of WMDE mismanagement, in general. In fact, after the last weekend, I cloned my applet page on github as backup, though their line is throttled down. Now can come what may! Best wishes, ralf

On Tue, Sep 25, 2012 at 08:42:19AM +0200, Mike Dupont wrote: Only important for WP chemistry authors (they were already informed): http://toolserver.org/~ayacop/EditorApplet.html = http://jchempaint.github.com/EditorApplet.html

Maybe someone can translate the German better than stupid google translate for me, but it sounds like he's willing to support the existing infrastructure but not go any further (as in upgrade or supply anything we need) because Labs is going to be 'the equivalent' of toolserver or better by 2014, supposedly. Honestly, right now I doubt many people on TS 1) want to move to Labs (because of no WMF database replication being the biggest complaint) 2) Understand labs enough to work it (I don't, someone has offered to teach me, but I have to find the time) 3) It takes a lot of time to migrate and test. We simply just don't have enough man hours with our real jobs/school to make this move over, especially for single person tools. DaB, I honestly support your decision if what I sad above is correct about the attitude, because I wouldn't want to work at toolserver either. No sysadmin wants to work with people constantly looking at them for why things will not work and then management who won't give the tools to do it. I'm personally starting to get my tools and talking with my MMP Co-Devs about moving over to labs, but one project needs the Antispoof table from WMF databases which labs doesn't have. So are we ending all support for it and scrapping the project because of issues with the toolserver not being funded enough to work? Never have I seen a more crazy proposal to end support for services that aren't working already for a volunteer community. DeltaQuad English Wikipedia Administrator and CheckUser ACC and UTRS Developer On Tue, Sep 25, 2012 at 5:10 AM, Platonides <platonides(a)gmail.com> wrote:

Is there any list of necessary items (minimal and optimal) together with estimated price so we can know what and how much we're talking about? That would be helpful for instance when looking for potential sponsors. Also, are we rather interested in couple brand new boxes or bunch of used is acceptable as well? Kind regards Danny B.

On Tue, Sep 25, 2012 at 02:15:35PM +0200, Delphine Ménard wrote: Just as a very quick comment to this one, the foundation already has hardware in europe, administrated from the US, even in the same datacentre as toolserver is now, so those structures do exist. Regards, Andre

On 25 September 2012 14:01, DaB. <WP(a)daniel.baur4.info> wrote: Am I correct in thinking that the WMF gives WMDE a grant for the running of the toolserver, or was this just a one time thing back in 2009 ( http://meta.wikimedia.org/wiki/Grants:WM_DE/Improve_toolserver_reliability)? If it's regular, does anyone know the details of it (is it annual, how much is it, is there a breakdown of how it is expected to be spent etc)? Actually, is there a page with a list of all the financial (and in-kind) support that is provided to the Toolserver, by the WMF, chapters and individuals? THO

On 25 September 2012 14:15, Ariel T. Glenn <ariel(a)wikimedia.org> wrote: There is a partial list at http://www.mediawiki.org/wiki/Wikimedia_Labs/Toolserver_features_wanted. According to the milestones at http://www.mediawiki.org/wiki/Wikimedia_Engineering/2012-13_Goals#Milestone…, we should be expecting database replication from production and user databases in January-March 2013.

The Account Request process and the Unblock Request process on the English Wikipedia also run on Toolserver. Account requests is interesting, because the en.wikipedia community recently decided not to move this process back on-wiki: https://en.wikipedia.org/wiki/Wikipedia_talk:ACC#Account_Request_extension.… Not sure if it matters, but my web-based tools are located at https://github.com/Matthewrbowker/toolserver and my bots will be located at https://github.com/Matthewrbowker/toolserverBots . Matthew Bowker On Sep 25, 2012, at 4:45 AM, Andre Koopal <andre(a)molens.org> wrote:

Hi all, let me weigh in with a few initial comments, and I'll ask the Labs folks to participate here and on Meta as well with regard to technical questions. The initial focus for Labs has been to provide functionality that toolserver doesn't - get root on a VM or set of VMs to install/test arbitrary software/services, and get it ready for production deployment. Labs doesn't have DB replication yet, and it doesn't yet have an environment optimized for the development of small tools that are not geared towards deployment in production. There are some communities within Labs, most notably bots.wmflabs.org, that have started to optimize their environment for certain categories of tools (in this case bots). The second phase of Wikimedia Labs is called "Tool Labs" and its explicit goal is to be an alternative to the toolserver. This is outlined here: https://www.mediawiki.org/wiki/Wikimedia_Engineering/2012-13_Goals#Mileston… and here: https://www.mediawiki.org/wiki/Wikimedia_Labs#Tool_Labs In other words, it's not suprising that Labs cannot yet function as an effective toolserver alternative for most purposes, because we've not started work on the required functionality yet. Our timeline is to do so beginning in Q1 of the next calendar year. With regard to DB replication in particular, we're investigating whether we can accelerate the schedule since it's so highly requested. It is definitely the goal of ''Tool Labs'' to support the kind of small-scale, non-deployed tools that toolserver authors love to create. Petr Bena, a Labs user, has created this page, which would definitely benefit from more participation as well: https://www.mediawiki.org/wiki/Wikimedia_Labs/Toolserver_features_wanted_in… It is true that we (WMF) have generally asked chapters to reduce investment in core infrastructure/services, and specifically asked WM-DE to work with us in transitioning from toolserver to Labs. There are a number of reasons for this: 1) WMF is a technology organization. Hosting the core infrastructure for Wikimedia projects is very much what we do. This includes data center operation, monitoring and backups, software deployments, software/service upgrades, code versioning infrastructure, bug tracking infrastructure, additional support systems and services (like this mailing list), etc. Toolserver is in fact hosted by the Wikimedia Foundation today, in our Amsterdam data-center. We provide space, power and racks for the toolserver cluster, at a cost of about $65,000/year to WMF according to our Director of TechOps. We also maintain the database replication on our end which enables tools to function. We can't provide the same level of service for the toolserver infrastructure as we do for core operations, and it makes no sense for a chapter to build out the required staffing and expertise to do so (set up/maintain all or some of the aforementioned functions). Even with slightly increased investment, toolserver would always suffer from being second or third tier infrastructure. 2) We're not comfortable hosting the toolserver infrastructure as-is. There are too many idiosyncratic aspects of its configuration; it has its own wiki, its own (closed source) version control system, its own (closed source) issue tracker. There are hacks like TUSC that we want to replace with better systems/services (e.g. OpenID/OAuth). So, what's next? Chapters are autonomous organizations, and it's WM-DE's call how much / whether it wants to continue to invest in infrastructure of any kind (and the decision of funding bodies like the FDC to accept or reject that proposition). However, for our part, we will not continue to support the current arrangement (DB replication, hosting in our data-center, etc.) indefinitely. The timeline we've discussed with Wikimedia Germany is roughly as follows: - Wind down new account creation on toolserver by Q2 of 2013 calendar year - Decommission toolserver by December 2013 WMF can't commit to providing technical support for tool transition (there are too many tools), so if there's any area where I think it would make sense to ramp up investments on WM-DE's part, it's in engineering capacity to support tool developers in porting tools to Labs. That said, there may be a need for emergency purchases/investments to keep TS in a usable state until December 2013 (and perhaps allow for some buffer room beyond that). That's not our call to make. Hope this clears up some questions around what's going on, and happy to answer further questions. All best, Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

Merlissimo <merl(a)toolserver.org> wrote: Apparently, it is not that smooth for WMF admins (http://permalink.gmane.org/gmane.org.wikimedia.toolserver/3166): | > one obvious solution would be to treat the Toolserver database servers | > as part of WMF's domain so they can use their SOPs (*1) to maintain | > them instead of manually coordinating with the Toolserver admins (or | > not). | I don't think that would be any easier. Our database systems are | completely different from theirs, so it would require all WMF admins to | learn (and remember) a new procedure before they could change anything | on their side. Since they usually don't think to notify us of major | changes in advance, I don't think they would be very happy with that. Tim

On 25/09/12 20:48, Erik Moeller wrote: Something we should all be grateful for. A replication which WMF is already doing for itself, adding an extra slave hosted by a trusted party doesn't make a techical difference. While WMF gains a backup server (at some point the toolserver was the only externally hosted backup). It has also inadvertedly dropped the binlogs needed by TS when doing server cleanup. So while very welcome, the WMF side of the db replication is not the most important piece of the toolserver (having a db copy still makes the TS better than any other alternative available at the time, though). The value is probably in how WMDE was able to open that database and many fruitful tools emerged. labs is also a second class citizen. Moreover, it is explicitely stated not to be for production-like level. What will happen if a really successful tool reaches to a point where it de-facto needs it ? (eg. a WLM tool, tools to Request to get an Account Created, or Request an Unblock appeal...) Since when is that a problem? To take an obvious example, labs also began by making its own wiki, instead of incorporating itself into an existing one. The vcs isn't closed source, it's just plain subversion. The *repository viewer* is closed source, although with a an Open Source license. I don't know if people really use it too much. I'm not a fan of jira, but this is the silliest reason to drop the toolserver :) OpenID nor OAuth are currently available. If they were, TUSC wouldn't have been developed in the first time. When they appear, they will -slowly at first- take over TUSC, as it should. This ball is at WMF side. Not to mention, if these «idiosyncratic aspects» were really a problem, they could easily be changed. This sounds as forcing them to go this route. So the tool authors are "on their own", while forced to move out. What is the WMF *offering*? But you are refraining WMDE from investing to it now. Would for instance WMF be willing to lend a few servers to the toolserver until December 2013?

On 25/09/12 23:38, Federico Leva (Nemo) wrote: Both approaches have its merits. The WMF could have copied the toolserver recipe much more easily than gooing the route of labs. They were brave pursuing the VM based system, and it allows solving different problems. But both of them have their use. It's not a case where one should "win" the other, or the toolserver is "bad" because it is currently hosted by WMDE instead of WMF. Organizations trying to defeat one another will end up with one big loser: the community.

+1. If people *really* love the Toolserver way of doing things, they are more than welcome to re-create the environment as a Labs project. - Ryan

The database servers (replicated and user-databases) are going to be provided by the Labs infrastructure. That's the majority of Toolserver's servers. We have shared storage provided already. The capacity of Labs is far greater than Toolserver's. We have a cluster in the pmtpa and eqiad datacenters. Each cluster has 7 compute nodes for virtual machines, 2 database servers for replicas, 1 database server for user databases and 4 storage nodes for shared storage access. In total that's: * 14 compute nodes with a total of 2.5TB of RAM, 224 CPU cores, and 14TB of storage for virtual machine images * 4 database nodes for replicas with a total of 740GB of RAM and 64 CPU cores * 2 database nodes for user-databases, with a total of 370GB of RAM and 32 CPU cores * 8 storage nodes with a total of 128TB of storage In a project you can create as many instances as you'd like. The default quota limit per-project is 10 instances, but we can increase that number as much as needed. It's definitely possible to re-create Toolserver inside of Labs. I'm not suggesting that's what should be done for a migration, but it's an option. - Ryan

I think the vast majority of the people on the Toolserver don't need a lot of that extra "beefy" stuff. To run our tools, all we need is: * A database providing replication from the Foundation's databases. * A web server capable of running PHP, CGI, and other web scripts. * A linux or solaris machine that we can run bots on. * A platform on which all of this runs reliably. If any of us needed a Mediawiki installation, we'd have set up our own site for it years ago. Sure, I can see you wanting to make options like that available to developers, particularly those building extensions to MW, but what the Toolserver community - and really, all of the Wikimedia projects - need right now is something that meets those four bullet points. With the Toolserver down as often as it's up lately, and Labs not ready to provide any of that support until December 2013 (not to mention no assistance moving there), either a) the Toolserver needs funding so it can get back to working order, even if it means no further accounts or projects can be created on it, or b) Labs needs to get that stuff running much sooner. And frankly, I don't see why WMDE should be footing the bill for all the transitioning work when the Foundation isn't providing them any support to do so while setting up a competing system at the same time. ---- User:Hersfold hersfoldwiki(a)gmail.com On 9/25/2012 9:30 PM, Erik Moeller wrote:

On Tue, Sep 25, 2012 at 9:41 PM, Hersfold <hersfoldwiki(a)gmail.com> wrote: Excluding the database replication, all of these things are possible now. Database replication should be done in the next 2-3 months. It may be done sooner, but that's our current goal date. Brave community members can start building any other missing infrastructure. I've never built Labs to be a competing system. Labs should have similar functionality, but I find the toolserver portions of Labs to be a small part of its function. My biggest goal with integrating Toolserver features is to bring our Toolserver community more closely together with our developer community. They are fairly disjoint right now. WMDE offered to help with the transition work. The decision to move the community from Toolserver to Labs was made jointly between WMDE and WMF. It's not possible for WMF to handle building the Labs infrastructure and to transition the tools. We don't have enough Toolserver experience to do so properly anyway. Toolserver was mostly built by the community from the start. Even WMDE working the transition won't be enough. We need people like you to help transition. My team will help you as much as needed, and will gladly answer any questions you have. - Ryan

Wrong. Do you know MMP https://wiki.toolserver.org/view/MMP ? Toolserver already allows collaboration as you describe it. What you seem to be saying is that you want to *force* people to collaborate: are you sure this works? I'd rather suggest to investigate what works or not in the MMP (very simple) model and why people don't use it more in Toolserver, so that you can build on that experience and understand better the users' needs. Nemo

So you think that even new users (I believe many users arrived after MMP were introduced) are not using MMP due to bad old habits of the other users? I've had a hard time trying to convince some users to use a MMP; I think it's not trivial to understand what models actually work and are liked. I'm not a technical person but yes, I can probably understand that MMP is not very elegant. But, do you think users are not using MMP because they're not elegant? Nemo

Just replying to the original mail here. While the discussion what the labs project should look like is usefull, the immediate problem at hand is a bit forgotten, that toolserver might end early next year if there is less support. As others stated, as some of the needed functionality is not present in labs, having an environment usefull for the current toolserver community will likely take till end of the year. To then have the community migrate there, setup the things there, perhaps migrate data, and perhaps change urls all over the place, will take some time already, so that is likely middle till end of 2014. So we will need toolserver up and running for a while, and that will take investments. And that is the problem that will need immediate attention at the moment. Regards, Andre On Tue, Sep 25, 2012 at 12:51:50AM +0200, DaB. wrote:

Hi Andre, 2012/9/26 Andre Koopal <andre(a)molens.org>rg>: Toolserver will not end early next year. Period. Wikimedia Deutschland will make all necessary investments to keep the Toolserver up and running, and we will use the next year to discuss together with the Toolserver users and community a way to transition into WikiLabs. And I can only urge you all to take active part in the design and development of Wikilabs, so that it will provide for your needs. With regards to new hardware: Our committment here dos not mean that we will simply replace broken parts only; it also means that we will buy new stuff, so that the Toolserver can handle increased demand. We do have the ressources to do so, and the money is available within our annual budget. Please support DaB and Sebastian Sooth (who is handling Toolserver matters from Wikimedia Deutschland side) in putting together a list of necessary items to be purchased. What we will not do, however, is ignoring the fact that the Toolserver will be replaced by Wikilabs sometime in the future. So every investment we make will have to take this into account. Pavel

Hello, At Wednesday 26 September 2012 12:18:12 DaB. wrote: while I'm at joing: We also need a live copy of s4 (commons) on every wmf-db- cluster (for the image- and commons-people). Sincerely, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885

2012/9/26 Platonides <platonides(a)gmail.com>om>: It is not hard to start coding in your own workspace and move code from your newly created directory to a MMP. A MMP doesn't necessarely have to be maintained by multiple users. While it's convenient having someone in the project that can jump in in case your tool breaks badly and you are not around to fix it, it is not by any means necessary.

On Wed, Sep 26, 2012 at 10:37:11AM +0200, Pavel Richter wrote: Hi Pavel, With all due respect, and I do believe your intentions are good, but if you can't convince DaB about this and he does quit, toolserver will break and stop. Although Nosy is still there, afaik she is still not up to date about all the details around toolserver, so there will be a lot of knowledge lost effectively ending toolserver. Regards, Andre

On 26/09/12 12:40, Sumurai8 (DD) wrote: Right. I was explaining why there aren't so many MMPs out there.

They can be created in LDAP by making a labsconsole account. Additionally, unless the account needs to directly log in via ssh, there's no request process needed for the user to be used. Of course right now there isn't open registration in labsconsole. We're waiting on some Jenkins changes for that to happen. Should be any day now. Alternatively, the user could be created as a system account via puppet. - Ryan

On 26/09/12 20:55, Ryan Lane wrote: I thought the jenkins changes were ready, after Antoine wrote:

Nope. More issues were run into. We're now deploying OpenStack's zuul framework to get this going. It needs backported packages, etc.. Either way, we're getting a little off topic :) - Ryan

On Sep 29, 2012, at 9:41 PM, Samuel Klein <meta.sj(a)gmail.com> wrote: [citation needed] I think you misunderstood. -- Krinkle

Samuel, you're obviously not on the same page as the executive staff at WMF (see Erik's email). Following this thread, it's becoming more and more clear that before any serious discussion of the toolserver's future, the WMF needs to get it's priorities straight. Do you or do you not want to replace the toolserver? If you do, it seems like a sensible thing to do to keep as much of the configuration as possible. I've been looking at the labs in the last few days and it seems to me that it's architecture is overkill for doing simple things. I think more and more people will choose to go with their own hosting for robots instead of using the labs. This probably means money spent by the WMF in traffic, since those users will be using the api instead of accessing the database directly.

On Sun, Sep 30, 2012 at 4:08 AM, Andrei Cipu <wiki(a)strainu.ro> wrote: That's possible, and as Krinkle says it may well be my misunderstanding. That's why I asked. Erik's email was thorough and detailed, I'm just not clear about those few points. I am not writing as a WMF trustee, but in my personal capacity as a community member interested in the toolserver. This is a fairly operational discussion, and not something discussed by the board; I only know about it because I read this list. Regards, SJ Following this thread, it's becoming more and more clear that before any

On Sun, Sep 30, 2012 at 10:24 AM, Krinkle <krinklemail(a)gmail.com> wrote: That's correct. Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

On 30 Sep 2012, at 18:24, Krinkle <krinklemail(a)gmail.com> wrote: This is where you have misunderstood Krinkle, if you re-read SJ's email: On 29 Sep 2012, at 20:41, Samuel Klein <meta.sj(a)gmail.com> wrote: The 'it' clearly refers to the toolserver and so as far as I can tell Sam's question has not been answered: Why is the WMF considering to not provide db replication for the Toolserver? As has previously been mentioned, db replication is what makes the Toolserver so useful and thus if the WMF stops supporting db replication for it, it is effectively replacing the Toolserver with Labs.

On 01/10/12 01:28, Carl (CBM) wrote: Note that it's false that MediaWiki doesn't perform joins. There are many of them. The statement about that was (probably) talking about cross-db joins. But the reason for using different dbs on toolserver is based on user permissions, not in that those tables conceptually should go with the data in the wiki db. Even for MediaWiki extensions, not having user tables can be hard. If your extension needs a new table, with you can create it at another db and link them with $wgSharedTables (a configuration change, without touching the extension code). Without that, testing one extension with an additional table would need it to be created in the master wiki db! (plus have a user with permissions to write it, and all related maintenance)

On Mon, Oct 1, 2012 at 10:02 AM, Carl (CBM) <cbm.wikipedia(a)gmail.com> wrote: Yes, and we should definitely find ways to support projects like the WP 1.0 assessment DB in Labs. The feature set of the Labs DB replication isn't final, and it's likely going to be iterative. We do recognize the tremendous effort that's gone into some of these tools, and the value they represent. We'll host an IRC meeting soon that we'll broadcast to toolserver-l@ as well to allow for more discussion of requirements for tool labs (the phase of the labs project dedicated to supporting tools development) and to answer questions about how folks can use Labs today. In the meantime, don't hesitate to join #wikimedia-labs on irc.freenode.net as well. Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06.10.2012 11:06, Dr. Trigon wrote: https://jira.toolserver.org/browse/TS-1534 should help as well... Thanks to everybody involved in helping me to come to that clue!!! Greetings