tools.wmflabs.org is supposed to be the replacement for the toolserver which the wmf is basically forcefully shutting down. I started the migration several months ago but got fed up with the difficulties and stopped. In the last month I have moved most of my tools to labs, and I have discovered that there are some serious issues that need addressed.
The toolserver was a fairly stable environment. I checked my primary host I connect to and it has been up for 4 months with continuous operations.
tools however is being treated like the red-headed step child. According to the people in charge of labs they dont care about ensuring stability and that if stuff breaks Oh well well get to it when we can. They say that tools is not a production service so we really don't give a <>, if it breaks it breaks, we will fix it when we can but since its not production its not a priority.
One good example of this is that a tool cannot connect to tools.wmflabs.orgdue to a host configuration issue. This is a known bug, we have a way of fixing it, but its still not implemented
Given that tools is replacing the toolserver I would expect at worst labs is just as good, however what I am seeing and hearing is that the wmf is throwing away one of their best assets, and driving away a lot of developers due to the management of tools.
I do want to give Coren credit as he is doing what he can to support the migration.
My question is why has the wmf decided to degrade the environment where tool developers design and host tools (quite a few of them are long term stable projects)? and what can we do to remedy this?
John
Hi,
can you please explain to me from where is this information:
"According to the people in charge of labs they dont care about ensuring stability and that if stuff breaks Oh well well get to it when we can. They say that tools is not a production service so we really don't give a <>, if it breaks it breaks"
If I can speak for myself as a volunteer sysadmin of tool labs, I do care if tool labs are stable or if they break, it's just that for many of the outages I just can't do anything but sit and wait for someone with access to servers which cause troubles (which are typically outside of tool labs, like nfs storage or mysql replicas)
I can't speak for others though, but I doubt that anyone who is really "in charge" told you they don't care about stability. Or at least I hope so :-)
On Wed, Sep 11, 2013 at 4:50 PM, John phoenixoverride@gmail.com wrote:
tools.wmflabs.org is supposed to be the replacement for the toolserver which the wmf is basically forcefully shutting down. I started the migration several months ago but got fed up with the difficulties and stopped. In the last month I have moved most of my tools to labs, and I have discovered that there are some serious issues that need addressed.
The toolserver was a fairly stable environment. I checked my primary host I connect to and it has been up for 4 months with continuous operations.
tools however is being treated like the red-headed step child. According to the people in charge of labs they dont care about ensuring stability and that if stuff breaks Oh well well get to it when we can. They say that tools is not a production service so we really don't give a <>, if it breaks it breaks, we will fix it when we can but since its not production its not a priority.
One good example of this is that a tool cannot connect to tools.wmflabs.org due to a host configuration issue. This is a known bug, we have a way of fixing it, but its still not implemented
Given that tools is replacing the toolserver I would expect at worst labs is just as good, however what I am seeing and hearing is that the wmf is throwing away one of their best assets, and driving away a lot of developers due to the management of tools.
I do want to give Coren credit as he is doing what he can to support the migration.
My question is why has the wmf decided to degrade the environment where tool developers design and host tools (quite a few of them are long term stable projects)? and what can we do to remedy this?
John
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
There was a recent mail saying that Labs is not considered "production" stability. Mainly a disagreement about how many 9s in the 99.99999% that represents.
On Wed, Sep 11, 2013 at 4:09 PM, Petr Bena benapetr@gmail.com wrote:
Hi,
can you please explain to me from where is this information:
"According to the people in charge of labs they dont care about ensuring stability and that if stuff breaks Oh well well get to it when we can. They say that tools is not a production service so we really don't give a <>, if it breaks it breaks"
If I can speak for myself as a volunteer sysadmin of tool labs, I do care if tool labs are stable or if they break, it's just that for many of the outages I just can't do anything but sit and wait for someone with access to servers which cause troubles (which are typically outside of tool labs, like nfs storage or mysql replicas)
I can't speak for others though, but I doubt that anyone who is really "in charge" told you they don't care about stability. Or at least I hope so :-)
On Wed, Sep 11, 2013 at 4:50 PM, John phoenixoverride@gmail.com wrote:
tools.wmflabs.org is supposed to be the replacement for the toolserver
which
the wmf is basically forcefully shutting down. I started the migration several months ago but got fed up with the difficulties and stopped. In
the
last month I have moved most of my tools to labs, and I have discovered
that
there are some serious issues that need addressed.
The toolserver was a fairly stable environment. I checked my primary
host I
connect to and it has been up for 4 months with continuous operations.
tools however is being treated like the red-headed step child. According
to
the people in charge of labs they dont care about ensuring stability and that if stuff breaks Oh well well get to it when we can. They say that
tools
is not a production service so we really don't give a <>, if it breaks it breaks, we will fix it when we can but since its not production its not a priority.
One good example of this is that a tool cannot connect to
tools.wmflabs.org
due to a host configuration issue. This is a known bug, we have a way of fixing it, but its still not implemented
Given that tools is replacing the toolserver I would expect at worst
labs is
just as good, however what I am seeing and hearing is that the wmf is throwing away one of their best assets, and driving away a lot of
developers
due to the management of tools.
I do want to give Coren credit as he is doing what he can to support the migration.
My question is why has the wmf decided to degrade the environment where
tool
developers design and host tools (quite a few of them are long term
stable
projects)? and what can we do to remedy this?
John
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
On Wed, Sep 11, 2013 at 8:45 AM, Magnus Manske magnusmanske@googlemail.comwrote:
There was a recent mail saying that Labs is not considered "production" stability. Mainly a disagreement about how many 9s in the 99.99999% that represents.
Indeed. I don't want to get into the debate about this again, but tools is considered "semi-production" which is a smaller set of nines. We're reasonably staffed and have a well designed enough infrastructure to properly support that, but it's not the case for "production"-level support. The specific discussion about levels of support was for a service that should be supported by WMF in production since it has uptime requirements that we aren't scoped to handle.
We handle the underlying infrastructure with production-level support, but we don't have the same level of support for projects inside of the infrastructure.
I think so far we've done a relatively good job of keeping stability and the level of stability has been increasing, not decreasing. If we did an analysis of tools project outages vs toolserver, I'm positive our level of unscheduled downtime would be far lower than toolserver's.
- Ryan
On Wed, 2013-09-11 at 10:50 -0400, John wrote:
One good example of this is that a tool cannot connect to tools.wmflabs.orgdue to a host configuration issue. This is a known bug
Could you point out the corresponding ticket among https://bugzilla.wikimedia.org/buglist.cgi?resolution=---&component=tool... please?
Thanks in advance! andre
Despite it's known bug (even Ryan was telling me it has something to do with nova) I could only find 1 bug so far... https://bugzilla.wikimedia.org/show_bug.cgi?id=45868 maybe some should be created, but people from ops know about this
On Wed, Sep 11, 2013 at 6:49 PM, Andre Klapper aklapper@wikimedia.org wrote:
On Wed, 2013-09-11 at 10:50 -0400, John wrote:
One good example of this is that a tool cannot connect to tools.wmflabs.orgdue to a host configuration issue. This is a known bug
Could you point out the corresponding ticket among https://bugzilla.wikimedia.org/buglist.cgi?resolution=---&component=tool... please?
Thanks in advance! andre -- Andre Klapper | Wikimedia Bugwrangler http://blogs.gnome.org/aklapper/
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Sep 11, 2013 at 7:50 AM, John phoenixoverride@gmail.com wrote:
tools.wmflabs.org is supposed to be the replacement for the toolserver which the wmf is basically forcefully shutting down. I started the migration several months ago but got fed up with the difficulties and stopped. In the last month I have moved most of my tools to labs, and I have discovered that there are some serious issues that need addressed.
The toolserver was a fairly stable environment. I checked my primary host I connect to and it has been up for 4 months with continuous operations.
I'm not going to do an analysis on this to disprove you, but there were periods of time where toolserver was down for over a week (more than once, at that). You're talking about connection to bastion hosts, which doesn't reflect the system as a whole.
tools however is being treated like the red-headed step child. According to the people in charge of labs they dont care about ensuring stability and that if stuff breaks Oh well well get to it when we can. They say that tools is not a production service so we really don't give a <>, if it breaks it breaks, we will fix it when we can but since its not production its not a priority.
The only major instability we've had in tools is with the NFS server. We've been working on it for months. I honestly think Labs is cursed when it comes to storage, because our other major instability since project inception was glusterfs.
As mentioned in another email, we do care, but we also need to be clear about our level of support. Our advertised level of support is semi-production. If something breaks in the middle of the night we have people that will wake up and fix it.
One good example of this is that a tool cannot connect to tools.wmflabs.org due to a host configuration issue. This is a known bug, we have a way of fixing it, but its still not implemented
This is due to the way the floating IP addresses work in OpenStack's nova service. There's workarounds for this issue. For instance, you can connect to the private DNS or IP address of the webserver, rather than using the public hostname.
This is definitely something we'd like to fix, but haven't had a chance to do so yet.
-Ryan
I'm not sure this needed to be broadcast to three lists...
John wrote:
The toolserver was a fairly stable environment. I checked my primary host I connect to and it has been up for 4 months with continuous operations.
I can only assume "fairly" was a typo for "rarely." :-) I love the German Toolserver and would love to see many more Toolservers, but the myth that German Toolserver I was or is stable is quickly debunked by a visit to http://stable.toolserver.org. It suffered frequent outages and database corruption, high replication lag, and unstable and unsupported services.
My question is why has the wmf decided to degrade the environment where tool developers design and host tools (quite a few of them are long term stable projects)? and what can we do to remedy this?
I'll echo what Andre said. There seems to be one issue mentioned in your e-mail (host connectivity something or other), but if you're having many issues with Wikimedia Labs, please file bugs: https://bugs.wikimedia.org. In addition to providing you with something substantive to demonstrate your claim that Labs isn't working well, filed bugs in Bugzilla will also allow people running Labs the opportunity to perhaps fix these issues.
MZMcBride
wikitech-l@lists.wikimedia.org