On Wed, Sep 11, 2013 at 7:50 AM, John <phoenixoverride(a)gmail.com> wrote:
tools.wmflabs.org is supposed to be the replacement
for the toolserver
which the wmf is basically forcefully shutting down. I started the
migration several months ago but got fed up with the difficulties and
stopped. In the last month I have moved most of my tools to labs, and I
have discovered that there are some serious issues that need addressed.
The toolserver was a fairly stable environment. I checked my primary host
I connect to and it has been up for 4 months with continuous operations.
I'm not going to do an analysis on this to disprove you, but there were
periods of time where toolserver was down for over a week (more than once,
at that). You're talking about connection to bastion hosts, which doesn't
reflect the system as a whole.
tools however is being treated like the red-headed
step child. According
to the people in charge of labs they dont care about ensuring stability and
that if stuff breaks Oh well well get to it when we can. They say that
tools is not a production service so we really don't give a <>, if it
breaks it breaks, we will fix it when we can but since its not production
its not a priority.
The only major instability we've had in tools is with the NFS server.
We've
been working on it for months. I honestly think Labs is cursed when it
comes to storage, because our other major instability since project
inception was glusterfs.
As mentioned in another email, we do care, but we also need to be clear
about our level of support. Our advertised level of support is
semi-production. If something breaks in the middle of the night we have
people that will wake up and fix it.
One good example of this is that a tool cannot connect
to
tools.wmflabs.org due to a host configuration issue. This is a known bug,
we have a way of fixing it, but its still not implemented
This is due to the way the floating IP addresses work in OpenStack's nova
service. There's workarounds for this issue. For instance, you can connect
to the private DNS or IP address of the webserver, rather than using the
public hostname.
This is definitely something we'd like to fix, but haven't had a chance to
do so yet.
-Ryan