[Wikimedia-l] Wikimedia labs-tools

[Ryan Lane]

On Wed, Sep 11, 2013 at 7:50 AM, John <phoenixoverride at gmail.com> wrote:

> tools.wmflabs.org is supposed to be the replacement for the toolserver
> which the wmf is basically forcefully shutting down. I started the
> migration several months ago but got fed up with the difficulties and
> stopped. In the last month I have moved most of my tools to labs, and I
> have discovered that there are some serious issues that need addressed.
>
> The toolserver was a fairly stable environment. I checked my primary host
> I connect to and it has been up for 4 months with continuous operations.
>
>
I'm not going to do an analysis on this to disprove you, but there were
periods of time where toolserver was down for over a week (more than once,
at that). You're talking about connection to bastion hosts, which doesn't
reflect the system as a whole.


> tools however is being treated like the red-headed step child. According
> to the people in charge of labs they dont care about ensuring stability and
> that if stuff breaks Oh well well get to it when we can. They say that
> tools is not a production service so we really don't give a <>, if it
> breaks it breaks, we will fix it when we can but since its not production
> its not a priority.
>
>
The only major instability we've had in tools is with the NFS server. We've
been working on it for months. I honestly think Labs is cursed when it
comes to storage, because our other major instability since project
inception was glusterfs.

As mentioned in another email, we do care, but we also need to be clear
about our level of support. Our advertised level of support is
semi-production. If something breaks in the middle of the night we have
people that will wake up and fix it.


> One good example of this is that a tool cannot connect to
> tools.wmflabs.org due to a host configuration issue. This is a known bug,
> we have a way of fixing it, but its still not implemented
>
>
This is due to the way the floating IP addresses work in OpenStack's nova
service. There's workarounds for this issue. For instance, you can connect
to the private DNS or IP address of the webserver, rather than using the
public hostname.

This is definitely something we'd like to fix, but haven't had a chance to
do so yet.

-Ryan