[Labs-l] What's up with brewster?

[Ryan Lane]

brewster's squid instance was dead because / was at 100%. Like most of our
infrastructure, we're missing some critical and necessary checks on
brewster. This service being down affects labs more dramatically than it
does the rest of the infrastructure.

I've brought the service back up. I also added an rt ticket for monitoring
of this service. Past monitoring, we should likely ensure it has a backup,
since it's critical for labs.


On Sun, Mar 3, 2013 at 12:21 AM, Matthew Walker <mwalker at wikimedia.org>wrote:

> It seems like brewster might be in some trouble? Either that or my
> security rules are a bit wonky (though they don't seem like they are.)
>
> In any case -- I cannot create a new labs instance: I get a whole bunch of
> errors that look like:
>
> Err http://apt.wikimedia.org precise-wikimedia Release.gpg
>   Unable to connect to brewster.wikimedia.org:8080:
>
>
> And... when I attempt to wget http://brewster.wikimedia.org:8080 from an
> already working instance I get a 'cannot connect to remote server' error.
> Ganglia shows that it's still up, but that it's had unusually little
> network activity (ie: close to none) since ~07:20 UTC.
>
> So! Is this just me, or did brewster actually wander off the grid?
>
> ~Matt Walker
> Wikimedia Foundation
> Fundraising Technology Team
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20130303/01e7a7cc/attachment.html>