[Labs-l] Yet another partial labs outage (Resolved)

[Andrew Bogott]

This turns out to not have been a heating issue, or at least not 
entirely -- it was some kind of kernel lockup.  Coren and others 
rebooted the system and restarted all instances, and things seem to be 
working fine now.  We don't have much explanation for what caused the 
problem, though, so we'll be on the lookout.

-A


On 5/15/15 11:31 PM, Andrew Bogott wrote:
> The hardware curse continues!
>
> One of the labs virt hosts (labvirt1003) is running very hot tonight, 
> presumably due to a broken fan.  It is intermittently scaling the CPU 
> speed way back to avoid melting; when that happens there are bound to 
> be lots of side-effects like unresponsive instances, clock drift, and 
> the like (not least of which is that right now I can't ssh into the 
> damn thing, or get performance metrics.)
>
> Naturally this started happening late on a Friday, so it may be a 
> while before I can get someone in the datacenter.  I'm leaving the 
> host up in the meantime, based on the notion that half a server is 
> better than none, but poor performance is likely to be the norm in the 
> meantime.
>
> I did shut off one instance:  wikidata-wdq-mm.  I don't have a 
> personal grudge, but it was gobbling CPU cycles and the system really 
> needs a rest.  If loss of that instance is a disaster for anyone, 
> contact me and I'll see if I can revive it and shut off ten or so 
> other instances to make room.
>
> Updates as events warrant!
>
> -Andrew