[Labs-l] Partial outage in progress (resolved)

[Andrew Bogott]

All concerned instances have been moved to new hardware (virt1012) and 
I've started every instance that was running before.

Sorry for the interruption, and please let me know if you encounter any 
lingering issues.

-Andrew



On 2/17/15 9:31 AM, Andrew Bogott wrote:
> One of the labs virtualization hosts, virt1005, is suffering a disk 
> failure.  I'm restarting right now -- that may allow us to gradually 
> recover.  If we're less lucky, then rebuilding from the outage will be 
> a prolonged process.
>
> A list of affected instances can be found here:
>
> https://phabricator.wikimedia.org/P305
>
> Note that this box was hosting the Tools web proxy, so the web 
> interface for most tools is currently down.  That should be easy to 
> rebuild if necessary, and will be a high priority.
>
> Updates as events warrant!
>
> -Andrew