[Labs-l] Partial outage in progress -- update

[Andrew Bogott]

On 2/17/15 10:49 AM, Andrew Bogott wrote:
> No data was lost.  I'm currently migrating VMs from virt1005 onto a 
> new, more trustworthy server. All affected instances will start back 
> up one by one over the next hour or so.
There turn out to be some MASSIVE instances on that box, so the copy is 
taking longer than I expected.  Still chugging along though.

-A



>
> -Andrew
>
>
> On 2/17/15 9:31 AM, Andrew Bogott wrote:
>> One of the labs virtualization hosts, virt1005, is suffering a disk 
>> failure.  I'm restarting right now -- that may allow us to gradually 
>> recover.  If we're less lucky, then rebuilding from the outage will 
>> be a prolonged process.
>>
>> A list of affected instances can be found here:
>>
>> https://phabricator.wikimedia.org/P305
>>
>> Note that this box was hosting the Tools web proxy, so the web 
>> interface for most tools is currently down.  That should be easy to 
>> rebuild if necessary, and will be a high priority.
>>
>> Updates as events warrant!
>>
>> -Andrew
>