[Labs-announce] Finished: Labs-wide security updates tomorrow, 2016-06-08

[Andrew Bogott]

This is (finally) done.  A handful of instances didn't handle the 
suspends properly and had to be rebooted, but everything should be up 
and running now.  Please let me know if you have instances with new issues.

-Andrew


On 6/7/16 5:26 PM, Andrew Bogott wrote:
> This didn't go very well, and then I got pulled into a related 
> emergency[1] (one that was related but largely invisible to users.)
>
> So, I may have another go at the suspending/resuming tomorrow 
> morning.  As before, if we wind up needing to do something more 
> intrusive than suspend/resume I will provide additional notice.
>
> Sorry about the endless maintenance window :(
>
> -Andrew
>
> [1] https://phabricator.wikimedia.org/T137241
>
> On 6/6/16 9:02 AM, Andrew Bogott wrote:
>>     Moritz and I will be applying some security patches to the 
>> virtualization hosts tomorrow, beginning at 14:00 UTC.  We're going 
>> to try to apply this change via suspend/resume cycles rather than via 
>> reboots -- that means that each VM should experience a minute or two 
>> (or three) of hesitation but should not register any reboots.
>>
>>     If this goes badly and it turns out that actual reboots are 
>> needed I will reschedule the maintenance for a later date with more 
>> notice.  For now, though, no action is required from any of you.
>>
>> -Andrew
>>
>