[Labs-l] Labs instance hesitations over the coming week

[Andrew Bogott]

Update:

I upgraded the kernel on all of the new hardware, and I'm /pretty sure/ 
that that fixed the problem.  I'm going to wait a few days and let the 
dust settle -- if everything looks good over the weekend then I'll 
resume scripted migration on Monday.

Apologies for any inconvenience caused by flappy VMs.

-Andrew



On 4/23/15 4:17 PM, Andrew Bogott wrote:
> We've encountered an unexpected problem with this -- there's a kernel 
> bug running on the new hardware which is causing instances to behave 
> poorly.
>
> So, I need to upgrade the kernel and reboot.  I'm moving all tools 
> instances out of the way first so they aren't hit by the reboot; a few 
> other projects (notably deployment-prep and staging) will suffer 
> rolling reboots.
>
> Fortunately this issue appeared early enough that most instances are 
> still running on the old hardware, so most of you will be unaffected.
>
> -Andrew
>
>
> On 4/22/15 11:09 AM, Andrew Bogott wrote:
>> Greetings!
>>
>> I'll be gradually moving most labs instances to new hardware over the 
>> coming 7-10 days.  For virtually all instances this move will be 
>> invisible to users -- the worst case scenario is that an instance 
>> will freeze for a minute or two during the final post-copy sync. I've 
>> already moved several projects without incident.
>>
>> Nevertheless, services which have extremely touchy timeouts may error 
>> out or throw warnings.  For example, a few things in deployment-prep 
>> sent 'service down' alerts which lasted a few seconds before being 
>> resolved.  So if you see something like that, it's probably a result 
>> of the move.
>>
>> I will be moving the Tools instances first, starting tomorrow morning 
>> (approximately 14:00 UTC).  The tools migration will take around 18 
>> hours.  On Friday morning I'll start a scripted move of all other 
>> instances; the complete move will take around 7 days.
>>
>> If this process concerns you and you'd like your instances moved at a 
>> pre-determined time, feel free to contact me off-list and we can make 
>> an appointment for your project.
>>
>> -Andrew
>>
>>
>