[Labs-l] Labs instance hesitations over the coming week

[Andrew Bogott]

We've encountered an unexpected problem with this -- there's a kernel 
bug running on the new hardware which is causing instances to behave poorly.

So, I need to upgrade the kernel and reboot.  I'm moving all tools 
instances out of the way first so they aren't hit by the reboot; a few 
other projects (notably deployment-prep and staging) will suffer rolling 
reboots.

Fortunately this issue appeared early enough that most instances are 
still running on the old hardware, so most of you will be unaffected.

-Andrew


On 4/22/15 11:09 AM, Andrew Bogott wrote:
> Greetings!
>
> I'll be gradually moving most labs instances to new hardware over the 
> coming 7-10 days.  For virtually all instances this move will be 
> invisible to users -- the worst case scenario is that an instance will 
> freeze for a minute or two during the final post-copy sync. I've 
> already moved several projects without incident.
>
> Nevertheless, services which have extremely touchy timeouts may error 
> out or throw warnings.  For example, a few things in deployment-prep 
> sent 'service down' alerts which lasted a few seconds before being 
> resolved.  So if you see something like that, it's probably a result 
> of the move.
>
> I will be moving the Tools instances first, starting tomorrow morning 
> (approximately 14:00 UTC).  The tools migration will take around 18 
> hours.  On Friday morning I'll start a scripted move of all other 
> instances; the complete move will take around 7 days.
>
> If this process concerns you and you'd like your instances moved at a 
> pre-determined time, feel free to contact me off-list and we can make 
> an appointment for your project.
>
> -Andrew
>
>