[Labs-l] Maintenance complete - please check your instances!

[Ryan Lane]

On Tue, Mar 6, 2012 at 12:14 PM, Ryan Lane <rlane32 at gmail.com> wrote:
> Maintenance is complete and was succesful. The api-service is
> currently overloaded, since I booted far too many instances at the
> same time, so labsconsole may occasionally act strange for the next
> few hours. Now would be a good time to check to ensure your instances
> are working properly, and to restart bots and such.
>
> If your instance is not alive for some reason, try rebooting it via
> labsconsole. If that doesn't work, let me know and I'll investigate
> for you. Please make sure to let me know if you notice any problems!
>

There were a number of instances that failed to reboot because they
were live-migrated at some point in the past and the live-migration
didn't fully update everything. Obviously live-migration is broken in
the version of nova we are using. Either way, I've fixed all of these
instances, and all of them should be able to reboot now, which is
somewhat of a win.

I also took the opportunity to apply all security updates during this
maintenance period. In the future for security updates, since
live-migration is broken, we'll need to bring instances down like
we've done today. We'll be informing the list of planned downtime at
minimum 48 hours in advance.

- Ryan