[Labs-l] New instance creation outage ongoing

[Ryan Lane]

This is now fixed. The query that was causing the problem was brought
down from 11-15s to .1s. The patch is live, and is being upstreamed to
newer versions of OpenStack nova.

- Ryan

On Wed, Jul 18, 2012 at 6:36 PM, Ryan Lane <rlane32 at gmail.com> wrote:
> OpenStack Nova does some inefficient queries for searching for
> instances, especially when pulling metadata information. When
> instances are created, we inject what's called "userdata". That
> userdata is used by a service in ubuntu called cloud-init. cloud-init
> pulls this userdata, along with the instance's metadata and does the
> initial bootstrapping of the system.
>
> For us, cloud-init, using the userdata, installs puppet, points it at
> our puppet server and does a full puppet run. This process is
> currently failing, as cloud-init thinks the metadata server is timing
> out. Apparently other deployments of openstack are having performance
> issues with metadata that are due to the same inefficient query.
>
> There's two things we'll be doing about this:
>
> 1. We're working on a fix with the other openstack devs
> 2. We'll purge deleted instances from the database. Nova keeps them
> for auditing purposes, but they are unneeded. This will bring the
> query speed down to a level that is below the cloud-init threshold.
> We're also waiting for another organization to push their solution for
> this upstream, rather than mucking around the database ourselves.
>
> Sorry for the inconvenience,
>
> - Ryan