I recently noticed that some of our standard kvm/nova monitoring never
got copied over from the labvirt puppet code to the cloudvirt puppet
code. Tomorrow I will merge
https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/478113/ to fix that.
Once that patch is merged, icinga will be a bit touchier on the
cloudvirts. In particular, it will alert for any cloudvirt that has 0
VMs running on it. (This turns out to be a useful thing to watch for
because we've had cases where every single kvm process died at once.)
So, all 'idle' cloudvirts should nonetheless have a canary instance.
For example, on the new analytics cloudvirts I created canaries like this:
$ OS_PROJECT_ID=testlabs openstack server create --image
7c6371d1-8411-48c7-bf73-2ef6d6ff2a15 --flavor m1.small --nic
net-id=7425e328-560c-4f00-8e99-706f3fb90bb4 --availability-zone
host:cloudvirtan1004 canary-an1004-01
Once a virt host is in full service we can leave the canaries there or
delete them -- there hasn't been any real consistent policy there.
In related news, I'm attempting to silence cloudvirt1019 and 1020
altogether with
https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/478115/ because
we reboot them twice a day and a reboot always kills any running VMs.
-Andrew