CI tests stopped working again a couple of hours ago; I got a ping on
IRC from paladox when I first woke up and checked my phone.
The issue this morning was different, but most likely fallout from last
night's issue. In short, nova's records of quota usage were all goofy
and nova was preventing VM creation due to perceived quota violation
even though we weren't actually over quota. I did some manual database
intervention to force nova to re-calculate usages and now all is well.
Details, including the commands I used, in
https://phabricator.wikimedia.org/T188820.
I also added example snippets to
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#….
-Andrew