Thanks Andrew. I thought Nova was supposed to recalculate that on the fly
but not shocking it was gummed up either.
On Mar 3, 2018 7:48 AM, "Andrew Bogott" <abogott(a)wikimedia.org> wrote:
CI tests stopped working again a couple of hours ago;
I got a ping on IRC
from paladox when I first woke up and checked my phone.
The issue this morning was different, but most likely fallout from last
night's issue. In short, nova's records of quota usage were all goofy and
nova was preventing VM creation due to perceived quota violation even
though we weren't actually over quota. I did some manual database
intervention to force nova to re-calculate usages and now all is well.
Details, including the commands I used, in
https://phabricator.wikimedia.
org/T188820.
I also added example snippets to
https://wikitech.wikimedia.org
/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#incorrect_quota_violations.
-Andrew
_______________________________________________
Cloud-admin mailing list
Cloud-admin(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/cloud-admin