CI tests stopped working again a couple of hours ago; I got a ping on IRC from paladox when I first woke up and checked my phone.
The issue this morning was different, but most likely fallout from last night's issue. In short, nova's records of quota usage were all goofy and nova was preventing VM creation due to perceived quota violation even though we weren't actually over quota. I did some manual database intervention to force nova to re-calculate usages and now all is well.
Details, including the commands I used, in https://phabricator.wikimedia.org/T188820.
I also added example snippets to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#i....
-Andrew
Thanks Andrew. I thought Nova was supposed to recalculate that on the fly but not shocking it was gummed up either.
On Mar 3, 2018 7:48 AM, "Andrew Bogott" abogott@wikimedia.org wrote:
CI tests stopped working again a couple of hours ago; I got a ping on IRC from paladox when I first woke up and checked my phone.
The issue this morning was different, but most likely fallout from last night's issue. In short, nova's records of quota usage were all goofy and nova was preventing VM creation due to perceived quota violation even though we weren't actually over quota. I did some manual database intervention to force nova to re-calculate usages and now all is well.
Details, including the commands I used, in https://phabricator.wikimedia. org/T188820.
I also added example snippets to https://wikitech.wikimedia.org /wiki/Portal:Cloud_VPS/Admin/Troubleshooting#incorrect_quota_violations.
-Andrew
Cloud-admin mailing list Cloud-admin@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/cloud-admin
cloud-admin@lists.wikimedia.org