CI tests stopped working again a couple of hours ago; I got a ping on IRC from paladox when I first woke up and checked my phone.

The issue this morning was different, but most likely fallout from last night's issue.  In short, nova's records of quota usage were all goofy and nova was preventing VM creation due to perceived quota violation even though we weren't actually over quota.  I did some manual database intervention to force nova to re-calculate usages and now all is well.

Details, including the commands I used, in https://phabricator.wikimedia.org/T188820.

I also added example snippets to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#incorrect_quota_violations.

-Andrew

_______________________________________________
Cloud-admin mailing list
Cloud-admin@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/cloud-admin