[Labs-announce] Instance creation and security group handling issues in Liberty

[Chase Pettet]

Currently running instances within Labs are fine.

This week we upgraded to Openstack Liberty[1][2].  Thursday (8/4) we had
reports of issues involving new instances[3].  We have now determined there
is errant behavior with Liberty managing source groups.  We use this to
allow instances within the same project to communicate with each other.
Attempts to resolve this behavior for the Tool Labs project resulted in a
short issue today[4].  Requests via the web proxy were failing to connect.
Tools and bots within Tool Labs were still running.

Currently:
* Newly created instances are not being integrated into their security
groups appropriately
* We have disabled the self-serve options for instance creation temporarily
* Modifying security groups can result in existing instances experiencing
issues
* We have disabled the self-serve options for security group management
temporarily as well

We'll update the task[3] as we have more information.  An incident report
will be filed as well.   As always, we can be found at labs-l or on IRC in
#wikimedia-labs.

Thanks,

Chase Pettet (on behalf of the Labs team)


[1] https://www.openstack.org/software/liberty/
[2] https://lists.wikimedia.org/pipermail/labs-l/2016-July/004564.html
[3] https://phabricator.wikimedia.org/T142165
[4] https://lists.wikimedia.org/pipermail/labs-l/2016-August/004575.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-announce/attachments/20160805/0c1f2e7b/attachment.html>