<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">The issues discussed in this email are
now resolved.<br>
<br>
Instance creation on Horizon and Wikitech has been re-enabled.
After some discussion we've decided to enable security group
editing on Horizon but leave it disabled on Wikitech -- the
Horizon interface is generally nicer, more feature-rich, and more
reliable. Please go to Horizon for any future security group
needs.<br>
<br>
There were two bugs that triggered this incident. One of them[1]
prevented enforcement of firewall rules in certain cases, and the
other[2] enforced rules but updated them very haphazardly. Both
issues are now well understood, with patches in place and proper
long-term solutions underway.<br>
<br>
We have not yet written a full incident report, but when we do it
will most likely be here:
<a class="moz-txt-link-freetext" href="https://wikitech.wikimedia.org/wiki/Incident_documentation/20160805-LabsSecurityGroups">https://wikitech.wikimedia.org/wiki/Incident_documentation/20160805-LabsSecurityGroups</a><br>
<br>
Sorry for the inconvenience!<br>
<br>
-Andrew<br>
<br>
[1] <a class="moz-txt-link-freetext" href="https://phabricator.wikimedia.org/T142388">https://phabricator.wikimedia.org/T142388</a><br>
<br>
[2] <a class="moz-txt-link-freetext" href="https://phabricator.wikimedia.org/T142165">https://phabricator.wikimedia.org/T142165</a><br>
<br>
<br>
<br>
On 8/5/16 3:21 PM, Chase Pettet wrote:<br>
</div>
<blockquote
cite="mid:CANBEROwxjdw=z1Hy8UJ-stBH6A3vCCTL7sL_RaW5GfgRSzR6dg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>Currently running instances within Labs are fine.<br>
<br>
</div>
This week we upgraded to Openstack Liberty[1][2]. Thursday
(8/4) we had reports of issues involving new instances[3]. We
have now determined there is errant behavior with Liberty
managing source groups. We use this to allow instances within
the same project to communicate with each other. Attempts to
resolve this behavior for the Tool Labs project resulted in a
short issue today[4]. Requests via the web proxy were failing
to connect. Tools and bots within Tool Labs were still
running.<br>
<br>
</div>
<div>Currently:<br>
</div>
<div>* Newly created instances are not being integrated into
their security groups appropriately<br>
* We have disabled the self-serve options for instance
creation temporarily<br>
* Modifying security groups can result in existing instances
experiencing issues<br>
* We have disabled the self-serve options for security group
management temporarily as well<br>
</div>
<br>
We'll update the task[3] as we have more information. An
incident report will be filed as well. As always, we can be
found at labs-l or on IRC in #wikimedia-labs.<br>
<div><br>
</div>
<div>Thanks,<br>
<br>
</div>
<div>Chase Pettet (on behalf of the Labs team)<br>
</div>
<div><br>
<br>
[1] <a moz-do-not-send="true"
href="https://www.openstack.org/software/liberty/">https://www.openstack.org/software/liberty/</a><br>
[2] <a moz-do-not-send="true"
href="https://lists.wikimedia.org/pipermail/labs-l/2016-July/004564.html">https://lists.wikimedia.org/pipermail/labs-l/2016-July/004564.html</a><br>
[3] <a moz-do-not-send="true"
href="https://phabricator.wikimedia.org/T142165">https://phabricator.wikimedia.org/T142165</a><br>
[4] <a moz-do-not-send="true"
href="https://lists.wikimedia.org/pipermail/labs-l/2016-August/004575.html">https://lists.wikimedia.org/pipermail/labs-l/2016-August/004575.html</a><br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Labs-announce mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Labs-announce@lists.wikimedia.org">Labs-announce@lists.wikimedia.org</a>
<a class="moz-txt-link-freetext" href="https://lists.wikimedia.org/mailman/listinfo/labs-announce">https://lists.wikimedia.org/mailman/listinfo/labs-announce</a>
</pre>
</blockquote>
<p><br>
</p>
</body>
</html>