<div dir="ltr">Oh, sorry, I was aware of the security hole and the Ops team's awesome (as usual) efforts to patch it. And I know this particular outage was due to the security hole, and I don't think anyone could reasonably expect it to have gone much better. But, in general, things like the updates to NFS have caused problems for people who value stability over improvements. Sorry if I brought this up in the middle of a different thread though.<div><br></div><div>As always, team Analytics appreciates all the great work that's done to keep Labs up. A lot of our tools run in Labs and that's great for many reasons.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 30, 2015 at 4:20 PM, John <span dir="ltr"><<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>This wasnt a case of improvements. This was an emergency patching of a security hole. All WMF servers had to be updated and rebooted, the only noticeable reboots where in labs where there isnt redundancy/standby services that can be swapped into place to prevent down time. <br><br></div>This is not to say that our process went without issues. I will be drafting a postmortem from the user's perspective with several ideas and suggestions this weekend. <br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 30, 2015 at 4:11 PM, Dan Andreescu <span dir="ltr"><<a href="mailto:dandreescu@wikimedia.org" target="_blank">dandreescu@wikimedia.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">FYI: some of us in Analytics are talking about a "graduate labs" kind of environment. Which would be more stable and treated a bit less experimentally. It's not something we're going to push for yet, but I was wondering how people here felt about it. It feels like our current labs environment is improving at an amazing rate, but sometimes we might want to prioritize stability over improvement.</div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Fri, Jan 30, 2015 at 4:05 PM, Harry Burt <span dir="ltr"><<a href="mailto:jarry1250@gmail.com" target="_blank">jarry1250@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div class="gmail_extra">Okay, gonna do something about my tools, they fall over every time this happens.<br><br></div><div class="gmail_extra">Is it just a case of<span style="font-family:monospace,monospace"> echo "webservice" > ".bigbrotherrc"</span> ? Or does the daemon have to be enabled?<br><br></div><div class="gmail_extra">Harry<br><br>--<br></div><div class="gmail_extra">Harry Burt (User:Jarry1250)<br></div><div class="gmail_extra"><br></div></div>
<br></div></div><span>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></span></blockquote></div><br></div>
<br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>