<div dir="ltr">(I added Ori)<br><div><br>On Mon, Aug 18, 2014 at 4:35 PM, Maryana Pinchuk <span dir="ltr"><<a href="mailto:mpinchuk@wikimedia.org" target="_blank">mpinchuk@wikimedia.org</a>></span> wrote:<br><div class="gmail_extra">
<div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Greetings, QAers,<div><br></div><div>I'm not entirely sure who the point person for Beta Labs is currently,</div>
</div></blockquote><div>Forces under Greg Grossmeier?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div> but I wanted to make sure you guys are aware that there have been a lot of issues (i.e., partial or total outages) with this environment in the past 2-3 weeks, most likely due to ongoing HHVM work.</div>
</div></blockquote><div><br>Perhaps, but also there was disk full and rsync problems that ErikB and hashar fixed. Many of the people with the ability to fix problems were out.<br></div><div>Beta labs has always been unstable, but (roughly since the eqiad move) it's been pretty good -- retry 20 minutes after a failure and it's working again. Maybe stability improved thanks to all our test automation, or maybe we got lucky.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div> Unfortunately, because several teams at WMF rely quite heavily on Beta Labs – such as Mobile Web for testing new user-facing features before they go live in production, </div>
</div></blockquote><div>Fine, so long as you realize you're also testing everyone else's changes that are about to go into production.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div>and Design/UX for running remote and in-person user tests</div></div></blockquote><div><br>That seems crazy. Beta labs constantly updates with the latest merged changes to core and dozens of extensions, I think 288 times a day. Any commit could break it, and they regularly do! Beta labs is there to test to find breaking changes before they go live, not for user testing.<br>
<br>Maybe we should document better how to set up a reasonably performant labs instance with a decent set of wiki pages, templates, images, etc. Then UX can spin up <a href="http://ux-wikimania.wmflabs.org">ux-wikimania.wmflabs.org</a> and be isolated from the firehose of changes.<br>
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">To avoid situations like this in the future, is there a way for teams who use Beta Labs for testing to stay in closer sync with its maintainers? </div>
</blockquote><div><br>People on #wikimedia-labs (bd808, Coren, hashar, Reedy, et al) are very responsive if they're not away :) , then I alert #wikimedia-qa for test failures and if there's no response and beta labs is badly broken (e.g. no Main_Page) then I visit #wikimedia-operations. If you can figure out what extension's causing the problem, visit its IRC channel.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I realize y'all aren't mind-readers ;) and there will of course be unexpected issues that crop up from time to time. But if there are likely to be more major breaking changes to the infrastructure while you continue working on HHVM, it would be great to get an advanced heads-up so we can plan accordingly.</div>
</blockquote><div>See Chris' proposal below for an alpha cluster for breaking changes. Until then I dunno... status page? Archived e-mail list?<br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">And when unexpected outages do occur, it'd also be good to know who to report them to and check in on progress with, because I'm not sure the current strategy of whining and hoping it'll fix itself is working/sane/scalable :) </div>
</blockquote><div><br>I can't see anything better than "Beta labs isn't working when I do X, what's up?" on IRC.<br><br></div></div><br></div><div class="gmail_extra">Chris McMahon replied<br><br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
we had a policy of using beta only for software already in production.
We broke that policy with CirrusSearch, followed by Flow, and now by
testing HHVM in beta labs.<br></blockquote><div><br></div><div>Flow was just following orders. We're implicitly told to test new extensions on Beta labs "for weeks" before getting the OK to deploy to production.[1]<br>
<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">I would like [more shared cluster environments] for modeling the production cluster (like the original
beta concept) and a different one for working on system-wide,
cross-cutting changes (like CirrusSearch, Flow, HHVM, etc.)</blockquote><div><br>That sounds fantastic. Currently beta labs is both for "Sanity check of merged code before it rolls out to <a href="http://mediawiki.org">mediawiki.org</a> on Thursday" and for "See how my new project fares in a production-like environment."<br>
</div><div> </div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"> I could see
having yet another "beta3" env for working with code not yet merged to
master. <br></blockquote><div><br></div><div>Isn't that what per-project labs instances are for? How would teams identify what goes onto beta3? It risks becoming an odd graveyard like <a href="http://test.wikipedia.org">test.wikipedia.org</a>.<br>
<br></div><div>Cheers, and thanks for making beta labs a part of our lives.<br></div><div> </div></div>[1] <a href="https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Case_1d:_new_extension">https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Case_1d:_new_extension</a><br>
<br>-- <br><div dir="ltr">=S Page Features engineer<br></div>
</div></div></div>