<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Aug 18, 2014 at 4:35 PM, Maryana Pinchuk <span dir="ltr"><<a href="mailto:mpinchuk@wikimedia.org" target="_blank">mpinchuk@wikimedia.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Greetings, QAers,<div><br></div><div>I'm not entirely sure who the point person for Beta Labs is currently, but I wanted to make sure you guys are aware that there have been a lot of issues (i.e., partial or total outages) with this environment in the past 2-3 weeks, most likely due to ongoing HHVM work. Unfortunately, because several teams at WMF rely quite heavily on Beta Labs – such as Mobile Web for testing new user-facing features before they go live in production, and Design/UX for running remote and in-person user tests – this is not so great. </div>
</div></blockquote><div><br></div><div>I've started talking to people informally about the current situation with beta labs. TLDR: it is FANTASTIC that beta is so valuable to so many teams and people; but it is difficult to support so many projects on a single beta labs cluster.</div>
<div><br></div><div>Originally beta labs was only to be a target for automated browser tests. We needed a shared test environment(*) where "works on my machine" was not an option(**). Beta labs unexpectedly and organically grew into something more important than just that. </div>
<div><br></div><div>Then we had a policy of using beta only for software already in production. We broke that policy with CirrusSearch, followed by Flow, and now by testing HHVM in beta labs.</div><div><br></div><div>I am starting to think we should have at least one more shared cluster test environment running the master branch of code, i.e. a "beta2" test environment. I would like one for modeling the production cluster (like the original beta concept) and a different one for working on system-wide, cross-cutting changes (like CirrusSearch, Flow, HHVM, etc.) I could see having yet another "beta3" env for working with code not yet merged to master. </div>
<div><br></div><div>And even the original beta labs could probably use more CPU/memory/etc than it has now. </div><div><br></div><div>Thanks for bringing this up Maryana, I think the current and future state of beta labs is something that needs scrutiny sooner rather than later.</div>
<div>-Chris</div><div><br></div><div>(*) Just a note, shared test environments like beta labs are out of fashion right now, I remain pleased that ours is so valuable. We should probably blog that somewhere. </div><div>(**) We should also take a serious look at what is and is not possible with beta and with vagrant, and where to draw the line between a shared test env like beta and shared dev envs like vagrant images. </div>
<div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>For example, the outage during Wikimania was particularly bad because Abbey was hosting a UX testing table during the conference that had scripted user tests all pointing to Beta Labs, so she was unable to run any of those tests and lost out on valuable user testing time. </div>
<div><br></div><div>To avoid situations like this in the future, is there a way for teams who use Beta Labs for testing to stay in closer sync with its maintainers? I realize y'all aren't mind-readers ;) and there will of course be unexpected issues that crop up from time to time. But if there are likely to be more major breaking changes to the infrastructure while you continue working on HHVM, it would be great to get an advanced heads-up so we can plan accordingly. And when unexpected outages do occur, it'd also be good to know who to report them to and check in on progress with, because I'm not sure the current strategy of whining and hoping it'll fix itself is working/sane/scalable :) </div>
<div><br></div><div>I don't know how this would work ideally (regular updates to a new/existing mailing list? dropping in on a new/existing IRC channel? a wiki page? some combo of all three?) but am open to any & all ideas! And as a start, it would be good if those of you currently tasked with maintaining Beta Labs could give a virtual wave – I *think* I know who you are, but Abbey's new and almost certainly doesn't ;)<span class="HOEnZb"><font color="#888888"><br clear="all">
<div><br></div>-- <br><div dir="ltr">Maryana Pinchuk<br>Product Manager, Wikimedia Foundation<br><a href="http://wikimediafoundation.org" target="_blank">wikimediafoundation.org</a><br></div>
</font></span></div></div>
<br>_______________________________________________<br>
QA mailing list<br>
<a href="mailto:QA@lists.wikimedia.org">QA@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/qa" target="_blank">https://lists.wikimedia.org/mailman/listinfo/qa</a><br>
<br></blockquote></div><br></div></div>