[QA] Flagging some Beta Labs issues

[Chris McMahon]

On Mon, Aug 18, 2014 at 4:35 PM, Maryana Pinchuk <mpinchuk at wikimedia.org>
wrote:

> Greetings, QAers,
>
> I'm not entirely sure who the point person for Beta Labs is currently, but
> I wanted to make sure you guys are aware that there have been a lot of
> issues (i.e., partial or total outages) with this environment in the past
> 2-3 weeks, most likely due to ongoing HHVM work. Unfortunately, because
> several teams at WMF rely quite heavily on Beta Labs – such as Mobile Web
> for testing new user-facing features before they go live in production, and
> Design/UX for running remote and in-person user tests – this is not so
> great.
>

I've started talking to people informally about the current situation with
beta labs.  TLDR: it is FANTASTIC that beta is so valuable to so many teams
and people; but it is difficult to support so many projects on a single
beta labs cluster.

Originally beta labs was only to be a target for automated browser tests.
 We needed a shared test environment(*) where "works on my machine" was not
an option(**). Beta labs unexpectedly and organically grew into something
more important than just that.

Then we had a policy of using beta only for software already in production.
 We broke that policy with CirrusSearch, followed by Flow, and now by
testing HHVM in beta labs.

I am starting to think we should have at least one more shared cluster test
environment running the master branch of code, i.e. a "beta2" test
environment.  I would like one for modeling the production cluster (like
the original beta concept) and a different one for working on system-wide,
cross-cutting changes (like CirrusSearch, Flow, HHVM, etc.)  I could see
having yet another "beta3" env for working with code not yet merged to
master.

And even the original beta labs could probably use more CPU/memory/etc than
it has now.

Thanks for bringing this up Maryana, I think the current and future state
of beta labs is something that needs scrutiny sooner rather than later.
-Chris

(*) Just a note, shared test environments like beta labs are out of fashion
right now, I remain pleased that ours is so valuable.  We should probably
blog that somewhere.
(**) We should also take a serious look at what is and is not possible with
beta and with vagrant, and where to draw the line between a shared test env
like beta and shared dev envs like vagrant images.




> For example, the outage during Wikimania was particularly bad because
> Abbey was hosting a UX testing table during the conference that had
> scripted user tests all pointing to Beta Labs, so she was unable to run any
> of those tests and lost out on valuable user testing time.
>
> To avoid situations like this in the future, is there a way for teams who
> use Beta Labs for testing to stay in closer sync with its maintainers? I
> realize y'all aren't mind-readers ;) and there will of course be unexpected
> issues that crop up from time to time. But if there are likely to be more
> major breaking changes to the infrastructure while you continue working on
> HHVM, it would be great to get an advanced heads-up so we can plan
> accordingly. And when unexpected outages do occur, it'd also be good to
> know who to report them to and check in on progress with, because I'm not
> sure the current strategy of whining and hoping it'll fix itself is
> working/sane/scalable :)
>
> I don't know how this would work ideally (regular updates to a
> new/existing mailing list? dropping in on a new/existing IRC channel? a
> wiki page? some combo of all three?) but am open to any & all ideas! And as
> a start, it would be good if those of you currently tasked with maintaining
> Beta Labs could give a virtual wave – I *think* I know who you are, but
> Abbey's new and almost certainly doesn't ;)
>
> --
> Maryana Pinchuk
> Product Manager, Wikimedia Foundation
> wikimediafoundation.org
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140818/53bef572/attachment.html>