[QA] Upcoming work on test environments

[Greg Grossmeier]

<quote name="S Page" date="2014-09-18" time="14:09:39 -0700">
> > ** ... a beta1 that adheres to our old policy of "nothing except master
> > branch of code and config already in prod" and a beta2 for our current
> > policy of "code that will be in prod eventually but is not now"
> >
> Names matter. "deploy-test" and "beta"?  "gamma" and "beta"?

Here's my current proposal (as defined in my head and I'm open to
critique/improvements), it's basically Chris's proposal[0] plus some extra
bits:

1) Two test clusters in the wmflabs environment (spoiler: that means
neither would be really useful for performance testing)

2) One cluster for things already in, or close to be in, production.
"conservative beta cluster"

3) One cluster for other things (ie: where HHVM would have gone
initially, had we had it). "liberal beta cluster"

NB: Just getting the second cluster up and responding to requests
correctly is a big undertaking right now, especially since we're going
to wait on my labs hardware before doing so.


4) On both of those clusters, we use multiversion to do:
  a) "constantly" updating master eg Beta Cluster right now
  b) a nightly branch that gets cut near end of day

Thus we have a matrix:

,------------------------------------------,
|             |   continuous   |   nightly |
|------------------------------------------|
|conservative |       A        |     B     |
|------------------------------------------|
|liberal      |       C        |     D     |
'------------------------------------------'

5) We run the whole barrage of tests as we do now against the two
clusters' continuous deployment wikis (A and C, above). This probably
happens at the same times as it does now.

6) We run the same barrage of tests against the two clusters' nightly
deployment wikis (B and D, above). This will give us a good sanity check
of "this branch is good to go out".

7) This would give us the ability to do the proposal I have outlined
here: 
https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Wishlist#Proposal

8) Help with naming A, B, C, and D much appreciated.



Greg

[0] https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Second_Beta_Cluster

-- 
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |