[QA] Upcoming work on test environments

[Chris McMahon]

The thread about beta labs issues points up some issues that we in QA have
been dealing with for some time, but that only affects individual teams
from time to time mostly. The Release Engineering group is wrapping up one
quarter's work and planning the next one, so this is a good time to think
about what happens next.  A subset of RelEng met today and we would like to
make a few themes for upcoming projects:

* Jenkins performance.  Jenkins controls more and more of our software
build and test, but as Jenkins has grown, it has become more complex and
also more slow.  Right now performance on Jenkins is the main cause of
false test failures, not only browser tests, but unit tests, qunit tests,
etc.  Right now we have essentially no performance monitoring of Jenkins.
 We can throttle this and move that, but we really don't know where the
bottlenecks are or what causes them.

* Test environments:  beta labs, vagrant, and the possibility of creating
new and novel test environment.   I would like to put some effort into
investigating:
** A second beta labs where we could test system-wide changes like HHVM
without disrupting normal operation of software in production.  In other
words, have a beta1 that adheres to our old policy of "nothing except
master branch of code and config already in prod" and a beta2 for our
current policy of "code that will be in prod eventually but is not now"
(CirrusSearch, Flow, and HHVM were all in beta before they were in prod).
 Note that we would want to target both environments with tests, so this
will put even more load on Jenkins.  This may not be possible, but I hope
it is.
** More better vagrant. Vagrant support for every significant extension out
of the box.
** Novel uses of vagrant and/or docker, such as the ability to easily
create and provision a shared test environment for a particular dedicated
purpose like a demo without having to build an entire mediawiki cluster
from scratch.

* A clear understanding of which test environments are appropriate for
particular purposes. Recently I had someone ask how to automate the CAPTCHA
when creating an account on our shared public test environment beta labs
and I had to point out that if it were possible to automate the CAPTCHA we
would have a really big problem on our hands.

Note that these are really good problems to have. Now that browser tests
are no longer relegated to the cloudbees Jenkins and the /qa/browsertests
repo, we can concentrate on being good citizens in gerrit and Jenkins.  Now
that our systems are reliable enough to contemplate big changes like
CirrusSearch and HHVM, we are in a position to provide an environment to
test those big changes reliably.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140819/d409a1e7/attachment.html>