[QA] Upcoming work on test environments

S Page spage at wikimedia.org
Thu Sep 18 21:09:39 UTC 2014

On Tue, Aug 19, 2014 at 12:38 PM, Chris McMahon <cmcmahon at wikimedia.org>

> * ...Right now performance on Jenkins is the main cause of false test
> failures, not only browser tests, but unit tests, qunit tests, etc.

Do you think the continual sporadic getaddrinfo failures[1] and  timeouts
running browsertests on SauceLabs are due to jenkins performance?

> ** ... a beta1 that adheres to our old policy of "nothing except master
> branch of code and config already in prod" and a beta2 for our current
> policy of "code that will be in prod eventually but is not now"
Names matter. "deploy-test" and "beta"?  "gamma" and "beta"?

>  ** Novel uses of vagrant and/or docker, such as the ability to easily
> create and provision a shared test environment for a particular dedicated
> purpose like a demo without having to build an entire mediawiki cluster
> from scratch.

That sounds like "create a labs instance", but with fewer pain points.
Clients like Jared Zimmerman can best express what they need for demos and
UX testing, but I think their problems include
* Loading a default useful set of content, media, and templates. I have no
idea how to automate this.
* Only having to do additional tweaking for the demo or test setup once. As
I understand it, OpenStack can take a snapshot of a running instance.[2]
but we don't expose this. So for demos or UX tests, one could set up the
test wiki, take a snapshot and then reload the snapshot for each demo or

It would also be useful to automate keeping test instances up to date and
matching production config. For code there's `vagrant git-update`, but it
doesn't work in labs-vagrant; for configuration the vagrant settings.d
system doesn't share anything with production's
wmf-config/{CommonSettings,InitialiseSettings}.php. Could the puppet code
that makes beta-labs work so well "do the right thing" when run on a single
labs instance?

> Note that these are really good problems to have.
+++ :)

[1] https://bugzilla.wikimedia.org/show_bug.cgi?id=68125
[2] http://docs.openstack.org/openstack-ops/content/snapshots.html

=S Page  Features engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20140918/267f07c0/attachment.html>

More information about the QA mailing list