[QA] Upcoming work on test environments

[S Page]

On Tue, Aug 19, 2014 at 12:38 PM, Chris McMahon <cmcmahon at wikimedia.org>
wrote:

> * ...Right now performance on Jenkins is the main cause of false test
> failures, not only browser tests, but unit tests, qunit tests, etc.
>

Do you think the continual sporadic getaddrinfo failures[1] and  timeouts
running browsertests on SauceLabs are due to jenkins performance?



> ** ... a beta1 that adheres to our old policy of "nothing except master
> branch of code and config already in prod" and a beta2 for our current
> policy of "code that will be in prod eventually but is not now"
>
Names matter. "deploy-test" and "beta"?  "gamma" and "beta"?


>  ** Novel uses of vagrant and/or docker, such as the ability to easily
> create and provision a shared test environment for a particular dedicated
> purpose like a demo without having to build an entire mediawiki cluster
> from scratch.
>

That sounds like "create a labs instance", but with fewer pain points.
Clients like Jared Zimmerman can best express what they need for demos and
UX testing, but I think their problems include
* Loading a default useful set of content, media, and templates. I have no
idea how to automate this.
* Only having to do additional tweaking for the demo or test setup once. As
I understand it, OpenStack can take a snapshot of a running instance.[2]
but we don't expose this. So for demos or UX tests, one could set up the
test wiki, take a snapshot and then reload the snapshot for each demo or
test.


It would also be useful to automate keeping test instances up to date and
matching production config. For code there's `vagrant git-update`, but it
doesn't work in labs-vagrant; for configuration the vagrant settings.d
system doesn't share anything with production's
wmf-config/{CommonSettings,InitialiseSettings}.php. Could the puppet code
that makes beta-labs work so well "do the right thing" when run on a single
labs instance?



> Note that these are really good problems to have.
>
+++ :)


[1] https://bugzilla.wikimedia.org/show_bug.cgi?id=68125
[2] http://docs.openstack.org/openstack-ops/content/snapshots.html

-- 
=S Page  Features engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20140918/267f07c0/attachment.html>