[QA] Upcoming work on test environments

[S Page]

Flow code that worked on beta labs broke Special:Contributions and
Special:RecentChanges when deployed to production today. We missed it
because beta is running HHVM and production is running Zend PHP (and
developers tend to be running newer versions of the latter).  ErikB and
Reedy quickly fixed it[1], but %$#@ :-( !

This indicates that it is a high priority to have separate test
environments for:
1) "if master works here it'll work in production on Thursday" and
2) "do these upcoming features work?"
which is what Chris McMahon describes, excerpted below. So +7 to that.

In the meantime is there anything we can do to give us 1)?  E.g. make beta
cluster run HHVM according to an environment variable, the way AIUI
production is set up, and re-run all our tests with/without that setting?
The Flow team has a "kind of close to production" labs instance that we use
for testing, but there's no automated way to keep its configuration in
sync. I know some devs have hacked their local mediawiki to use
CommonSettings/InitialiseSettings , but I don't think there's a puppet role
for this.

Thanks for any ideas.

On Tue, Aug 19, 2014 at 12:38 PM, Chris McMahon <cmcmahon at wikimedia.org>
wrote:

> ...
> * Test environments:  beta labs, vagrant, and the possibility of creating
> new and novel test environment.   I would like to put some effort into
> investigating:
> ** A second beta labs where we could test system-wide changes like HHVM
> without disrupting normal operation of software in production.  In other
> words, have a beta1 that adheres to our old policy of "nothing except
> master branch of code and config already in prod" and a beta2 for our
> current policy of "code that will be in prod eventually but is not now"
> (CirrusSearch, Flow, and HHVM were all in beta before they were in prod).
>  Note that we would want to target both environments with tests, so this
> will put even more load on Jenkins.  This may not be possible, but I hope
> it is.
>

[1]  https://gerrit.wikimedia.org/r/#/c/161330/

-- 
=S Page  Features engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20140918/a326c1e8/attachment-0001.html>