Over the last week Željko and I changed the QA automated browser tests to
target test environments on the beta cluster as well as on test2wiki, where
they had been running for some time.
This was possible because of a number of improvements made to beta since
the QA Quarterly Review and the f2f meeting in SF in February:
* automatic updates to the beta database
* enabling Search on beta
* Varnish on beta to support MobileFrontend
* etc.
There is still more to do:
* get more extensions running and configured on beta
* change settings to more closely match prod and/or test2 as appropriate
* moar tests
Automated browser tests have a proven record of exposing real issues in the
test2wiki environment before code is put in production. Here are some of
the bugs we have identified by way of browser tests so far:
https://bugzilla.wikimedia.org/buglist.cgi?bug_status=UNCONFIRMED&bug_s…
By running these tests in the beta cluster also, we gain both better test
coverage features in disparate environments and also a longer window to
identify issues, since beta is updated much more often than test2wiki.
As always, the current status of the browser test builds are available in
Jenkins at
https://wmf.ci.cloudbees.com, and any changes to the status of
those builds are reported by wmf-jenkins-bot in #wikimedia-dev