When we began creating cross-browser regression tests using Selenium, we
kept them all in a repository named '/qa/browsertests'. We created
regression tests for features across all WMF development teams and Platform
functions, from the appearance of Preferences to navigating UploadWizard to
ArticleFeedback and PageTriage to MobileFrontend to
UniversalLanguageSelector etc. etc. etc. We run these tests against some
combinations of environments in beta labs cluster, test2wiki, and
production.
There will probably always be a need for a grab-bag repo of general
cross-browser regression tests, but managing tests in this way has some
drawbacks:
* builds run for all features at the same time, regardless of the
development process for each feature
* failed builds can be caused by any feature, and it takes some effort to
find the source of any failed build
We have taken some steps recently to improve how we manage and run these
tests. In particular, we have begun moving certain browser tests out of
the /qa/browsertests repo and into the repos of the particular extensions
or features being tested. From there, we are running individual builds for
those features in the appropriate test environments.
Our first model for doing this was MobileFrontend. The browser tests for
MobileFrontend have resided in /MobileFrontend/tests/acceptance for some
time now, and building from that repo has proven quite effective.
Not long ago Nik Everett created an extensive set of browser tests for
CirrusSearch and very quickly put them under /CirrusSearch/tests/browser.
Work on CirrusSearch is proceeding quickly and these tests fail often for
now, so it is nice to have them in a repo where they can be examined
outside of the main /qa/browsertest builds.
Back in May of this year we created a number of browser tests for the
UniversalLanguageSelector with help from Runa Bhattacharjee and Niklas
Laxström, and Amir Aharoni has been doing work on those in the time since.
Recently at the request of the Language team we moved the ULS browser
tests to the repo at UniversalLanguageSelector/tests/browser, and we're
running those in their own build as well.
Finally, we have been creating some browser tests for VisualEditor that
will be particularly useful for cross-browser acceptance testing and
regression testing. Of course the tests are valuable right now as well,
they help us identify important issues like
https://bugzilla.wikimedia.org/show_bug.cgi?id=53360. These tests today
reside under VisualEditor/modules/ve-mw/test/browser. Rachel Thomas, our
intern with the Outreach Program for Women has contributed a number of
VisualEditor browser tests.