As QA Lead for WMF, one of the things I want to do is to create an
institutional suite of automated cross-browser regression tests using
Selenium. I have two goals for this suite: first, to make it attractive
and convenient for the greater software testing community to both use the
suite locally and also to contribute to it. Second, to have the suite be a
useful regression test tool within WMF, tied to Beta Labs and controlled by
Jenkins.
For various reasons, I think the best language for this project is Ruby. I
realize that is a controversial choice, and I would like to explain my
reasoning. First let me address what I think will be the most serious
objections:
** Ruby gems are incompatible with Debian/Ubuntu apt packaging, making it
difficult or impossible to maintain Ruby code in production.
The selenium test suite is not intended to run on production servers.
There are two targets for this code. The first target is users' local
machines, including and especially Windows users. The second target is a
single dedicated headless Labs Ubuntu instance, controlled by Jenkins,
serving as a client to selenium-server, where selenium-server is running on
the various Windows etc. VMs that exist today on the WMF VPN.
** It's not PHP.
As of today, PHP has no complete or authoritative implementation of
selenium-webdriver/Selenium 2.0. That situation is unlikely to change any
time soon. This leaves a choice of Ruby or Python. For various reasons I
think Ruby is the superior choice.
** Design goals and their implementation.
In the interest of making this project as attractive as possible to the
greater testing community, I have defined a browser automation "stack",
using the most current and accepted practices for browser test automation.
The stack looks like this:
* Selenium-webdriver low-level "toolbox" API.
* Higher-level API for consistent access to pages and elements without the
user having to create their own handling for timeouts, exceptions, "stale
objects", multiple access criteria, etc.
* Modern, BDD-style assertions for pass/fail criteria (as opposed to xUnit
style assertions)
* "Page Object" design pattern in place and functioning out of the box
* Support for mobile emulators
* Institutional support for Jenkins integration
I submit that having these things in place and working when the user
downloads the suite is what will make this project attractive to the global
testing community.
Taking these point by point:
Even as I write this, the W3C is meeting in London to approve webdriver as
an internet standard, with Selenium 2.0 as the reference implementation of
that standard. This means that the selenium 2.0 API is only a fraction the
size of that of Selenium 1.0, and serves a very different purpose. The
selenium 2.0 API can be considered a "toolbox" from which higher-level APIs
are constructed. Both Ruby and Python have full implementation and support
for the Selenium 2.0 API.
At higher levels of the stack, the Python and Ruby communities take
different design approaches, with Python being more DIY/NIH and Ruby being
more shared and organized around communities of practice. This is what
makes Ruby so attractive as I implemented the rest of the stack.
Watir (Web Application Testing In Ruby) is a browser test project that
pre-dates Selenium itself. Watir provides an intuitive, thoughtful,
well-designed high-level API for access to pages and elements that the
native selenium-webdriver API does not. While historically Watir and
Selenium have been different projects, as of about 2010 watir-webdriver is
simply a wrapper for the low-level selenium API that preserves all of the
convenience and good design Watir has always had. Watir is in use today at
Facebook, The Gap, and many other places. Python has no such equivalent
high-level API.
Behavior Driven Development (BDD) style assertions are implemented in Ruby
with the Rspec library. This approach to assertions has been a generally
accepted standard in the Ruby community for some time. "The Rspec Book"
was published by Pragmatic Press in 2010. Python has no equivalent
BDD-style assertion library. In fact, Rspec takes advantage of Ruby's
affordance for metaprogramming, something not possible in Python.
"Page Objects" is a design pattern for browser testing that has become a
generally accepted practice in the browser testing community, with a lot of
momentum particularly in the last one or two years. Page Objects have
institutional support in Ruby with the 'page-object' gem, which supports
both watir-webdriver and selenium-webdriver syntax. Python has no such
institutional support for page objects, each Python project using page
objects implements the pattern locally, from scratch.
Some support for mobile emulators is provided by Ruby's
webdriver-user-agent gem, created by Alister Scott of Thoughtworks. This
gem piggybacks on either the watir-webdriver or selenium-webdriver APIs in
Ruby. I am unaware of any such mobile emulator support in Python.
Ruby selenium tests are run via 'rake', the Ruby version of make. Jenkins
has an officially supported plugin for rake.
Both Ruby and Python are viable choices for this project. The difference
is that Python would require a lot of custom infrastructure and
scaffolding, with all of the risk and maintenance that that entails, where
support for the elements of the "stack" in Ruby is already in place,
well-designed, and supported by the greater Ruby community. Not only does
this make maintenance of the stack for WMF purposes much simpler, it also
makes the project significantly more attractive to casual users in the
testing community who can get started with the suite immediately by simply
installing a few gems locally and reading some public documentation, rather
than having to face a daunting pile of custom code.
For reference and further research, I created a prototype of this project
that implements this stack here:
https://github.com/chrismcmahon/Page-Object-WMF-spike
Addendum: I am aware of two projects at WMF already using Selenium, one
with OmniTI for AFTv5, the other from Jeremy Postlethwaite. Since the
purpose of this project is to attract members of the global testing
community, and to provide an institutional regression test suite for
Mediawiki/Wikipedia, I see no reason to require individual WMF projects to
use this framework, although I suspect that individual WMF projects might
find this framework more convenient than other approaches.