You should talk to Jeremy Postlethwaite jpostlethwaite@wikimedia.org. He did some work on setting up automated Selenium tests for our fundraiser.
Ryan Kaldari
On 4/5/12 2:45 PM, Chris McMahon wrote:
As QA Lead for WMF, one of the things I want to do is to create an institutional suite of automated cross-browser regression tests using Selenium. I have two goals for this suite: first, to make it attractive and convenient for the greater software testing community to both use the suite locally and also to contribute to it. Second, to have the suite be a useful regression test tool within WMF, tied to Beta Labs and controlled by Jenkins.
For various reasons, I think the best language for this project is Ruby. I realize that is a controversial choice, and I would like to explain my reasoning. First let me address what I think will be the most serious objections:
** Ruby gems are incompatible with Debian/Ubuntu apt packaging, making it difficult or impossible to maintain Ruby code in production.
The selenium test suite is not intended to run on production servers. There are two targets for this code. The first target is users' local machines, including and especially Windows users. The second target is a single dedicated headless Labs Ubuntu instance, controlled by Jenkins, serving as a client to selenium-server, where selenium-server is running on the various Windows etc. VMs that exist today on the WMF VPN.
** It's not PHP.
As of today, PHP has no complete or authoritative implementation of selenium-webdriver/Selenium 2.0. That situation is unlikely to change any time soon. This leaves a choice of Ruby or Python. For various reasons I think Ruby is the superior choice.
** Design goals and their implementation.
In the interest of making this project as attractive as possible to the greater testing community, I have defined a browser automation "stack", using the most current and accepted practices for browser test automation. The stack looks like this:
- Selenium-webdriver low-level "toolbox" API.
- Higher-level API for consistent access to pages and elements without the
user having to create their own handling for timeouts, exceptions, "stale objects", multiple access criteria, etc.
- Modern, BDD-style assertions for pass/fail criteria (as opposed to xUnit
style assertions)
- "Page Object" design pattern in place and functioning out of the box
- Support for mobile emulators
- Institutional support for Jenkins integration
I submit that having these things in place and working when the user downloads the suite is what will make this project attractive to the global testing community.
Taking these point by point:
Even as I write this, the W3C is meeting in London to approve webdriver as an internet standard, with Selenium 2.0 as the reference implementation of that standard. This means that the selenium 2.0 API is only a fraction the size of that of Selenium 1.0, and serves a very different purpose. The selenium 2.0 API can be considered a "toolbox" from which higher-level APIs are constructed. Both Ruby and Python have full implementation and support for the Selenium 2.0 API.
At higher levels of the stack, the Python and Ruby communities take different design approaches, with Python being more DIY/NIH and Ruby being more shared and organized around communities of practice. This is what makes Ruby so attractive as I implemented the rest of the stack.
Watir (Web Application Testing In Ruby) is a browser test project that pre-dates Selenium itself. Watir provides an intuitive, thoughtful, well-designed high-level API for access to pages and elements that the native selenium-webdriver API does not. While historically Watir and Selenium have been different projects, as of about 2010 watir-webdriver is simply a wrapper for the low-level selenium API that preserves all of the convenience and good design Watir has always had. Watir is in use today at Facebook, The Gap, and many other places. Python has no such equivalent high-level API.
Behavior Driven Development (BDD) style assertions are implemented in Ruby with the Rspec library. This approach to assertions has been a generally accepted standard in the Ruby community for some time. "The Rspec Book" was published by Pragmatic Press in 2010. Python has no equivalent BDD-style assertion library. In fact, Rspec takes advantage of Ruby's affordance for metaprogramming, something not possible in Python.
"Page Objects" is a design pattern for browser testing that has become a generally accepted practice in the browser testing community, with a lot of momentum particularly in the last one or two years. Page Objects have institutional support in Ruby with the 'page-object' gem, which supports both watir-webdriver and selenium-webdriver syntax. Python has no such institutional support for page objects, each Python project using page objects implements the pattern locally, from scratch.
Some support for mobile emulators is provided by Ruby's webdriver-user-agent gem, created by Alister Scott of Thoughtworks. This gem piggybacks on either the watir-webdriver or selenium-webdriver APIs in Ruby. I am unaware of any such mobile emulator support in Python.
Ruby selenium tests are run via 'rake', the Ruby version of make. Jenkins has an officially supported plugin for rake.
Both Ruby and Python are viable choices for this project. The difference is that Python would require a lot of custom infrastructure and scaffolding, with all of the risk and maintenance that that entails, where support for the elements of the "stack" in Ruby is already in place, well-designed, and supported by the greater Ruby community. Not only does this make maintenance of the stack for WMF purposes much simpler, it also makes the project significantly more attractive to casual users in the testing community who can get started with the suite immediately by simply installing a few gems locally and reading some public documentation, rather than having to face a daunting pile of custom code.
For reference and further research, I created a prototype of this project that implements this stack here: https://github.com/chrismcmahon/Page-Object-WMF-spike
Addendum: I am aware of two projects at WMF already using Selenium, one with OmniTI for AFTv5, the other from Jeremy Postlethwaite. Since the purpose of this project is to attract members of the global testing community, and to provide an institutional regression test suite for Mediawiki/Wikipedia, I see no reason to require individual WMF projects to use this framework, although I suspect that individual WMF projects might find this framework more convenient than other approaches. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l