Summary: architecture, design, hosting, and APIs for the WMF automated browser test project are in place, useful testing is happening now, and we are beginning to build out the test suite for both desktop and mobile browsers. About a year ago the WMF created a mandate to implement browser-level automated testing. I did the preliminary work, and in October 2012 Željko Filipin joined WMF to be the technical lead for this project. This is an excellent time to describe the current status of the project and to discuss where it goes next. We are using the Ruby programming language for this project because the tools available for this sort of testing are more sophisticated and better supported in Ruby than in other languages. While the entire testing "stack" is remarkable, three aspects are particularly worth noting: A project called webdriver is in the process of being approved by the W3C as an internet standard for browser automation. Selenium is the client-side implementation of the webdriver standard. The Ruby implementation of selenium-webdriver is built and tested automatically according to the webdriver standard. As the webdriver standard changes, so selenium-webdriver in Ruby changes also, automatically. [1] But selenium is a low-level API for driving browsers, essentially a limited set of tools to build with. Ruby is unique in also having a standard high-level API. Today the watir-webdriver project provides a powerful, well-designed, well-supported high-level API for selenium-webdriver based on the HTML5 standard.[2] Cucumber is a high-level API for doing Acceptance Test Driven Development, or ATDD. Cucumber allows us to create executable specifications in the form of Given/When/Then statements. For example, here is a working Cucumber specification for a login test: Scenario: Log in without entering credentials Given I am at test2 Log in page When I log in without entering credentials Then Log in page should open And feedback should be You have not specified a valid username. Cucumber is valuable because it creates a communication channel that does not usually exist for software development projects. For one thing, casual users and the greater community can read the Cucumber specifications[3] and understand what is being tested (and what is failing!). In a similar way, Cucumber allows non-programming members of the community to contribute to the test automation project by submitting test scenarios to be automated. We have created a test backlog page on mediawiki.org[4] for such scenarios. The tests themselves are controlled by an instance of the Jenkins continuous integration server provided by a hosting service called Cloudbees. The tests are run on hosts managed by Sauce Labs, and are currently running on Firefox, Chrome, IE6/7/8/9/10, android, iphone, and ipad, using test2wiki, beta labs, and the mobile web site as targets. As of today, the status of all the tests are available publicly[5]. Using Cloudbees to host browser tests provides a number of advantages. Cloudbees provides free and seamless integration with Sauce Labs and all of their tools. With Cloudbees we have fine control over versions of the software we use, and we can easily update versions of Ruby gems and Ruby itself at will. It is possible to publish information between the Cloudbees Jenkins instance and the WMF Jenkins also, although we have not yet done this. Cloudbees is economical. Finally, although managed by a commercial company, every aspect of the integrated service is open source software. There is little risk in using Cloudbees. The source code for the tests themselves resides in WMF's gerrit system[6]. If Cloudbees were to disappear tomorrow, we could re-create the services it provides on WMF hosts or other hosts, although at some expense both financially and in terms of maintenance. Željko and I spent significant time working on the design and implementation of our first few tests. After that tweaking was essentially done, the tests we have came fully online November 26th. Since that time even our few existing tests have identified the following issues: UploadWizard broken for IE9 only: https://bugzilla.wikimedia.org/show_bug.cgi?id=42512 UploadWizard mistakenly reverted to broken version on test2, in deployment pipeline. (no BZ ticket, fixed on the fly by Ryan Kaldari) AFTv5 user experience issue with IE9: https://bugzilla.wikimedia.org/show_bug.cgi?id=42551 AFTv5 user experience issue with all IEs, and later breakage in the same code: https://bugzilla.wikimedia.org/show_bug.cgi?id=42517 The tests have also exposed important issues with our test environments themselves: https://bugzilla.wikimedia.org/show_bug.cgi?id=42736 And seen just today, UW broken for IE6/IE7: https://bugzilla.wikimedia.org/show_bug.cgi?id=4292 This week I will be publishing detailed technical design and architecture documentation for those who might like to contribute beyond simply adding to the backlog of tests to be implemented. It is worth noting that browser test automation projects are expensive and often fail, and even WMF has at least one example of such a failure in its history. In my experience, such failures are almost always caused by poor design. Željko and I have been specialists in browser test automation with open source tools since such tools came into existence, and we intend for this project to continue to be a world class reference implementation of such a project as it grows. We will be presenting a walkthrough of the project this Thursday[7]. [1] https://speakerdeck.com/jarib/automating-130-browser-platform-and-language-… [2] http://rdoc.info/gems/watir-webdriver/#API_docs [3] a sophisticated Cucumber test specification in use right now: https://github.com/wikimedia/qa-browsertests/blob/master/features/uploadwiz… [4] http://www.mediawiki.org/wiki/Qa/test_backlog [5] https://wmf.ci.cloudbees.com/ NOTE: current Firefox failures are due to a bug in selenium to be fixed in next release; IE6/IE7 failure due to https://bugzilla.wikimedia.org/show_bug.cgi?id=42922 (new behavior today), IE6 failure because IE6 is not supported for AFTv5 those test should be turned off. [6] https://gerrit.wikimedia.org/r/#/admin/projects/qa/browsertests also https://github.com/wikimedia/qa-browsertests [7] https://www.mediawiki.org/wiki/Meetings/2012-12-13