Chris McMahon <cmcmahon <at> wikimedia.org> writes:
As QA Lead for WMF, one of the things I want to do is to create an institutional suite of automated cross-browser regression tests using Selenium. I have two goals for this suite: first, to make it attractive and convenient for the greater software testing community to both use the suite locally and also to contribute to it. Second, to have the suite be a useful regression test tool within WMF, tied to Beta Labs and controlled by Jenkins.
For various reasons, I think the best language for this project is Ruby. I realize that is a controversial choice, and I would like to explain my reasoning. First let me address what I think will be the most serious objections:
[...]
** It's not PHP.
As of today, PHP has no complete or authoritative implementation of selenium-webdriver/Selenium 2.0. That situation is unlikely to change any time soon. This leaves a choice of Ruby or Python. For various reasons I think Ruby is the superior choice.
Not sure what counts as authoritative, but there are a number of fairly usable PHP implementations such as php-webdriver [1] from Facebook or phpunit-selenium [2] from the PHPUnit framework, both of which are non-complete but very easy to extend (and in practice, you don't use most Selenium commands anyway). Using one of them is more troublesome than choosing a language in which there is a reference Selenium implementation, but on the other hand, you don't need to introduce another language, you can write the tests in a language all MediaWiki developers are comfortable with, and you leave open the option of reusing MediaWiki components in tests to handle setup/teardown of fixtures in a clean way.
Also, my (admittedly very superficial) experience with BDD is that Cucumber/Gherkin is much better for acceptance testing than RSpec (which is more suited for unit testing). Gherkin tests are clean, human-readable descriptions which are easier to read than program code, and can be easily understood by non- developers (end users, QA people, managers) even if they have no idea what a programming language is. On the other hand Gherkin is not a real programming language, so you lose some flexibility (such as the ability to use page objects), but IMO it is well worth it. And while RSpec relies on Ruby's elegant but obscure poetry mode, and thus cannot be easily copied in other languages, Gherkin has a simple custom syntax which is trivial to implement in any language; specifically, there is a good PHP implementation called Behat [3] which has its own Selenium implementation (Mink [4]) but also can be used with any other Selenium library.
Mink has the additional advantage that it abstracts away the Selenium interface so that Selenium can be replaced with some other browser simulator without changing the tests; while that doing Selenium-specific things more complicated, it can yield huge speedups for test which don't require Javascript and so Selenium can be replaced with some simple browser emulator. (Yes, Selenium2 has its own browser emulator, but it is still a fair bit slower than something like Goutte [5]).
So maybe a PHP - Mink (or other Selenium library) - Behat stack instead of a Ruby - Watir - RSpec stack would be worth considering.
[1] https://github.com/facebook/php-webdriver [2] https://github.com/sebastianbergmann/phpunit-selenium [3] http://behat.org/ [4] http://mink.behat.org/ [5] https://github.com/fabpot/Goutte