Chris McMahon <cmcmahon <at> wikimedia.org> writes:
As QA Lead for WMF, one of the things I want to do is
to create an
institutional suite of automated cross-browser regression tests using
Selenium. I have two goals for this suite: first, to make it attractive
and convenient for the greater software testing community to both use the
suite locally and also to contribute to it. Second, to have the suite be a
useful regression test tool within WMF, tied to Beta Labs and controlled by
Jenkins.
For various reasons, I think the best language for this project is Ruby. I
realize that is a controversial choice, and I would like to explain my
reasoning. First let me address what I think will be the most serious
objections:
[...]
** It's not PHP.
As of today, PHP has no complete or authoritative implementation of
selenium-webdriver/Selenium 2.0. That situation is unlikely to change any
time soon. This leaves a choice of Ruby or Python. For various reasons I
think Ruby is the superior choice.
Not sure what counts as authoritative, but there are a number of fairly usable
PHP implementations such as php-webdriver [1] from Facebook or phpunit-selenium
[2] from the PHPUnit framework, both of which are non-complete but very easy to
extend (and in practice, you don't use most Selenium commands anyway). Using one
of them is more troublesome than choosing a language in which there is a
reference Selenium implementation, but on the other hand, you don't need to
introduce another language, you can write the tests in a language all MediaWiki
developers are comfortable with, and you leave open the option of reusing
MediaWiki components in tests to handle setup/teardown of fixtures in a clean
way.
Also, my (admittedly very superficial) experience with BDD is that
Cucumber/Gherkin is much better for acceptance testing than RSpec (which is more
suited for unit testing). Gherkin tests are clean, human-readable descriptions
which are easier to read than program code, and can be easily understood by non-
developers (end users, QA people, managers) even if they have no idea what a
programming language is. On the other hand Gherkin is not a real programming
language, so you lose some flexibility (such as the ability to use page
objects), but IMO it is well worth it. And while RSpec relies on Ruby's elegant
but obscure poetry mode, and thus cannot be easily copied in other languages,
Gherkin has a simple custom syntax which is trivial to implement in any
language; specifically, there is a good PHP implementation called Behat [3]
which has its own Selenium implementation (Mink [4]) but also can be used with
any other Selenium library.
Mink has the additional advantage that it abstracts away the Selenium interface
so that Selenium can be replaced with some other browser simulator without
changing the tests; while that doing Selenium-specific things more complicated,
it can yield huge speedups for test which don't require Javascript and so
Selenium can be replaced with some simple browser emulator. (Yes, Selenium2 has
its own browser emulator, but it is still a fair bit slower than something like
Goutte [5]).
So maybe a PHP - Mink (or other Selenium library) - Behat stack instead of a
Ruby - Watir - RSpec stack would be worth considering.
[1]
https://github.com/facebook/php-webdriver
[2]
https://github.com/sebastianbergmann/phpunit-selenium
[3]
http://behat.org/
[4]
http://mink.behat.org/
[5]
https://github.com/fabpot/Goutte