On Thu, 05 Apr 2012 14:53:27 -0700, K. Peachey <p858snake(a)gmail.com> wrote: We really need to start surveying real statistics on what programming languages community members know. I've seen this assertion waved around again and again, but don't see where it originates from, besides the very unreliable fact that we just don't talk much about ruby around here. I for one know ruby. And frankly ruby isn't like our other issue. Ruby is nothing like how hard OCaml would be to learn and maintain. We really should beat down this notion that anything written in ruby should be avoided. I don't like how we reject the possibility of using well written existing open-source projects simply because they were written using ruby. It's almost as bad as NIH. -- ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

On Thu, Apr 5, 2012 at 5:25 PM, Ryan Lane <rlane32(a)gmail.com> wrote: I was initially pretty skeptical on the Ruby choice, and I'm not going to say that I'm sold yet. However, the part of Chris's proposal that's most persuasive to me is the fact that Ruby/Selenium seems to be the most built out, with the largest community. In particular, RSpec seems to be a rather important piece of all of this. I decided to see if there was a Python equivalent of RSpec, and as often is the case these days, the exact question was asked on Stack Overflow: http://stackoverflow.com/questions/7079855/are-there-technical-reasons-a-ru… In short, Ruby lends itself to that kind of thing a lot more. In reading the Stack Overflow thing, I was reminded of a talk I saw a couple of years ago titled "Python vs. Ruby: A Battle to The Death" http://blog.extracheese.org/2010/02/python-vs-ruby-a-battle-to-the-death.ht… ...which, as it turns out, specifically talks about RSpec. Anyway, the point that I'm making here is that it's quite possible that Ruby really is the right tool for the job. Since I'm much more comfortable with Python than Ruby, I'd be much more comfortable if we stuck with Python. Sticking with Python may mean a lot of wheel reinvention to accommodate language orthodoxy which really kinda sucks. I'd like us to test the assertion that the Ruby/Selenium combination is much more mature than the Python/Selenium combination, but if that's true, then I think we may want to suppress the anti-Ruby bias and figure out how we can work with Ruby/Selenium. Rob

You should talk to Jeremy Postlethwaite <jpostlethwaite(a)wikimedia.org>rg>. He did some work on setting up automated Selenium tests for our fundraiser. Ryan Kaldari On 4/5/12 2:45 PM, Chris McMahon wrote:

Le 06/04/12 08:43, Asher Feldman a écrit : At least that was using a standard: RFC 1149 http://www.ietf.org/rfc/rfc1149.txt "A Standard for the Transmission of IP Datagrams on Avian Carriers"

Jeremy, yes, I'm aware of what you're doing and I think it makes sense in context. In my original post I said " Since the purpose of this project is to attract members of the global testing community, and to provide an institutional regression test suite for Mediawiki/Wikipedia, I see no reason to require individual WMF projects to use this framework..." Watir is pretty slick: http://rubydoc.info/gems/watir-webdriver/0.5.4/frames . I'm not sure if you're using selenium-webdriver or the old Selenium 1.0, but Watir solves a lot of issues for selenium-webdriver. In fact, if you're using selenium-webdriver in any language, you probably end up hacking up something that looks like Watir anyway. Watir has the advantage of nearly a decade of design and refiinement behind it, plus the API is generated directly from the HTML5 spec today. (just an aside, Selenium 1.0 is not yet fully deprecated, there is still a ton of Se1.0 out in the world, but the Se1.0 API will eventually be fully retired.)

On Fri, 06 Apr 2012 12:33:40 -0700, Ryan Lane <rlane32(a)gmail.com> wrote: Ryan, I only ran into it recently. But look over bundler: http://gembundler.com/rationale.html If another situation where something needs gems without existing apt packages comes up it may be a helpful thing to have in your toolkit of solutions. Two useful things that bundler seems to provide: - In deployment it can install gems in a place local to the application. So instead of using `gem` or apt to globally install the gems needed they'll be installed locally in a way that won't conflict with other applications. - Bundler uses a Gemfile.lock setup; When you initially install and updated gems on development it tracks the installed versions of every single dependency installed (even indirect ones you didn't depend on directly). And when you develop you check this file into version control with the rest of your source code. Using this setup bundler ensures that every gem you install using bundler (especially under deployment) is installed using the exact same version of the gem that you used during development.; So if you're using apt and a central apt sources server to ensure that all servers have the same package versions installed, bundler should help you attain that same goal with gems that don't have apt packages. -- ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

I wasn't able to make the selenium work (presumably selenium 1.0). How hard would it be to make this work? Who would be responsible of sorting out any problems encountered doing that and ensuring all developers can locally run those tests? (Actually, I'd try this before creating too many tests in that platform) Say I've fixed a bug and want to add a test. How would I do it? In which language? Do I need to know ruby for that? Suppose there's a test: * Failing in jenkins but passing locally. OR * Failing just for a single person. OR * Failing but it's apparently ok. OR * Passing but should be failing. How do you debug it? Do you need to know ruby for that? To which extent?

Some time ago some people from the test framework team started working on a Selenium Framework for MediaWiki [1], in PHP and with Selenium 1.0. One of the reasons the project discontinued was that there was no clear case of when Selenium would be useful as opposed to unit tests, esp. using QUnit and TestSwarm for UI testing. I still see some use cases, though: * Smoketesting the whole application on a very high level * Testing complex interaction patterns with several page reloads (e.g. maybe producing edit conflicts) * Testing cross-browser compatibility (e.g. using the screenshot feature) * It's easy to record tests using the Selenium IDE. So basically anyone could write tests, esp. for some less used extensions * This also might be useful when filing bugs (make them reproducible) While I see the case for new approach based on Ruby and Watir, I still think, some of the experiences we made back then could be useful in the design of the new testing environment. Here are some points of what we have so far: * a (rudimentary) set of methods that generalize login, page calls, page preparation with wikitext etc. * a configuration system to run the tests against different wikis, and also a test suite layout that helps to select a subset of tests. * a command-line test runner * a way of reconfiguring the wiki under test so that we can standartize some settings, such as language (which is an issue in some cases when testing UI). This also provides means to switch the wiki database to a test db and test images folder. * some ideas about the design of tests, suites and individual assertions I would be happy to contribute and take some of this to the next level within the new environment :) Cheers, Markus (mglaser) [1] https://www.mediawiki.org/wiki/Selenium -----Ursprüngliche Nachricht----- Von: wikitech-l-bounces(a)lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] Im Auftrag von Chris McMahon Gesendet: Donnerstag, 5. April 2012 23:46 An: wikitech-l(a)lists.wikimedia.org Betreff: [Wikitech-l] selenium browser testing proposal and prototype As QA Lead for WMF, one of the things I want to do is to create an institutional suite of automated cross-browser regression tests using Selenium. I have two goals for this suite: first, to make it attractive and convenient for the greater software testing community to both use the suite locally and also to contribute to it. Second, to have the suite be a useful regression test tool within WMF, tied to Beta Labs and controlled by Jenkins. For various reasons, I think the best language for this project is Ruby. I realize that is a controversial choice, and I would like to explain my reasoning. First let me address what I think will be the most serious objections: ** Ruby gems are incompatible with Debian/Ubuntu apt packaging, making it difficult or impossible to maintain Ruby code in production. The selenium test suite is not intended to run on production servers. There are two targets for this code. The first target is users' local machines, including and especially Windows users. The second target is a single dedicated headless Labs Ubuntu instance, controlled by Jenkins, serving as a client to selenium-server, where selenium-server is running on the various Windows etc. VMs that exist today on the WMF VPN. ** It's not PHP. As of today, PHP has no complete or authoritative implementation of selenium-webdriver/Selenium 2.0. That situation is unlikely to change any time soon. This leaves a choice of Ruby or Python. For various reasons I think Ruby is the superior choice. ** Design goals and their implementation. In the interest of making this project as attractive as possible to the greater testing community, I have defined a browser automation "stack", using the most current and accepted practices for browser test automation. The stack looks like this: * Selenium-webdriver low-level "toolbox" API. * Higher-level API for consistent access to pages and elements without the user having to create their own handling for timeouts, exceptions, "stale objects", multiple access criteria, etc. * Modern, BDD-style assertions for pass/fail criteria (as opposed to xUnit style assertions) * "Page Object" design pattern in place and functioning out of the box * Support for mobile emulators * Institutional support for Jenkins integration I submit that having these things in place and working when the user downloads the suite is what will make this project attractive to the global testing community. Taking these point by point: Even as I write this, the W3C is meeting in London to approve webdriver as an internet standard, with Selenium 2.0 as the reference implementation of that standard. This means that the selenium 2.0 API is only a fraction the size of that of Selenium 1.0, and serves a very different purpose. The selenium 2.0 API can be considered a "toolbox" from which higher-level APIs are constructed. Both Ruby and Python have full implementation and support for the Selenium 2.0 API. At higher levels of the stack, the Python and Ruby communities take different design approaches, with Python being more DIY/NIH and Ruby being more shared and organized around communities of practice. This is what makes Ruby so attractive as I implemented the rest of the stack. Watir (Web Application Testing In Ruby) is a browser test project that pre-dates Selenium itself. Watir provides an intuitive, thoughtful, well-designed high-level API for access to pages and elements that the native selenium-webdriver API does not. While historically Watir and Selenium have been different projects, as of about 2010 watir-webdriver is simply a wrapper for the low-level selenium API that preserves all of the convenience and good design Watir has always had. Watir is in use today at Facebook, The Gap, and many other places. Python has no such equivalent high-level API. Behavior Driven Development (BDD) style assertions are implemented in Ruby with the Rspec library. This approach to assertions has been a generally accepted standard in the Ruby community for some time. "The Rspec Book" was published by Pragmatic Press in 2010. Python has no equivalent BDD-style assertion library. In fact, Rspec takes advantage of Ruby's affordance for metaprogramming, something not possible in Python. "Page Objects" is a design pattern for browser testing that has become a generally accepted practice in the browser testing community, with a lot of momentum particularly in the last one or two years. Page Objects have institutional support in Ruby with the 'page-object' gem, which supports both watir-webdriver and selenium-webdriver syntax. Python has no such institutional support for page objects, each Python project using page objects implements the pattern locally, from scratch. Some support for mobile emulators is provided by Ruby's webdriver-user-agent gem, created by Alister Scott of Thoughtworks. This gem piggybacks on either the watir-webdriver or selenium-webdriver APIs in Ruby. I am unaware of any such mobile emulator support in Python. Ruby selenium tests are run via 'rake', the Ruby version of make. Jenkins has an officially supported plugin for rake. Both Ruby and Python are viable choices for this project. The difference is that Python would require a lot of custom infrastructure and scaffolding, with all of the risk and maintenance that that entails, where support for the elements of the "stack" in Ruby is already in place, well-designed, and supported by the greater Ruby community. Not only does this make maintenance of the stack for WMF purposes much simpler, it also makes the project significantly more attractive to casual users in the testing community who can get started with the suite immediately by simply installing a few gems locally and reading some public documentation, rather than having to face a daunting pile of custom code. For reference and further research, I created a prototype of this project that implements this stack here: https://github.com/chrismcmahon/Page-Object-WMF-spike Addendum: I am aware of two projects at WMF already using Selenium, one with OmniTI for AFTv5, the other from Jeremy Postlethwaite. Since the purpose of this project is to attract members of the global testing community, and to provide an institutional regression test suite for Mediawiki/Wikipedia, I see no reason to require individual WMF projects to use this framework, although I suspect that individual WMF projects might find this framework more convenient than other approaches. _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

One example I would like to have scripted is the following: - have a sysop account to watch the non-existing page name - create that page with some content - have a sysop to delete this page Very good testing case for DB transaction related problems. I doing such tests should be possible in the new framework? //Saper

Chris McMahon <cmcmahon <at> wikimedia.org> writes: Not sure what counts as authoritative, but there are a number of fairly usable PHP implementations such as php-webdriver [1] from Facebook or phpunit-selenium [2] from the PHPUnit framework, both of which are non-complete but very easy to extend (and in practice, you don't use most Selenium commands anyway). Using one of them is more troublesome than choosing a language in which there is a reference Selenium implementation, but on the other hand, you don't need to introduce another language, you can write the tests in a language all MediaWiki developers are comfortable with, and you leave open the option of reusing MediaWiki components in tests to handle setup/teardown of fixtures in a clean way. Also, my (admittedly very superficial) experience with BDD is that Cucumber/Gherkin is much better for acceptance testing than RSpec (which is more suited for unit testing). Gherkin tests are clean, human-readable descriptions which are easier to read than program code, and can be easily understood by non- developers (end users, QA people, managers) even if they have no idea what a programming language is. On the other hand Gherkin is not a real programming language, so you lose some flexibility (such as the ability to use page objects), but IMO it is well worth it. And while RSpec relies on Ruby's elegant but obscure poetry mode, and thus cannot be easily copied in other languages, Gherkin has a simple custom syntax which is trivial to implement in any language; specifically, there is a good PHP implementation called Behat [3] which has its own Selenium implementation (Mink [4]) but also can be used with any other Selenium library. Mink has the additional advantage that it abstracts away the Selenium interface so that Selenium can be replaced with some other browser simulator without changing the tests; while that doing Selenium-specific things more complicated, it can yield huge speedups for test which don't require Javascript and so Selenium can be replaced with some simple browser emulator. (Yes, Selenium2 has its own browser emulator, but it is still a fair bit slower than something like Goutte [5]). So maybe a PHP - Mink (or other Selenium library) - Behat stack instead of a Ruby - Watir - RSpec stack would be worth considering. [1] https://github.com/facebook/php-webdriver [2] https://github.com/sebastianbergmann/phpunit-selenium [3] http://behat.org/ [4] http://mink.behat.org/ [5] https://github.com/fabpot/Goutte

On 16/04/12 16:45, Chris McMahon wrote: Which seem to be the only ones that will be able to write our tests. You can talk about how "there's a standard way" or "it's documented", but that doesn't help if they aren't easy to develop. For example X.509 is all standard you want, really flexible, but you wouldn't want to manually touch it :) Moreover, making tests is not generally fun to do, so you want the barrier to be as low as possible. Not that I have anything against making tests by opening bugs, and letting the QA team (who forms that?) struggle to make them work. It's simple to just be lazy and let you test it manually, with a monkey farm, gems or a street rat. But I don't think that's the way we should go. The developers should be able to maintain them, even if they're not daily involved with it. Not only for being able to keep them the day you stop doing it, but also for detecting why they fail and maintaining them when changing the expected output. It wouldn't be fun to held a discussion in gerrit about why a bunch of tests suddenly fail after merging a big feature, wondering what it could have been trying to test.