On Wed, Mar 10, 2010 at 1:43 PM, Michael Dale &lt;mdale(a)wikimedia.org&gt; wrote: Perhaps "/testing/extensions/<name>/" since not all extensions will ever have tests so it would make it easier and cleaner to sort the directory. -Peachey

2010/3/10 Conrad Irwin &lt;conrad.irwin(a)googlemail.com&gt;om>: On the other hand, we intend on setting up a server that runs these tests automatically. Instead of having to pull these tests from a million different places, it'd be nice if this server could just update from one dir and be done with it, which is exactly why I set it up the way I did. I agree that this is kind of the enemy of nice bundling of tests with extensions, but I think having an automatic test setup that tests stuff /before/ it's released is more important and more frequently used than providing easy access to tests to those .3% of downloaders that are actually gonna run them. Roan Kattouw (Catrope)

2010/3/10 Platonides &lt;Platonides(a)gmail.com&gt;om>: The point is it's not a single server testing them against localhost: it's a virtual machine testing against a number other VMs, each running a different OS/browser. Roan Kattouw (Catrope)

On Wed, Mar 10, 2010 at 11:08 AM, Lane, Ryan &lt;Ryan.Lane(a)ocean.navo.navy.mil&gt; wrote: It would definitely be an extremely bad thing if it were hard to test on localhost. Ideally, it should be as easy to run as parser tests. We can't ask people (whether committers or not) to write tests along with their patches if they can't confirm that the tests pass. Parser tests just set a bunch of configuration variables to particular values when they start. From maintenance/parserTests.inc, line 507 ff.: $settings = array( 'wgServer' => 'http://localhost', 'wgScript' => '/index.php', 'wgScriptPath' => '/', etc. This works fine, although when a new config variable is introduced, it might have to have a value added to the array if it causes test failures with some settings. I'm a little surprised by this altogether, actually. I had seen some mention of Selenium adoption, but nothing concrete. Was this discussed publicly anywhere, or has this kind of project started to migrate behind closed doors now that there are so many paid people? I'm subscribed to all major MediaWiki development discussion fora, as far as I know, but I've felt increasingly out of the loop lately. Having more paid developers is great (and so is a good testing framework!), but that shouldn't mean that decisions are made where volunteers can't weigh in.

On Wed, Mar 10, 2010 at 5:10 PM, Roan Kattouw &lt;roan.kattouw(a)gmail.com&gt; wrote: Okay, but it seems like several employees (including you) already knew this, and several volunteers (including me) did not. This implies that there's some communications channel that employees are reading, but not volunteers. Is it some public place that we just don't visit? If so, where? Or do you have internal face-to-face meetings, private mailing lists, something like that? Assuming Wikimedia intends to maintain a bazaar development model, it's quite important that interested volunteers can be on the same page as employees.

On 03/12/2010 06:33 PM, Ryan Lane wrote: Sure, the main problem I see with the proposed set up [1] is that you have no way of ensuring the MediaWiki you are hitting is in a consistent state; having tests fail because of edit conflicts, modified pages, users that already exist, have been blocked, etc. as a result of other tests, is very annoying; (tests can't be relied on to clean up after themselves, and one failure should not cause the rest of the suite to fail until it is manually fixed) To some extent this can be worked around by using carefully selected random parameters for many things, but that is a horrible hack, and requires extra work in writing test scripts; though as I assume/hope they will be written in PHP, not a huge difficultly, providing we teach everyone how. Much cleaner is to have a set up like the current parser tests where each test can specify which articles it expects to exist with what content (selenium tests may also wish to specify which users exist with what privileges/preferences as well) in addition to being able to tweak configuration settings (otherwise we're going to need a fair few MediaWiki's even to test configurations that are live at WikiMedia). This is quite readily doable, if you run a MediaWiki instance on the same machine as the test runner, and I imagine it would also be possible to do by building a communication protocol between the two, though that seems like a waste of effort. This won't be a problem for local developers, the test runner, the browser and MediaWiki are all on localhost, the tests can be written in PHP (or exported from selenium IDE into PHP) and run with a wrapper script (maintenance/seleniumTests.php - or w/e) that handles the configuration, output is handled by PHPUnit, all happy. For a selenium-grid setup, it's not so obvious how to do it, I'd suggest that, instead of having developers run scripts against the grid themselves, they simply request a run on a server designed for this task, which runs the test through the grid using a hostname that will resolve back to the runner. This allows easy local control over the MediaWiki, and makes it reasonably easy to write an interface for normal developers who won't/can't run selenium to run tests against MediaWiki. [developer] (remote) | "request run" (could just be php seleniumTests.php over SSH) | [selenium runner <-> MediaWiki] (both in PHP, load balance at will) | | "run test" | | | [selenium grid] "HTTP requests" | | "run test" | | | [browser virtual machines] I don't think reconfiguring MediaWiki per test-script, or per set of test-scripts is an outrageous overhead, Selenium is a very "enterprise" tool, and booting virtual machines with browsers in is likely much more costly than that. The advantages it gives are obvious, tests should not fail because of faults in the testing environment, that just wastes time. Cleaning the state of the browsers is probably not so critical here, but it's another "gotcha", if one test leaves the user logged in, and the next test tries to click the "Login" link, it explodes, and vice/versa. Selenium can helps somewhat here (if you persuade it to, and to varying extents in various browser versions), but it's likely easier to cleanse the database. [1] http://usability.wikimedia.org/wiki/File:Selenium_architecture_diagram.svg Hope this gives some food for thought, it's a bit longer than I intended. Conrad

On 03/12/2010 07:48 PM, Ryan Lane wrote: Yes, if you look at extensions/Poem/Poem.php, you see $wgParserTestFiles[] = dirname( __FILE__ ) . "/poemParserTests.txt"; I'd hope this would work in exactly the same way with $wgSeleniumTestFiles. The parserTests are currently in their own light-weight file format (which isn't great, but it's certainly not terrible). I imagine that selenium tests will be (initially at least) based on the PHP exuded by the selenium IDE [see below], with code added to the setUp() for MediaWiki configuration. Great, though obviously some cleverness is needed to avoid running all tests on all browsers on every commit, but nothing too challenging. That's what the docs say :), I did not find this to always be the case. Conrad <?php require_once 'PHPUnit/Extensions/SeleniumTestCase.php'; class Example extends PHPUnit_Extensions_SeleniumTestCase { function setUp() { $this->setBrowser("*chrome"); $this->setBrowserUrl("http://en.wiktionary.org/"); } function testMyTestCase() { $this->open("/wiki/idiom"); $this->type("searchInput", "adsa"); $this->click("searchGoButton"); $this->waitForPageToLoad("30000"); } } ?>

On Fri, Mar 12, 2010 at 3:48 PM, Ryan Lane &lt;rlane32(a)gmail.com&gt; wrote: Unless it's very easy to set up Selenium on localhost on all platforms (is it?), committers should be able to submit code to run on the test servers. Otherwise, there's no way to tell if your code causes no regressions without committing it -- and code should be tested *before* commit. (As well as an automatic run after commit, since we don't know if people actually did test before committing.) As far as I know, projects with test servers tend to also allow committers to submit code changes to them for testing.

2010/3/12 Aryeh Gregor &lt;Simetrical+wikilist(a)gmail.com&gt;om>: Most of this communication happened by internal e-mail (we don't have an internal mailing list, but we do have some group e-mail aliases that are mostly used for, well, internal stuff, mostly stuff that's not ready yet or just not interesting to the general public). Some of it happened in #wikipedia_usability , which is a public channel keeping public logs. Of course most of the usability team are in the same office, so obviously face-to-face conversations and meetings happen; these are not necessarily "internal" or "secret" or whatever, just a natural consequence of being in the same room. Again, as Ryan pointed out, we started out exploring Selenium for internal usage only; he described that pretty well, so I won't elaborate on that. Roan Kattouw (Catrope)

It would be good if all testing frameworks were fairly consistent. I believe a project is going to be set up to coordinate this. As Roan mentioned, we are primarily setting up a selenium grid architecture up for doing testing for usability initiative. The long term goal for this is to integrate with code review, and to have a general selenium test suite for MediaWiki. We are in the initial stages for planning how selenium will fit in with the MediaWiki development process, and we welcome all input. If you, or anyone, want to be included in the planning process, let us know. I was asking if we had a planning page so that we can keep this public and get input from all of the developers. Here's the small of amount of documentation we currently have: http://usability.wikimedia.org/wiki/Resources#Interaction_testing_automation http://usability.wikimedia.org/wiki/File:Selenium_architecture_diagram.svg Respectfully, Ryan Lane

Roan Kattouw wrote: I see. Still, sharing the checkout read-only with the client VMs is probably a good idea, since that ensures no client will be using non-synced tests.

2010/3/10 Aryeh Gregor &lt;Simetrical+wikilist(a)gmail.com&gt;om>: Good point. Since we're trying to make these things mirror the parser tests (see other thread), it's a good idea to make the directory structure the same as well. Roan Kattouw (Catrope)

On Wed, Mar 10, 2010 at 8:18 AM, Roan Kattouw &lt;roan.kattouw(a)gmail.com&gt; wrote: It makes more sense to me to have it be in trunk/extensions/UsabilityInitiative/testing/. This gives a consistent model that even unofficial extensions can follow. It's also closer to how parser tests currently work -- see trunk/extensions/Cite/citeParserTests.txt, for instance.