On Wed, 22 Sep 2010 12:30:35 -0700, Brion Vibber wrote:
On Wed, Sep 22, 2010 at 11:09 AM, Dan Nessett
<dnessett(a)yahoo.com>
wrote:
Some have mentioned the possibility of using the
wiki family logic to
help achieve these objectives. Do you have any thoughts on this? If you
think it is a good idea, how do we find out more about it?
I'd just treat it same as any other wiki. Whether you're running one or
multiple wiki instances out of one copy of the code base, it just
doesn't make a difference here.
-- brion
Oh. Well, perhaps we don't agree exactly 100%. As you suggest, let's step
back a bit.
Once we get the selenium framework working I assume it will be used for a
regression test. This will comprise a set of individual tests. Generally,
these tests will write into the wiki db (some may not, but many will). To
ensure test reproducibility, the state of the wiki should be the same
each time one of these individual tests runs.
But, there is a problem. With parserTests, each individual test runs
serially. That is fine for parserTests, since (I just ran this on my
machine) while there are 610 individual tests, each takes about .08
seconds to run (on average). So, on my machine the whole parserTest takes
about 48 seconds.
Selenium tests are far more heavy-weight. A rough ball-park figure is
each takes about 10 seconds to run (this does not include the time it
would take to setup and tear down a "clean wiki"). So, a selenium-based
regression test comprising 180 individual tests would take around 30
minutes.
Not too bad. But, things are a bit more complicated. Each individual test
runs multiple times, once for every browser/OS combination chosen for the
regression test. For example, right now there are 13 configured browser/
OS combinations on the WMF Selenium Grid (see http://
grid.tesla.usability.wikimedia.org/console). So even if you only test 4
of these browser/OS configurations, the regression test (if individual
tests run serially) would take 2 hours. If you test 8 of them, it would
take 4 hours.
This is starting to get onerous. If an individual developer wishes to
ensure his modifications don't break things before committing his
changes, then waiting 4 hours for a regression test to complete is a
pretty heavy penalty. Generally, very few will pay the price.
So, running the individual tests of a selenium-based regression test
serially is not very attractive. This means you need to achieve some
concurrency in the regression test. Since individual tests may interfere
with each other, you need a way to protect them from each other. This is
what the switching functionality is for. You switch in a base set of
temporary resources for each test (or perhaps more likely for a
particular test suite comprised of mulitple individual tests) consisting
of a db, images directory, etc. This allows tests to run without
interfering with each other.
--
-- Dan Nessett