[QA] Acceptance tests that require account creation

[Chris McMahon]

I'm Ccing the QA mail list, this might be the place for this discussion.

What you're talking about is often called "hermetic" test environments.
 Where at run time a test environment is created sufficient to support the
tests being run.  Right now we're thinking about this in two different
ways:

One is, as you mentioned, creating a set of tests that are intended to run
in a bare wiki with no test data.  We're identifying these tests now in
order to create a PhantomJS smoke test suite.

We are also exploring using the Mediawiki API to create or update pages and
users at run time to avoid the overhead of doing it through the UI.  I've
had Jeff Hall start looking into this:
https://gerrit.wikimedia.org/r/#/c/105835/ .

Another avenue we have discussed is to use Vagrant.  I haven't gone very
far down that path, but it should be possible to share a Vagrant image with
essential test data on it.

As to why we don't have this today, my first priority was to create a
viable *shared* test environment with shared test data.  We needed beta
labs and test2wiki working properly to support fast deployments and to
avoid the "works on my machine" problems.  Now that beta and test2 are
functioning reasonably well, we can turn attention to the hermetic test
environments.

-Chris


On Mon, Jan 6, 2014 at 6:43 PM, Jon Robson <jrobson at wikimedia.org> wrote:

> The best way to get round this would be to setup a new database every
> time the tests begin and remove it at the end of the tests.
> Somewhat related to https://bugzilla.wikimedia.org/show_bug.cgi?id=56057
>
> Chris any ideas?
>
>
> On Mon, Jan 6, 2014 at 5:33 PM, Ryan Kaldari <rkaldari at wikimedia.org>
> wrote:
> > Right now, we have a lot of important features that don't have any
> > acceptance tests because in order to test them you have to create a new
> > account. This would result in thousands of bogus accounts being created
> on
> > all the wikis we test on. Does anyone have any ideas about how we could
> > solve this problem?
> >
> > Ryan Kaldari
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140107/fee389a5/attachment.html>