[QA] Discussion of tests for a clean wiki

[Chris Steipp]

On Tue, Jan 21, 2014 at 12:21 PM, Chris McMahon <cmcmahon at wikimedia.org>wrote:

>
> Hello QAers,
>
> I installed a new 1.22 Mediawiki locally and ran the browsertests
> repository against it. Only one test passed, create_account.feature.
>
> After a little investigation, I want to start a discussion about the value
> of tests for a clean wiki, and how they should be managed.
>
> A bare Mediawiki wiki is very different than a Wikipedia wiki, not only in
> the lack of important features, but even for example the default editor
> controls the user sees are the primitive ones Wikipedia users see only when
> using IE6.
>
> Some options for bare-wiki browser tests:
>
> Raw installs:
> * To run as a smoke test in WMF Jenkins using PhantomJS or headless
> Firefox.  We can make a proof of concept for this using the
> create_account.feature test. Beyond that would take significant thought and
> design.
> * To run as a regression test for public releases of Mediawiki, outside of
> Wikipedia and the work of the Wikimedia Foundation.  These would have to be
> in a repo of their own, because they will share almost no code with tests
> pointed to Wikipedia environments.  Given the difference between a new
> Mediawiki install and a Wikipedia wiki, would this be better done by
> volunteers, with support from QA helpers?
>
> Development environments:
> * What may be more valuable would be bare-wiki tests for particular
> development environment, for example a bare wiki with MobileFrontend
> installed. Of course, the regression tests for an Extension on a clean
> Mediawiki install would have to be managed from within that repo.
>

I'd vote for having a set of tests that should pretty much pass against a
vagrant environment with just the mediawiki role enabled (basically core
with no extensions). When we do MediaWiki major releases, we can take that
and move it to whatever repo(s) get greated for the regression tests
against the public releases.


>
> Test data and test configuration
> * We will soon have the ability to create wiki pages and users on target
> wikis at run time via the Mediawiki API. This may be required in order to
> create valuable bare-wiki  tests.
> * Could we create the ability for tests to update configuration and also
> install Extensions in a bare wiki?  That seems ambitious and even out of
> scope, but would it be valuable?
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140121/53b64cce/attachment-0001.html>