[QA] Discussion of tests for a clean wiki

[Chris McMahon]

Hello QAers,

I installed a new 1.22 Mediawiki locally and ran the browsertests
repository against it. Only one test passed, create_account.feature.

After a little investigation, I want to start a discussion about the value
of tests for a clean wiki, and how they should be managed.

A bare Mediawiki wiki is very different than a Wikipedia wiki, not only in
the lack of important features, but even for example the default editor
controls the user sees are the primitive ones Wikipedia users see only when
using IE6.

Some options for bare-wiki browser tests:

Raw installs:
* To run as a smoke test in WMF Jenkins using PhantomJS or headless
Firefox.  We can make a proof of concept for this using the
create_account.feature test. Beyond that would take significant thought and
design.
* To run as a regression test for public releases of Mediawiki, outside of
Wikipedia and the work of the Wikimedia Foundation.  These would have to be
in a repo of their own, because they will share almost no code with tests
pointed to Wikipedia environments.  Given the difference between a new
Mediawiki install and a Wikipedia wiki, would this be better done by
volunteers, with support from QA helpers?

Development environments:
* What may be more valuable would be bare-wiki tests for particular
development environment, for example a bare wiki with MobileFrontend
installed. Of course, the regression tests for an Extension on a clean
Mediawiki install would have to be managed from within that repo.

Test data and test configuration
* We will soon have the ability to create wiki pages and users on target
wikis at run time via the Mediawiki API. This may be required in order to
create valuable bare-wiki  tests.
* Could we create the ability for tests to update configuration and also
install Extensions in a bare wiki?  That seems ambitious and even out of
scope, but would it be valuable?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140121/1eac3928/attachment.html>