[QA] [WikimediaMobile] Qualifiers for selecting test articles for vagrant role

[Dan Duvall]

On Wed, Jul 22, 2015 at 10:43 AM, Brian Gerstle <bgerstle at wikimedia.org>
wrote:
>
>
> Also, one neat thing I saw done at Spotify was a "Test Data Service." This
> service's job was to abstract away concrete test data from its qualities.
> This was mostly used to acquire test users, thus allowing testers to
> declaratively acquire test data.  For example, you could say "I want a free
> user with X flags" or "I want a premium user who has tracks in their
> starred playlist."  The service's other job was to lock users to prevent
> parallel tests from corrupting a shared test user's state (also resulting
> in a false negative).
>

That's really interesting. Do you have a link to further reading/docs?

I had thought of exploring a simple rotating queue of users for Beta
Cluster a while back to improve isolation, but given the variance of
setup/teardown operations across different test suites, it seemed like it
still might be difficult to maintain a clean state after a while. Maybe
once we have a proper staging cluster and the liberty to wipe the database
clean periodically, this type of setup would be worth exploring further.

For now, I think making strides toward more isolated environments and using
the API for creation of initial users/pages seems to be the right approach.
It's relatively inexpensive—probably the least expensive aspect of
browser-driven tests anyway—and can be easily extended in the framework or
in test helpers for specific test cases, e.g. creating Flow topics, etc.

Keeping the provisioning of preconditions in code, closer to the test
implementation, also makes tests easier to reason about—you known that test
case A depends on the precondition that the user that has 3 edits because
the test case creates the user and makes 3 edits, instead of having to
inspect a preexisting account or rely on comments that may not exist.


> We might want something similar for retrieving and locking test articles
> for editing, but this might not be necessary if our vagrant/virtualization
> infrastructure is mature enough to ensure tests are isolated.
>

The environment that's setup for CI browser tests (per patch tests, not the
daily runs) is completely isolated for a single project/patchset.[1] I'd
like to push this further though, by isolating pages to unique
prefixes/namespaces per scenario, so that we might run scenarios in
parallel. Another possibility would be to use a multiwiki setup with
*n* wikis and
simply isolate each subset of scenarios to one of them*.* Of course you'd
have to be using our CI infrastructure to benefit from the latter. :P
(Couldn't resist ... in all seriousness we could figure out a way to
generalize CI-type MW setup as libraries/tools.)

[1] https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests

-- 
Dan Duvall
Automation Engineer
Wikimedia Foundation <http://wikimediafoundation.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20150722/92a87229/attachment.html>