On Wed, Jul 22, 2015 at 10:43 AM, Brian Gerstle <bgerstle@wikimedia.org> wrote:Also, one neat thing I saw done at Spotify was a "Test Data Service." This service's job was to abstract away concrete test data from its qualities. This was mostly used to acquire test users, thus allowing testers to declaratively acquire test data. For example, you could say "I want a free user with X flags" or "I want a premium user who has tracks in their starred playlist." The service's other job was to lock users to prevent parallel tests from corrupting a shared test user's state (also resulting in a false negative).That's really interesting. Do you have a link to further reading/docs?
I had thought of exploring a simple rotating queue of users for Beta Cluster a while back to improve isolation, but given the variance of setup/teardown operations across different test suites, it seemed like it still might be difficult to maintain a clean state after a while.
Maybe once we have a proper staging cluster and the liberty to wipe the database clean periodically, this type of setup would be worth exploring further.For now, I think making strides toward more isolated environments and using the API for creation of initial users/pages seems to be the right approach. It's relatively inexpensive—probably the least expensive aspect of browser-driven tests anyway—and can be easily extended in the framework or in test helpers for specific test cases, e.g. creating Flow topics, etc.
Keeping the provisioning of preconditions in code, closer to the test implementation, also makes tests easier to reason about—you known that test case A depends on the precondition that the user that has 3 edits because the test case creates the user and makes 3 edits, instead of having to inspect a preexisting account or rely on comments that may not exist.
We might want something similar for retrieving and locking test articles for editing, but this might not be necessary if our vagrant/virtualization infrastructure is mature enough to ensure tests are isolated.The environment that's setup for CI browser tests (per patch tests, not the daily runs) is completely isolated for a single project/patchset.[1] I'd like to push this further though, by isolating pages to unique prefixes/namespaces per scenario, so that we might run scenarios in parallel. Another possibility would be to use a multiwiki setup with n wikis and simply isolate each subset of scenarios to one of them. Of course you'd have to be using our CI infrastructure to benefit from the latter. :P (Couldn't resist ... in all seriousness we could figure out a way to generalize CI-type MW setup as libraries/tools.)--Dan DuvallAutomation Engineer