<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 22, 2015 at 2:51 PM, Dan Duvall <span dir="ltr"><<a href="mailto:dduvall@wikimedia.org" target="_blank">dduvall@wikimedia.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Wed, Jul 22, 2015 at 10:43 AM, Brian Gerstle <span dir="ltr"><<a href="mailto:bgerstle@wikimedia.org" target="_blank">bgerstle@wikimedia.org</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><br></div><div>Also, one neat thing I saw done at Spotify was a "Test Data Service." This service's job was to abstract away concrete test data from its qualities. This was mostly used to acquire test users, thus allowing testers to declaratively acquire test data. For example, you could say "I want a free user with X flags" or "I want a premium user who has tracks in their starred playlist." The service's other job was to lock users to prevent parallel tests from corrupting a shared test user's state (also resulting in a false negative).</div></div></div></blockquote><div><br></div></span><div>That's really interesting. Do you have a link to further reading/docs?</div></div></div></div></blockquote><div><br></div><div>Nothing about the the Test Data Service (referred to commonly as TDS) specifically. They're main drum to beat was <a href="http://www.cs.tut.fi/tapahtumat/testaus12/kalvot/Karl_ta.pdf">Model Based Testing</a> (mostly driven by Kristian Karl). Towards the end of my time there, I was talking to people about creating users on-demand to side-step locking and hand-curating test users, which it seems like you're nudging us towards. I'd be happy to reach out and see if they'd be willing to talk to us more about what they're doing now.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>I had thought of exploring a simple rotating queue of users for Beta Cluster a while back to improve isolation, but given the variance of setup/teardown operations across different test suites, it seemed like it still might be difficult to maintain a clean state after a while.</div></div></div></div></blockquote><div><br></div><div>Deadlocks (or missed unlocks) were occasionally a problem, which is why locks were always given a specific life span (e.g. lock for 30 minutes—hopefully your tests finish first!!).</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> Maybe once we have a proper staging cluster and the liberty to wipe the database clean periodically, this type of setup would be worth exploring further.</div><div><br></div><div>For now, I think making strides toward more isolated environments and using the API for creation of initial users/pages seems to be the right approach. It's relatively inexpensive—probably the least expensive aspect of browser-driven tests anyway—and can be easily extended in the framework or in test helpers for specific test cases, e.g. creating Flow topics, etc.</div></div></div></div></blockquote><div><br></div><div>This is where Spotify was going around the time I left. They were very SOA, and trying to push adoption of docker for testing and production. There were some promising results of orchestrating an isolated ecosystem of services on demand, testing them, then tearing it all down. Combine this with pre-baked Docker containers and I think you see where it could lead.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>Keeping the provisioning of preconditions in code, closer to the test implementation, also makes tests easier to reason about—you known that test case A depends on the precondition that the user that has 3 edits because the test case creates the user and makes 3 edits, instead of having to inspect a preexisting account or rely on comments that may not exist.</div></div></div></div></blockquote><div><br></div><div>As usual, all about trade-offs. If you want to keep your tests short (ideally you have a lot of them), side-loading or pre-baking service images/containers/etc. with data can be a big time saver.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class=""><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><br></div><div>We might want something similar for retrieving and locking test articles for editing, but this might not be necessary if our vagrant/virtualization infrastructure is mature enough to ensure tests are isolated.</div></div></div></blockquote><div><br></div></span><div>The environment that's setup for CI browser tests (per patch tests, not the daily runs) is completely isolated for a single project/patchset.[1] I'd like to push this further though, by isolating pages to unique prefixes/namespaces per scenario, so that we might run scenarios in parallel. Another possibility would be to use a multiwiki setup with <i>n</i> wikis and simply isolate each subset of scenarios to one of them<i>.</i> Of course you'd have to be using our CI infrastructure to benefit from the latter. :P (Couldn't resist ... in all seriousness we could figure out a way to generalize CI-type MW setup as libraries/tools.)</div><div><br></div><div>[1] <a href="https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests" target="_blank">https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests</a></div></div><span class=""><div><br></div>-- <br><div><div dir="ltr">Dan Duvall<div>Automation Engineer</div><div><a href="http://wikimediafoundation.org" target="_blank">Wikimedia Foundation</a><br></div></div></div>
</span></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr">EN Wikipedia user page: <a href="https://en.wikipedia.org/wiki/User:Brian.gerstle" target="_blank">https://en.wikipedia.org/wiki/User:Brian.gerstle</a><br>IRC: bgerstle</div></div></div></div>
</div></div>