<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Jul 2, 2014 at 10:52 PM, S Page <span dir="ltr"><<a href="mailto:spage@wikimedia.org" target="_blank">spage@wikimedia.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">I'd like to meet with the brains in QA about Flow browser tests. Chris McMahon is going on vacation and Željko might be difficult to schedule. So here are some topics for a non-meeting. Thanks for ideas and comments, or we can meet later.<br>
</div></blockquote><div><br></div><div>I am available for pairing session this week 8-9am San Francisco time (5-6pm my time) Wednesday-Friday. If somebody is closer to my time zone (UTC+2), we can pair earlier.</div><div>
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Background runs once per scenario, when really I only need it once per feature (from Matt Flaschen)<br>
</div></div></blockquote><div><br></div><div>What do you mean by that? Every scenario is a test for itself, and it should be able to run in parallel with other scenarios. The most important thing for a test is to be robust, not to be fast. If it is possible, we try to do both, but if it is not possible we always choose robust over fast.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><span>* <i>generally useful?</i></span><span> Speed up the "Given I am logged in" by using an API call to login.</span></div>
</div></blockquote><div><br></div>But that would log in the API client, not the browser, right? There are some things that we could do to speed up the tests[1] (see #4 Pre-populate the cookies)<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><span>If you look at e.g. </span><span><a href="https://saucelabs.com/jobs/465f257ffdeb422e95b029c5f8df3446" target="_blank">https://saucelabs.com/jobs/465f257ffdeb422e95b029c5f8df3446</a></span><span> , after login the test does</span></div>
<div><span> POST elements</span></div><div><span> using: "xpath"</span></div>
<div><span> value: ".//a"</span><span><br>then spends the next 8 seconds getting dozens of attribute hrefs from the Main_Page. And then it seems to repeat this. What is it doing?</span></div>
</div></blockquote><div><br></div><div>This sounds like a good topic for a pairing session. :)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><span>* Reuse
existing content instead of always creating new topics and posts. E.g.
for Load More/infinite scroll, instead of creating 11 topics, use
"Pending there are 11 topics"; for sorting, use "Pending there are more
than 2 topics" and add a reply to one of them; etc.<br></span></div></div></blockquote><div><br></div><div>To make tests as robust as possible, every test should create everything it needs and clean up after it is done. If speed is the problem, creating/deleting should be done via the API[2].</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><i>generally useful?</i> Always check for ResourceLoader errors<br>
</div><div><ul><li><span>We want a way (@RLcheck annotation?) to assert <br>
And page has no ResourceLoader errors<br>on start and end of every single page</span></li></ul></div></div></blockquote><div>If we have consensus that this is a good thing, we can make it so.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><span><i>generally useful?</i></span><span> Always check for a pink errorbox on the page </span><span><a href="https://bugzilla.wikimedia.org/show_bug.cgi?id=61304" target="_blank">https://bugzilla.wikimedia.org/show_bug.cgi?id=61304</a></span> <br>
<ul><li>Similarly this should be a default assertion.</li></ul></div></div></blockquote><div><br></div><div>The same as above.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><span><b>Increase Flow browser test reliability</b></span></div><div><span><br>
Problem:
Typical test creates a topic or post on Talk:Flow_QA, then checks elements in the
first topic or post on the board. But in Continuous Integration (and sometimes on
shared labs servers), other browsers are simultaneously hitting the same
page, and we get weird failures.</span></div><div><br></div><div><span>Solutions:</span></div><div><ul><li><span>Start
a new board for some tests. This would mean adding a dedicated Flow_QA
namespace on test hosts, and adding it to $wgFlowOccupyNamespaces.</span></li></ul></div><div><ul><li><span>Change tests to target </span><span><i>the topic or post they just created.</i></span><span> Tests know what the text of the topic is, so search for that and look within it for elements.</span></li>
</ul></div></div></blockquote><div>I vote for tests creating things they need and cleaning up after they are done.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><span>Problem:
Flow tests require MEDIAWIKI_USER to have admin and oversight rights
because they assume block user, delete topic/post, and suppress
topic/post are available. Maybe tests should skip if the MEDIAWIKI_USER doesn't have the right. Or could we grant the Selenium_user these rights but only in a dedicated Flow_QA namespace? </span><span><a href="https://bugzilla.wikimedia.org/show_bug.cgi?id=67158" target="_blank">https://bugzilla.wikimedia.org/show_bug.cgi?id=67158</a></span></div>
</div></blockquote><div><br></div><div>We should run tests on machines/sites that are set up to run tests, so problems like this do not happen. </div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><span><b>New features for new tests</b></span></div><div><br></div><div>
<span><i>generally useful</i></span><span>? To test Thank, mentions appearing in the thanked/mentioned user's notifications, and future subscription features, we need a second QA user.</span></div></div></blockquote>
<div><br></div><div>This is doable. Is there a problem here? The most robust way would be when every test would create user(s) it needs.</div><div><br></div><div>Željko</div><div>--</div><div>1: <a href="https://saucelabs.com/resources/selenium/speed-up-your-selenium-tests">https://saucelabs.com/resources/selenium/speed-up-your-selenium-tests</a> </div>
</div>2: <a href="https://rubygems.org/gems/mediawiki_api">https://rubygems.org/gems/mediawiki_api</a></div></div>