<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jun 6, 2014 at 6:20 PM, James Forrester <span dir="ltr"><<a href="mailto:jforrester@wikimedia.org" target="_blank">jforrester@wikimedia.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial">On 6 June 2014 16:21, Chris McMahon </span><span dir="ltr" style="font-family:arial"><<a href="mailto:cmcmahon@wikimedia.org" target="_blank">cmcmahon@wikimedia.org</a>></span><span style="font-family:arial"> wrote:</span><br>
</div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>
<div>On Fri, Jun 6, 2014 at 4:16 PM, Matthew Flaschen <span dir="ltr"><<a href="mailto:mflaschen@wikimedia.org" target="_blank">mflaschen@wikimedia.org</a>></span> wrote:<br>
</div></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>Are they voting or non-voting?<br>
</div></blockquote><div><br></div><div>I think browser tests should always be non-voting. Therein lies a day-long training session... </div>
</div></div></div></div></blockquote><div><br></div><div><div style="font-family:arial,helvetica,sans-serif">And FWIW I fundamentally disagree. :-)</div><div style="font-family:arial,helvetica,sans-serif">
<br></div><div style="font-family:arial,helvetica,sans-serif">Non-voting tests are routinely ignored. If your change breaks a test, you should either fix your code or update the test; both of these actions are conscious responses to the outcome of the test, and if they're not part of the pre-merge workflow they too rarely don't happen.</div>
</div></div></div></div>
</blockquote><div><br></div><div>We should probably have a clear understanding about what "voting" means. As I understand it, having an automated test be voting means that a failing test will generate a "-2/DO NOT MERGE" review and prevent a branch from being merged to the master branch. </div>
<div><br></div><div>I would encourage a set of browser tests that runs on an automated, highly-controlled, pre-master-branch, non-beta-labs dev environments that might be voting. We don't have that now, but Dan Duvall's current work updating the Vagrant dev environments may lead in that direction. </div>
<div><br></div><div>I think the MobileFrontend team has come the closest when they run browser tests against local dev envs, but those local dev envs are maintained by hand and are some way away from being able to be e.g. deployed automatically via Jenkins. The MF folks generally merge new browser tests to master in the same branch as the new feature being tested.</div>
<div><br></div><div>Post-merge browser tests look like:</div><div><br></div><div>Launch a selenium process</div><div>To reach out over the internet</div><div>to talk to a browser</div><div>
Running on a VM of some sort</div><div>To reach out over a different part of the internet</div><div>To manipulate a shared public test environment like beta labs or test2wiki</div>
<div>Whose result is matched to expectations</div><div><br></div><div>And these tests are intended to expose system and interaction issues as well as code issues in individual features.</div><div><br></div><div>Also, I encourage thinking along the lines of "if it's a bug in beta labs then it is a bug, period, and we need to fix it".</div>
<div><br></div><div>-Chris </div><div><br></div><div><br></div></div><br></div></div>