[QA] [EE] migrating Flow tests to WMF Jenkins

[Chris McMahon]

On Fri, Jun 6, 2014 at 6:20 PM, James Forrester <jforrester at wikimedia.org>
wrote:

> On 6 June 2014 16:21, Chris McMahon <cmcmahon at wikimedia.org> wrote:
>
>>  On Fri, Jun 6, 2014 at 4:16 PM, Matthew Flaschen <
>> mflaschen at wikimedia.org> wrote:
>>
>>> Are they voting or non-voting?
>>>
>>
>> I think browser tests should always be non-voting.  Therein lies a
>> day-long training session...
>>
>
> ​And FWIW I fundamentally disagree. :-)
>
> Non-voting tests are routinely ignored. If your change breaks a test, you
> should either fix your code or update the test; both of these actions are
> conscious responses to the outcome of the test, and if they're not part of
> the pre-merge workflow they too rarely don't happen.
>

We should probably have a clear understanding about what "voting" means.
 As I understand it, having an automated test be voting means that a
failing test will generate a "-2/DO NOT MERGE" review and prevent a branch
from being merged to the master branch.

I would encourage a set of browser tests that runs on an automated,
highly-controlled, pre-master-branch, non-beta-labs dev environments that
might be voting.  We don't have that now, but Dan Duvall's current work
updating the Vagrant dev environments may lead in that direction.

I think the MobileFrontend team has come the closest when they run browser
tests against local dev envs, but those local dev envs are maintained by
hand and are some way away from being able to be e.g. deployed
automatically via Jenkins.  The MF folks generally merge new browser tests
to master in the same branch as the new feature being tested.

Post-merge browser tests look like:

Launch a selenium process
To reach out over the internet
to talk to a browser
Running on a VM of some sort
To reach out over a different part of the internet
To manipulate a shared public test environment like beta labs or test2wiki
Whose result is matched to expectations

And these tests are intended to expose system and interaction issues as
well as code issues in individual features.

Also, I encourage thinking along the lines of "if it's a bug in beta labs
then it is a bug, period, and we need to fix it".

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140609/e9ab41bb/attachment.html>