[QA] [E2] running and monitoring Echo and Flow browser tests

[Jon Robson]

+ qa mailing list

On Fri, Oct 3, 2014 at 10:46 AM, Jon Robson <jrobson at wikimedia.org> wrote:
> Would it be useful to send an email first thing Wednesday morning
> listing all the failing tests per project with links and how long they
> have been failing for.
>
> I personally would find this sort of email useful to remind me to
> check status of code about to be merged.
>
> Thoughts?
> Jon
>
>
> On Thu, Sep 25, 2014 at 5:16 PM, Chris McMahon <cmcmahon at wikimedia.org> wrote:
>>
>>
>> On Thu, Sep 25, 2014 at 4:43 PM, S Page <spage at wikimedia.org> wrote:
>>>
>>> The great news is Echo and Flow browser tests are catching real bugs:
>>> * unwanted change in the behavior of moderated topics
>>> * a template bug only for anonymous users
>>>
>>> The problem is we don't notice these when merging code. We rely on someone
>>> later figuring out that the redness in a Jenkins browser run is significant
>>> and not beta instability, and that sometimes happens after deployment.
>>
>>
>> So to begin with, in no case in the last couple of weeks has "redness" been
>> caused by "beta instability".  I have merged five patches to Echo and two to
>> Flow this week in the service of a) maintenance to make the builds green
>> when they should be b) refactoring for ease of analysis and maintenance and
>> c) workarounds for weirdness in our front end code.
>>
>> Echo is essentially done for now, I think it is about as good as it is going
>> to get.  It should be reliable and maintainable at this point. If the Echo
>> tests go red, then Something Is Wrong (although it might only be that the
>> tests need updating for new UI.) Echo is also a rather small set of tests.
>>
>> I'm going to keep refactoring the Flow tests until they are in the same
>> state.  I'll be looking for review help from S and from Dan Duvall along the
>> way.
>>
>>> So "ACTION ....then need to run the tests related to what they're
>>> changing" is a work in progress.
>>>
>>> What steps can the team take to address this?
>>
>>
>> My suggestions:
>>
>> * run the browser tests in a Vagrant env before you merge changes to master
>> ** if this is inconvenient, engage Dan Duvall until it is convenient. We've
>> already done this for VE and it is in process for MobileFrontend, which is a
>> bit more complicated.
>> * if the build goes red and you don't know why, ask me or Zeljko or Dan or
>> post to the QA mail list
>> * if you can't figure out how to test some path through the app, ask me or
>> Zeljko or Dan or post to the QA mail list.
>> * if you want something tested but you have to write features dammit and you
>> have no time to write browser tests, check in a patch that contains some
>> Given/When/Then plain English test steps and put me on the review for it,
>> we'll sort it out.
>>
>> _______________________________________________
>> E2 mailing list
>> E2 at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/e2
>>