[QA] Failing MobileFrontend browser tests

[Arthur Richards]

Greg, Rob, Tomasz and I just had an IRL conversation about this. Given some
of the ambiguity of the test failures we've been discussing as related to
'infrastructure/architecture issues', we should be filing specific bug
reports in Bugzilla in regards to the issues we see. I'll followup with the
mobile web team directly to start digging into this. Further, it was
clarified that Greg G has ownership of getting the issues resolved. We also
agreed that for the time being, mobile-tech will be removed from the list
of recipients of the failure emails until the issues are resolved. However,
no one in the room was sure how to actually do this - Zeljko, Chris, Dan,
is this something one of you can help out with?

Finally, I'd like to mention that none of the conversation on this thread
was intended to question the integrity or validity of the hard work that
the QA team has put in to making improvements to the testing
infrastructure. We're all on the same (figurative) team, and we understand
that it takes time to iron out inevitable issues particularly when it
pertains to complex systems, migrations, etc. At the end of the day, we're
very eager to be able to fully leverage an automated test system to help us
ship better quality stuff, and the heart of this conversation is about
resolving the things currently standing in the way of that goal.


On Fri, Jul 11, 2014 at 12:39 PM, Rob Lanphier <robla at wikimedia.org> wrote:

> On Fri, Jul 11, 2014 at 10:12 AM, Željko Filipin <zfilipin at wikimedia.org>
> wrote:
>
>> On Thu, Jul 10, 2014 at 12:57 AM, Juliusz Gonera <jgonera at wikimedia.org>
>> wrote:
>>
>>>
>>> https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Wikitext%20Editor%20%28TEST%20RUN%20ON%20WIKIPEDIA%20SO%20SHOULD%20NOT%20CAUSE%20SAVES%29/Opening_editor/
>>> Editor doesn't seem to load, possible causes: beta labs API error, or
>>> problem with connection between saucelabs and beta labs
>>
>>
>> My guess is that you are talking about this failure[1]. Looking at the
>> Sauce Labs screencast, it looks to me that labs was just slow to respond
>> and the test failed after 5 seconds with pretty descriptive error
>> message[2]. It is hard for me to say why labs is slow. As Antoine has
>> suggested, looking at logs for that date/time could help.
>>
>> If that happens a lot, a short workaround would be to make the test wait
>> for 10 seconds instead of 5.
>>
>
>
> Thanks for the response, Zeljko.  I'm going to make the dumb manager
> response and ask the question (not just to Zeljko, but to everyone): as a
> permanent fix, can we change the wait to 60 seconds and call it good?  How
> was 5 seconds arrived at as the time for an automated test to fail?  Labs
> was never intended for performance testing, and it's not suited for it, so
> if the rationale is because we're testing end-user performance, we should
> stop.  SauceLabs also isn't designed for it, so any effort to use it will
> also end in sadness.  60 seconds may be a bit extreme, but really, let's
> set a number here (and everywhere we have timeouts) that's high enough to
> stop getting false positives, and leave it there.  In cases where we want
> to automate a responsiveness test, let's make sure we're doing it against
> test2 or some production cluster machine, and that we're doing it from a
> client that isn't also likely to introduce random delays.
>
> Rob
>
>
>> 1:
>> https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/55/testReport/junit/(root)/Wikitext%20Editor%20(TEST%20RUN%20ON%20WIKIPEDIA%20SO%20SHOULD%20NOT%20CAUSE%20SAVES)/Opening_editor/
>> 2: timed out after 5 seconds, waiting for {:css=>".wikitext-editor",
>> :tag_name=>"textarea"} to become present (Watir::Wait::TimeoutError)
>>
>
>
>


-- 
Arthur Richards
Team Practices Lead
[[User:Awjrichards]]
IRC: awjr
+1-415-839-6885 x6687
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140711/0689dd01/attachment.html>