[QA] Failing MobileFrontend browser tests

[Rob Lanphier]

On Fri, Jul 11, 2014 at 10:12 AM, Željko Filipin <zfilipin at wikimedia.org>
wrote:

> On Thu, Jul 10, 2014 at 12:57 AM, Juliusz Gonera <jgonera at wikimedia.org>
> wrote:
>
>>
>> https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Wikitext%20Editor%20%28TEST%20RUN%20ON%20WIKIPEDIA%20SO%20SHOULD%20NOT%20CAUSE%20SAVES%29/Opening_editor/
>> Editor doesn't seem to load, possible causes: beta labs API error, or
>> problem with connection between saucelabs and beta labs
>
>
> My guess is that you are talking about this failure[1]. Looking at the
> Sauce Labs screencast, it looks to me that labs was just slow to respond
> and the test failed after 5 seconds with pretty descriptive error
> message[2]. It is hard for me to say why labs is slow. As Antoine has
> suggested, looking at logs for that date/time could help.
>
> If that happens a lot, a short workaround would be to make the test wait
> for 10 seconds instead of 5.
>


Thanks for the response, Zeljko.  I'm going to make the dumb manager
response and ask the question (not just to Zeljko, but to everyone): as a
permanent fix, can we change the wait to 60 seconds and call it good?  How
was 5 seconds arrived at as the time for an automated test to fail?  Labs
was never intended for performance testing, and it's not suited for it, so
if the rationale is because we're testing end-user performance, we should
stop.  SauceLabs also isn't designed for it, so any effort to use it will
also end in sadness.  60 seconds may be a bit extreme, but really, let's
set a number here (and everywhere we have timeouts) that's high enough to
stop getting false positives, and leave it there.  In cases where we want
to automate a responsiveness test, let's make sure we're doing it against
test2 or some production cluster machine, and that we're doing it from a
client that isn't also likely to introduce random delays.

Rob


> 1:
> https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/55/testReport/junit/(root)/Wikitext%20Editor%20(TEST%20RUN%20ON%20WIKIPEDIA%20SO%20SHOULD%20NOT%20CAUSE%20SAVES)/Opening_editor/
> 2: timed out after 5 seconds, waiting for {:css=>".wikitext-editor",
> :tag_name=>"textarea"} to become present (Watir::Wait::TimeoutError)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140711/d11b18d8/attachment-0001.html>