[QA] Failing MobileFrontend browser tests

[Željko Filipin]

Hi Juliusz,

comments are inline.

On Thu, Jul 10, 2014 at 12:57 AM, Juliusz Gonera <jgonera at wikimedia.org>
wrote:

> Today I worked a bit on fixing failing browser tests. The good news is
> that some tests detected a regression in core that caused full text search
> on mobile to not work. The bad news is that many of the failures seem to be
> caused by problems with Saucelabs and/or beta labs, examples:
>
>
> https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Wikitext%20Editor%20%28TEST%20RUN%20ON%20WIKIPEDIA%20SO%20SHOULD%20NOT%20CAUSE%20SAVES%29/Opening_editor/
> Editor doesn't seem to load, possible causes: beta labs API error, or
> problem with connection between saucelabs and beta labs
>
>
> https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Issues/Closing_page_issues__browser_back_/
> getaddrinfo: Name or service not known (SocketError) - seems like a
> problem with network on saucelabs
>
>
> https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Page%20actions%20menu%20when%20logged%20in/I_can_add_the_page_to_my_watchlist/
> Saucelabs recording shows "no data received" error in Chrome, either beta
> labs problem or saucelabs network problem
>
>
> https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Generic%20special%20page%20features/Search_from_Watchlist/
> same as above
>

All links now lead to passing tests. When copy/pasting links from Jenkins,
please make sure to use URLs with build number instead of URLs with
"lastBuild".


> Those are just a few examples from recent failures, but they make tracking
> regressions really tedious and time consuming. I know we are planning to
> move away from Saucelabs and use our own servers to run the tests.
>

We have tried moving away from third party services (Cloudbees, Sauce Labs)
and we have succeeded to move all Jenkins jobs from Cloudbees to Wikimedia
Jenkins.

We have tried running tests in local browsers (instead at Sauce Labs) but
the tests were also sometimes failing for unclear reasons, so we are at the
moment again using Sauce Labs. Better the devil you know than the devil you
don't...[1]

I will continue testing and debugging tests with both local and Sauce Labs
browsers and I will let you know the results.

>
> When will this happen? Is there any deadline?
>

As far as I know, there is no deadline.

Željko
--
1:
http://www.usingenglish.com/reference/idioms/better+the+devil+you+know.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140711/26856d46/attachment-0001.html>