[QA] Failing MobileFrontend browser tests

[Juliusz Gonera]

Today I worked a bit on fixing failing browser tests. The good news is that
some tests detected a regression in core that caused full text search on
mobile to not work. The bad news is that many of the failures seem to be
caused by problems with Saucelabs and/or beta labs, examples:

https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Wikitext%20Editor%20%28TEST%20RUN%20ON%20WIKIPEDIA%20SO%20SHOULD%20NOT%20CAUSE%20SAVES%29/Opening_editor/
Editor doesn't seem to load, possible causes: beta labs API error, or
problem with connection between saucelabs and beta labs

https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Issues/Closing_page_issues__browser_back_/
getaddrinfo: Name or service not known (SocketError) - seems like a problem
with network on saucelabs

https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Page%20actions%20menu%20when%20logged%20in/I_can_add_the_page_to_my_watchlist/
Saucelabs recording shows "no data received" error in Chrome, either beta
labs problem or saucelabs network problem

https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Generic%20special%20page%20features/Search_from_Watchlist/
same as above

Those are just a few examples from recent failures, but they make tracking
regressions really tedious and time consuming. I know we are planning to
move away from Saucelabs and use our own servers to run the tests. When
will this happen? Is there any deadline?

Thanks,
-- 
Juliusz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140709/fe4fb0c1/attachment.html>