Hi Juliusz,
comments are inline.
On Thu, Jul 10, 2014 at 12:57 AM, Juliusz Gonera jgonera@wikimedia.org wrote:
Today I worked a bit on fixing failing browser tests. The good news is that some tests detected a regression in core that caused full text search on mobile to not work. The bad news is that many of the failures seem to be caused by problems with Saucelabs and/or beta labs, examples:
https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-Mobi... Editor doesn't seem to load, possible causes: beta labs API error, or problem with connection between saucelabs and beta labs
https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-Mobi... getaddrinfo: Name or service not known (SocketError) - seems like a problem with network on saucelabs
https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-Mobi... Saucelabs recording shows "no data received" error in Chrome, either beta labs problem or saucelabs network problem
https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-Mobi... same as above
All links now lead to passing tests. When copy/pasting links from Jenkins, please make sure to use URLs with build number instead of URLs with "lastBuild".
Those are just a few examples from recent failures, but they make tracking regressions really tedious and time consuming. I know we are planning to move away from Saucelabs and use our own servers to run the tests.
We have tried moving away from third party services (Cloudbees, Sauce Labs) and we have succeeded to move all Jenkins jobs from Cloudbees to Wikimedia Jenkins.
We have tried running tests in local browsers (instead at Sauce Labs) but the tests were also sometimes failing for unclear reasons, so we are at the moment again using Sauce Labs. Better the devil you know than the devil you don't...[1]
I will continue testing and debugging tests with both local and Sauce Labs browsers and I will let you know the results.
When will this happen? Is there any deadline?
As far as I know, there is no deadline.
Željko -- 1: http://www.usingenglish.com/reference/idioms/better+the+devil+you+know.html