Today I worked a bit on fixing failing browser tests. The good news is that some tests detected a regression in core that caused full text search on mobile to not work. The bad news is that many of the failures seem to be caused by problems with Saucelabs and/or beta labs, examples:
https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-Mobi... Editor doesn't seem to load, possible causes: beta labs API error, or problem with connection between saucelabs and beta labs
https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-Mobi... getaddrinfo: Name or service not known (SocketError) - seems like a problem with network on saucelabs
https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-Mobi... Saucelabs recording shows "no data received" error in Chrome, either beta labs problem or saucelabs network problem
https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-Mobi... same as above
Those are just a few examples from recent failures, but they make tracking regressions really tedious and time consuming. I know we are planning to move away from Saucelabs and use our own servers to run the tests. When will this happen? Is there any deadline?
Thanks,