<div dir="ltr">Hi Juliusz,<div><br></div><div>comments are inline.<br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 10, 2014 at 12:57 AM, Juliusz Gonera <span dir="ltr"><<a href="mailto:jgonera@wikimedia.org" target="_blank">jgonera@wikimedia.org</a>></span> wrote:<br>

<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Today I worked a bit on fixing failing browser tests. The good news is that some tests detected a regression in core that caused full text search on mobile to not work. The bad news is that many of the failures seem to be caused by problems with Saucelabs and/or beta labs, examples:<br>


<br><a href="https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Wikitext%20Editor%20%28TEST%20RUN%20ON%20WIKIPEDIA%20SO%20SHOULD%20NOT%20CAUSE%20SAVES%29/Opening_editor/" target="_blank">https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Wikitext%20Editor%20%28TEST%20RUN%20ON%20WIKIPEDIA%20SO%20SHOULD%20NOT%20CAUSE%20SAVES%29/Opening_editor/</a><br>


</div>Editor doesn't seem to load, possible causes: beta labs API error, or problem with connection between saucelabs and beta labs<br><br><a href="https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Issues/Closing_page_issues__browser_back_/" target="_blank">https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Issues/Closing_page_issues__browser_back_/</a><br>


getaddrinfo: Name or service not known (SocketError) - seems like a problem with network on saucelabs<br><br><a href="https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Page%20actions%20menu%20when%20logged%20in/I_can_add_the_page_to_my_watchlist/" target="_blank">https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Page%20actions%20menu%20when%20logged%20in/I_can_add_the_page_to_my_watchlist/</a><br>


</div>Saucelabs recording shows "no data received" error in Chrome, either beta labs problem or saucelabs network problem<br><br><a href="https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Generic%20special%20page%20features/Search_from_Watchlist/" target="_blank">https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Generic%20special%20page%20features/Search_from_Watchlist/</a><br>


</div>same as above<br></div></div></div></blockquote><div><br></div>All links now lead to passing tests. When copy/pasting links from Jenkins, please make sure to use URLs with build number instead of URLs with "lastBuild".<div>

 </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Those are just a few examples from recent failures, but they make tracking regressions really tedious and time consuming. I know we are planning to move away from Saucelabs and use our own servers to run the tests. </div>

</div></blockquote><div><br></div><div>We have tried moving away from third party services (Cloudbees, Sauce Labs) and we have succeeded to move all Jenkins jobs from Cloudbees to Wikimedia Jenkins.</div><div><br></div>We have tried running tests in local browsers (instead at Sauce Labs) but the tests were also sometimes failing for unclear reasons, so we are at the moment again using Sauce Labs. Better the devil you know than the devil you don't...[1]</div>

<div class="gmail_quote"><br></div><div class="gmail_quote">I will continue testing and debugging tests with both local and Sauce Labs browsers and I will let you know the results.<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">

<div dir="ltr"><div>When will this happen? Is there any deadline?<br></div></div></blockquote><div><br></div><div>As far as I know, there is no deadline.</div><div><br></div><div>Željko </div><div>--</div><div>1: <a href="http://www.usingenglish.com/reference/idioms/better+the+devil+you+know.html">http://www.usingenglish.com/reference/idioms/better+the+devil+you+know.html</a></div>

</div></div></div></div>