<div dir="ltr"><div>Hi all,</div><div><br></div>Antoine and Zeljko are the right people to talk about this while Chris is out, and it's late in the day for them. I'm sure they'll get back to you tomorrow. Greg may be able to say more about this, but honestly, the nature of this thread is a little bit like little kids in the backseat saying "are we there yet? are we there yet?" repeatedly :-) <div>
<br></div><div>Antoine's response seems to answer the substance of what y'all are asking about. We moved from Cloudbees to directly using Saucelabs so that we could debug these issues directly. Now that we're on Saucelabs, we have the info (see Antoine's mail). As he said, we have no plans to set up our own version of Saucelabs.</div>
<div><br></div><div>It may be that the very first thing we need to do is put some sort of environment health check prior to executing the actual test portion to avoid these false failures. Given that the team just completed the migration off of Cloudbees, give them a little time to figure things out.</div>
<div><br></div><div>Thanks</div><div><div><div>Rob</div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 10, 2014 at 11:09 AM, Tomasz Finc <span dir="ltr"><<a href="mailto:tfinc@wikimedia.org" target="_blank">tfinc@wikimedia.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">RobLa, is this something that we should be pulling in Greg for?<br>
<span class="HOEnZb"><font color="#888888"><br>
--tomasz<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On Thu, Jul 10, 2014 at 11:08 AM, Arthur Richards<br>
<<a href="mailto:arichards@wikimedia.org">arichards@wikimedia.org</a>> wrote:<br>
> Head's up that Chris Mcmahon is on vacation all this week. That said, it<br>
> would be great to hear from anyone in QA about this - it has been a long<br>
> standing issue.<br>
><br>
><br>
> On Thu, Jul 10, 2014 at 10:51 AM, Tomasz Finc <<a href="mailto:tfinc@wikimedia.org">tfinc@wikimedia.org</a>> wrote:<br>
>><br>
>> ChrisMC,<br>
>><br>
>> Are these failures unique to mobile? They seem look to be at the<br>
>> infrastructure level so i'm guessing it would affect others.<br>
>><br>
>> What other information do you need from us to be able to remedy these?<br>
>><br>
>> --tomasz<br>
>><br>
>> On Wed, Jul 9, 2014 at 5:54 PM, Jon Robson <<a href="mailto:jdlrobson@gmail.com">jdlrobson@gmail.com</a>> wrote:<br>
>> > Indeed. The tests have been failing for a month now, and had been<br>
>> > passing green before the move to <a href="http://integration.wikimedia.org" target="_blank">integration.wikimedia.org</a><br>
>> > It would be really good to get these back to being useful.<br>
>> ><br>
>> > I'm not sure how our interaction with saucelabs changed during that<br>
>> > move, but is there anything that can be done on the short term to get<br>
>> > it back to how they were before when we were on cloudbees?<br>
>> ><br>
>> > Thanks Juliusz for the good summary of the problems!<br>
>> ><br>
>> ><br>
>> > On Wed, Jul 9, 2014 at 3:57 PM, Juliusz Gonera <<a href="mailto:jgonera@wikimedia.org">jgonera@wikimedia.org</a>><br>
>> > wrote:<br>
>> >> Today I worked a bit on fixing failing browser tests. The good news is<br>
>> >> that<br>
>> >> some tests detected a regression in core that caused full text search<br>
>> >> on<br>
>> >> mobile to not work. The bad news is that many of the failures seem to<br>
>> >> be<br>
>> >> caused by problems with Saucelabs and/or beta labs, examples:<br>
>> >><br>
>> >><br>
>> >> <a href="https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Wikitext%20Editor%20%28TEST%20RUN%20ON%20WIKIPEDIA%20SO%20SHOULD%20NOT%20CAUSE%20SAVES%29/Opening_editor/" target="_blank">https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Wikitext%20Editor%20%28TEST%20RUN%20ON%20WIKIPEDIA%20SO%20SHOULD%20NOT%20CAUSE%20SAVES%29/Opening_editor/</a><br>
>> >> Editor doesn't seem to load, possible causes: beta labs API error, or<br>
>> >> problem with connection between saucelabs and beta labs<br>
>> >><br>
>> >><br>
>> >> <a href="https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Issues/Closing_page_issues__browser_back_/" target="_blank">https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Issues/Closing_page_issues__browser_back_/</a><br>
>> >> getaddrinfo: Name or service not known (SocketError) - seems like a<br>
>> >> problem<br>
>> >> with network on saucelabs<br>
>> >><br>
>> >><br>
>> >> <a href="https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Page%20actions%20menu%20when%20logged%20in/I_can_add_the_page_to_my_watchlist/" target="_blank">https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Page%20actions%20menu%20when%20logged%20in/I_can_add_the_page_to_my_watchlist/</a><br>
>> >> Saucelabs recording shows "no data received" error in Chrome, either<br>
>> >> beta<br>
>> >> labs problem or saucelabs network problem<br>
>> >><br>
>> >><br>
>> >> <a href="https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Generic%20special%20page%20features/Search_from_Watchlist/" target="_blank">https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/lastBuild/testReport/junit/%28root%29/Generic%20special%20page%20features/Search_from_Watchlist/</a><br>
>> >> same as above<br>
>> >><br>
>> >> Those are just a few examples from recent failures, but they make<br>
>> >> tracking<br>
>> >> regressions really tedious and time consuming. I know we are planning<br>
>> >> to<br>
>> >> move away from Saucelabs and use our own servers to run the tests. When<br>
>> >> will<br>
>> >> this happen? Is there any deadline?<br>
>> >><br>
>> >> Thanks,<br>
>> >> --<br>
>> >> Juliusz<br>
>> >><br>
>> >> _______________________________________________<br>
>> >> Mobile-l mailing list<br>
>> >> <a href="mailto:Mobile-l@lists.wikimedia.org">Mobile-l@lists.wikimedia.org</a><br>
>> >> <a href="https://lists.wikimedia.org/mailman/listinfo/mobile-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/mobile-l</a><br>
>> >><br>
>> ><br>
>> ><br>
>> ><br>
>> > --<br>
>> > Jon Robson<br>
>> > * <a href="http://jonrobson.me.uk" target="_blank">http://jonrobson.me.uk</a><br>
>> > * <a href="https://www.facebook.com/jonrobson" target="_blank">https://www.facebook.com/jonrobson</a><br>
>> > * @rakugojon<br>
>> ><br>
>> > _______________________________________________<br>
>> > Mobile-l mailing list<br>
>> > <a href="mailto:Mobile-l@lists.wikimedia.org">Mobile-l@lists.wikimedia.org</a><br>
>> > <a href="https://lists.wikimedia.org/mailman/listinfo/mobile-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/mobile-l</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Arthur Richards<br>
> Team Practices Lead<br>
> [[User:Awjrichards]]<br>
> IRC: awjr<br>
> <a href="tel:%2B1-415-839-6885%20x6687" value="+14158396885">+1-415-839-6885 x6687</a><br>
</div></div></blockquote></div><br></div>