<div dir="ltr">While the migration from Cloudbees was completed earlier this month, the MobileFrontend jobs were migrated off of Cloudbees over one month ago. <br><br>Chris McMahon sent an email announcing this on June 6 [0]. In addition, Chris said 'The tests for MF on beta labs running in headless Firefox under xvfb are reliably green as of today and we'll be working to keep them that way.' <br>
<br>However, those tests have been consistently failing since early June. After digging through MobileFrontend test failures done in particular by Jon and Juliusz, <b>it's clear that the consistency of the failures are related to architectural/infrastructural issues</b>. Jon brought this up on June 9 [1], with a response from Zeljko on June 28 [2] mentioning that the issue was known, but that they haven't had time to debug the problem. Fast forward two weeks, and Juliusz resurfaced the problem with this thread since we haven't heard any additional information in regards to resolving the issue, while dealing with a high degree of noise from build failures.<br>
<br>We really appreciate all of the hard work that release/qa/platform/etc has put into this and we understand that resolving issues takes time. When we had more reliable builds, we found the automated browser tests to be incredibly valuable. We want to regain that value so that we can again more reliably catch issues before they find their way to production.<br>
<br><b>Is there anyone currently owning or willing to own digging into and resolving these issues? Can we get any kind of timeline for resolving this </b>- even if it's just in regards to when the issue will be able to be investigated? In the mean time, let's remove the mobile web team from the failure notifications until such time that the builds are reliable and we can depend on a better signal-to-noise ratio.<br>
<br>As an aside - I didn't know the migration work to Cloudbees was complete until it was addressed on this thread. It looks like the only announcement about it was made on the qa list, where I and many folks affected by this change are not subscribed. Please make announcements about things as significant as this on broad-reaching lists like wikitech-l.<div>
<br></div><div>[0] <span style="font-family:arial"><a href="http://lists.wikimedia.org/pipermail/qa/2014-June/001515.html">http://lists.wikimedia.org/pipermail/qa/2014-June/001515.html</a></span></div><div><span style="font-family:arial">[1] </span><span style="font-family:arial"><a href="http://lists.wikimedia.org/pipermail/qa/2014-June/001535.html">http://lists.wikimedia.org/pipermail/qa/2014-June/001535.html</a></span></div>
<div><span style="font-family:arial">[2] </span><span style="font-family:arial"><a href="http://lists.wikimedia.org/pipermail/qa/2014-June/001615.html">http://lists.wikimedia.org/pipermail/qa/2014-June/001615.html</a></span></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 10, 2014 at 6:37 PM, Rob Lanphier <span dir="ltr"><<a href="mailto:robla@wikimedia.org" target="_blank">robla@wikimedia.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div class="">On Thu, Jul 10, 2014 at 1:38 PM, Arthur Richards <span dir="ltr"><<a href="mailto:arichards@wikimedia.org" target="_blank">arichards@wikimedia.org</a>></span> wrote:<br>
</div><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the replies, everyone - however I haven't seen Antoine's response (beyond the bit quoted in Chris Steipp's reply), did it not get reply all'd or otherwise not sent to mobile-l? Further response inline.<div class="gmail_extra">
<br><div class="gmail_quote"><div>On Thu, Jul 10, 2014 at 12:06 PM, Rob Lanphier <span dir="ltr"><<a href="mailto:robla@wikimedia.org" target="_blank">robla@wikimedia.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><span style="font-family:arial">Antoine and Zeljko are the right people to talk about this while Chris is out, and it's late in the day for them. I'm sure they'll get back to you tomorrow. Greg may be able to say more about this, but honestly, the nature of this thread is a little bit like little kids in the backseat saying "are we there yet? are we there yet?" repeatedly :-) </span></div>
</div></blockquote><div><br></div></div><div>Rob, the last time we inquired about this (to my knowledge) was one month ago (see email subject 'migrating MobileFrontend browser tests to WMF Jenkins'). We saw no substantive followup, and our browser test builds have been broken since the migration over one month ago as a result of infrastructural issues. This not only rendered the automated browser tests essentially useless for us but it has also become a significant drain on time and focus. This isn't a case of us asking 'are we there yet' - rather it's a case of us trying to understand when we'll be able to rely on our browser tests again and to see if there's any way we can help to improve the situation.</div>
</div><div><div><br></div></div></div></div></blockquote><div><br></div></div></div><div>The completion of the migration to Saucelabs was announced July 3. If you aren't getting replies, it's probably because they are stuck in the mobile-l moderation queue. Antoine did send his email to the mobile-l list, but is probably not a member. In general, there has been a fair amount of conversation on the topic on the qa list, so I'd encourage you to check out the activity there if you haven't already seen it.</div>
<div><br></div><div>Sorry for being glib earlier. I understand the current situation must be frustrating for you all, and I hope Zeljko and Antoine are able to provide a satisfactory answer on this. Chris will also be back on Monday, though unfortunately Antoine will be gone at that point.</div>
<span class="HOEnZb"><font color="#888888">
<div><br></div><div>Rob</div><div> </div>
<div><br></div><div><br></div></font></span></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Arthur Richards<div>Team Practices Lead</div><div>[[User:Awjrichards]]</div><div>IRC: awjr</div><div>+1-415-839-6885 x6687</div></div>
</div>