[QA] [WikimediaMobile] Failing MobileFrontend browser tests

[Arthur Richards]

While the migration from Cloudbees was completed earlier this month, the
MobileFrontend jobs were migrated off of Cloudbees over one month ago.

Chris McMahon sent an email announcing this on June 6 [0]. In addition,
Chris said 'The tests for MF on beta labs running in headless Firefox under
xvfb are reliably green as of today and we'll be working to keep them that
way.'

However, those tests have been consistently failing since early June. After
digging through MobileFrontend test failures done in particular by Jon and
Juliusz, *it's clear that the consistency of the failures are related to
architectural/infrastructural issues*. Jon brought this up on June 9 [1],
with a response from Zeljko on June 28 [2] mentioning that the issue was
known, but that they haven't had time to debug the problem. Fast forward
two weeks, and Juliusz resurfaced the problem with this thread since we
haven't heard any additional information in regards to resolving the issue,
while dealing with a high degree of noise from build failures.

We really appreciate all of the hard work that release/qa/platform/etc has
put into this and we understand that resolving issues takes time. When we
had more reliable builds, we found the automated browser tests to be
incredibly valuable. We want to regain that value so that we can again more
reliably catch issues before they find their way to production.

*Is there anyone currently owning or willing to own digging into and
resolving these issues? Can we get any kind of timeline for resolving this *-
even if it's just in regards to when the issue will be able to be
investigated? In the mean time, let's remove the mobile web team from the
failure notifications until such time that the builds are reliable and we
can depend on a better signal-to-noise ratio.

As an aside - I didn't know the migration work to Cloudbees was complete
until it was addressed on this thread. It looks like the only announcement
about it was made on the qa list, where I and many folks affected by this
change are not subscribed. Please make announcements about things as
significant as this on broad-reaching lists like wikitech-l.

[0] http://lists.wikimedia.org/pipermail/qa/2014-June/001515.html
[1] http://lists.wikimedia.org/pipermail/qa/2014-June/001535.html
[2] http://lists.wikimedia.org/pipermail/qa/2014-June/001615.html


On Thu, Jul 10, 2014 at 6:37 PM, Rob Lanphier <robla at wikimedia.org> wrote:

> On Thu, Jul 10, 2014 at 1:38 PM, Arthur Richards <arichards at wikimedia.org>
> wrote:
>
>> Thanks for the replies, everyone - however I haven't seen Antoine's
>> response (beyond the bit quoted in Chris Steipp's reply), did it not get
>> reply all'd or otherwise not sent to mobile-l? Further response inline.
>>
>> On Thu, Jul 10, 2014 at 12:06 PM, Rob Lanphier <robla at wikimedia.org>
>> wrote:
>>
>>> Antoine and Zeljko are the right people to talk about this while Chris
>>> is out, and it's late in the day for them.  I'm sure they'll get back to
>>> you tomorrow.  Greg may be able to say more about this, but honestly, the
>>> nature of this thread is a little bit like little kids in the backseat
>>> saying "are we there yet? are we there yet?" repeatedly  :-)
>>>
>>
>> Rob, the last time we inquired about this (to my knowledge) was one month
>> ago (see email subject 'migrating MobileFrontend browser tests to WMF
>> Jenkins'). We saw no substantive followup, and our browser test builds have
>> been broken since the migration over one month ago as a result of
>> infrastructural issues. This not only rendered the automated browser tests
>> essentially useless for us but it has also become a significant drain on
>> time and focus. This isn't a case of us asking 'are we there yet' - rather
>> it's a case of us trying to understand when we'll be able to rely on our
>> browser tests again and to see if there's any way we can help to improve
>> the situation.
>>
>>
> The completion of the migration to Saucelabs was announced July 3.  If you
> aren't getting replies, it's probably because they are stuck in the
> mobile-l moderation queue.  Antoine did send his email to the mobile-l
> list, but is probably not a member.  In general, there has been a fair
> amount of conversation on the topic on the qa list, so I'd encourage you to
> check out the activity there if you haven't already seen it.
>
> Sorry for being glib earlier.  I understand the current situation must be
> frustrating for you all, and I hope Zeljko and Antoine are able to provide
> a satisfactory answer on this.  Chris will also be back on Monday, though
> unfortunately Antoine will be gone at that point.
>
> Rob
>
>
>
>


-- 
Arthur Richards
Team Practices Lead
[[User:Awjrichards]]
IRC: awjr
+1-415-839-6885 x6687
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140711/e44c7de9/attachment.html>