I didn't Cc mobile-ll
---------- Forwarded message ----------
From: Chris McMahon <cmcmahon(a)wikimedia.org>
Date: Mon, Jul 14, 2014 at 6:45 AM
Subject: Re: [QA] [WikimediaMobile] Failing MobileFrontend browser tests
To: "QA (software quality assurance) for Wikimedia projects." <
qa(a)lists.wikimedia.org>
On Mon, Jul 14, 2014 at 5:52 AM, Željko Filipin <zfilipin(a)wikimedia.org>
wrote:
On Fri, Jul 11, 2014 at 9:35 PM, Arthur Richards
<arichards(a)wikimedia.org>
wrote:
Chris McMahon sent an email announcing this on
June 6 [0]. In addition,
Chris said 'The tests for MF on beta labs running in headless Firefox under
xvfb are reliably green as of today and we'll be working to keep them that
way.'
Running tests using xvfb proved to be more unstable that Sauce Labs. We
have moved to Sauce until we have some time to investigate failures.
Specifically, more than a few simultaneous headless Firefox processes
brings the WMF Jenkins host to its knees, with no viable browser processes
that are usable. And SauceLabs has great diagnostics tools available.
*Is there anyone currently owning or willing to own
digging into and
resolving these issues? Can we get any kind of
timeline for resolving this *-
even if it's just in regards to when the issue will be able to be
investigated?
Rob, Chris, as far as I know, I have no big projects at the moment. Should
I focus on this?
The original post in this thread makes me think that Juliusz was analyzing
and resolving these, and then we never heard from him again on this thread.
I normally do this along with Juliusz, Jon, Kaldari, Max, etc. when I'm
not on vacation.
There are three ways a browser test can fail:
1) A bug in the feature.
2) Something wrong with beta labs
3) New, proper behavior from the feature but the test has not been updated..
4) Something wrong with the internet or the browser itself.
In the case of 1 and 2, we file a bug in bugzilla. In the case of 3, we
update the test code. In the case of 4, we code the tests as defensively
as we possibly can.
In all these cases, we have to actually read and understand the test
results in order to know what to do about them. Apropos of that, I have
been thinking for some time about a tutorial for how to analyze browser
test failures and work with the results, I think a number of people would
benefit from something like that.
Apropos of 4), I really want to continue the discussion about the changes I
left hanging for MF tests in gerrit before going on vacation. This one I
think should get +2 very soon, but I really want you MF test developers to
understand why the Page Object design pattern is important:
https://gerrit.wikimedia.org/r/#/c/142605/ . This one is causing problems
in other builds, so I separated out the part that protects the page. This
should really be an API call, but this change is an interim step while we
add the ability to protect a page to the APIPage object:
https://gerrit.wikimedia.org/r/#/c/142605/