I didn't Cc mobile-ll
---------- Forwarded message ---------- From: Chris McMahon cmcmahon@wikimedia.org Date: Mon, Jul 14, 2014 at 6:45 AM Subject: Re: [QA] [WikimediaMobile] Failing MobileFrontend browser tests To: "QA (software quality assurance) for Wikimedia projects." < qa@lists.wikimedia.org>
On Mon, Jul 14, 2014 at 5:52 AM, Željko Filipin zfilipin@wikimedia.org wrote:
On Fri, Jul 11, 2014 at 9:35 PM, Arthur Richards arichards@wikimedia.org wrote:
Chris McMahon sent an email announcing this on June 6 [0]. In addition, Chris said 'The tests for MF on beta labs running in headless Firefox under xvfb are reliably green as of today and we'll be working to keep them that way.'
Running tests using xvfb proved to be more unstable that Sauce Labs. We have moved to Sauce until we have some time to investigate failures.
Specifically, more than a few simultaneous headless Firefox processes brings the WMF Jenkins host to its knees, with no viable browser processes that are usable. And SauceLabs has great diagnostics tools available.
*Is there anyone currently owning or willing to own digging into and
resolving these issues? Can we get any kind of timeline for resolving this *- even if it's just in regards to when the issue will be able to be investigated?
Rob, Chris, as far as I know, I have no big projects at the moment. Should I focus on this?
The original post in this thread makes me think that Juliusz was analyzing and resolving these, and then we never heard from him again on this thread. I normally do this along with Juliusz, Jon, Kaldari, Max, etc. when I'm not on vacation.
There are three ways a browser test can fail:
1) A bug in the feature. 2) Something wrong with beta labs 3) New, proper behavior from the feature but the test has not been updated.. 4) Something wrong with the internet or the browser itself.
In the case of 1 and 2, we file a bug in bugzilla. In the case of 3, we update the test code. In the case of 4, we code the tests as defensively as we possibly can.
In all these cases, we have to actually read and understand the test results in order to know what to do about them. Apropos of that, I have been thinking for some time about a tutorial for how to analyze browser test failures and work with the results, I think a number of people would benefit from something like that.
Apropos of 4), I really want to continue the discussion about the changes I left hanging for MF tests in gerrit before going on vacation. This one I think should get +2 very soon, but I really want you MF test developers to understand why the Page Object design pattern is important: https://gerrit.wikimedia.org/r/#/c/142605/ . This one is causing problems in other builds, so I separated out the part that protects the page. This should really be an API call, but this change is an interim step while we add the ability to protect a page to the APIPage object: https://gerrit.wikimedia.org/r/#/c/142605/