[QA] [WikimediaMobile] Failing MobileFrontend browser tests

[Chris McMahon]

On Mon, Jul 14, 2014 at 5:52 AM, Željko Filipin <zfilipin at wikimedia.org>
wrote:

> On Fri, Jul 11, 2014 at 9:35 PM, Arthur Richards <arichards at wikimedia.org>
> wrote:
>
>> Chris McMahon sent an email announcing this on June 6 [0]. In addition,
>> Chris said 'The tests for MF on beta labs running in headless Firefox under
>> xvfb are reliably green as of today and we'll be working to keep them that
>> way.'
>>
>
> Running tests using xvfb proved to be more unstable that Sauce Labs. We
> have moved to Sauce until we have some time to investigate failures.
>

Specifically, more than a few simultaneous headless Firefox processes
brings the WMF Jenkins host to its knees, with no viable browser processes
that are usable.  And SauceLabs has great diagnostics tools available.


> *Is there anyone currently owning or willing to own digging into and
>> resolving these issues? Can we get any kind of timeline for resolving this *-
>> even if it's just in regards to when the issue will be able to be
>> investigated?
>>
>
> Rob, Chris, as far as I know, I have no big projects at the moment. Should
> I focus on this?
>

The original post in this thread makes me think that Juliusz was analyzing
and resolving these, and then we never heard from him again on this thread.
 I normally do this along with Juliusz, Jon, Kaldari, Max, etc. when I'm
not on vacation.

There are three ways a browser test can fail:

1) A bug in the feature.
2) Something wrong with beta labs
3) New, proper behavior from the feature but the test has not been updated..
4) Something wrong with the internet or the browser itself.

In the case of 1 and 2, we file a bug in bugzilla.  In the case of 3, we
update the test code.  In the case of 4, we code the tests as defensively
as we possibly can.

In all these cases, we have to actually read and understand the test
results in order to know what to do about them.  Apropos of that, I have
been thinking for some time about a tutorial for how to analyze browser
test failures and work with the results, I think a number of people would
benefit from something like that.

Apropos of 4), I really want to continue the discussion about the changes I
left hanging for MF tests in gerrit before going on vacation.  This one I
think should get +2 very soon, but I really want you MF test developers to
understand why the Page Object design pattern is important:
https://gerrit.wikimedia.org/r/#/c/142605/ .  This one is causing problems
in other builds, so I separated out the part that protects the page.  This
should really be an API call, but this change is an interim step while we
add the ability to protect a page to the APIPage object:
https://gerrit.wikimedia.org/r/#/c/142605/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140714/75dd2128/attachment.html>