[QA] Recently failing browser tests

Tue Jun 2 00:49:28 UTC 2015

Update:
Dan Duval and I sat down and wrote this task to help the generic case for
these problems in future - https://phabricator.wikimedia.org/T101062

The good news:
The caching issue has gone and Gather has gone green (yay!)
https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/

MobileFrontend cache problem has also gone away in the smoke tests but
actually has surfaced a bug. The next build should go green thanks to
https://gerrit.wikimedia.org/r/215257 so watch this space.

Bad news:
the mega builds are still failing (so 2 e-mails should be coming in rather
than 4). Dan Duval and I are going to pair on this tomorrow and fix those
up: https://phabricator.wikimedia.org/T101071

On Mon, Jun 1, 2015 at 10:11 AM, Jon Robson <jrobson at wikimedia.org> wrote:

> Who would be suitable to debug these with and/or examine the caching on
> beta labs? Whilst they fail like this they are useless to us (I've already
> had a request to turn off the e-mail notifications).
>
> It seems Bryan had an idea on the related bug
> https://phabricator.wikimedia.org/T99096
>
> I am free today (and in the office) if anyone wants to pair and get this
> fixed...
>
> On Thu, May 28, 2015 at 9:53 AM, Greg Grossmeier <greg at wikimedia.org>
> wrote:
>
>> (let's use mailing lists instead of manually cc'ing people, please)
>>
>> Timo/Jon: Do you have tasks in phabricator that show what you've tried
>> doing/debugging already? Would hate to re-do it all.
>>
>> Greg
>>
>>
>> <quote name="Timo Tijhof" date="2015-05-28" time="11:57:05 +0200">
>> > I've seen this and similar issues on beta cluster for almost a year now
>> and am reasonably certain that there is no bug in ResourceLoader related to
>> this. We might uncover an indirect cause in MediaWiki core later, but the
>> immediate cause is most likely in Beta cluster. This because I've observed
>> the relevant issues with non-ResourceLoader requests as well. And because
>> we've not been able to reproduce this in production.
>> >
>> > I don't know enough about Beta cluster or our http caching to know how
>> to help. I've already exhausted possibly causes I can think of on my own –
>> last month. However I'm happy to join a hangout session at some point to
>> consult and help pinpoint the problem from a MediaWiki perspective, we
>> should have a prod opsen (knowledgable about Varnish), and a beta devop
>> present there as well.
>> >
>> > -- Timo
>> >
>> > > On 28 May 2015, at 11:37, Jon Robson <jrobson at wikimedia.org> wrote:
>> > >
>> > > Firstly apologises for all the browser test spam recently. This is an
>> update on what is going on and to give you confidence it is being taken
>> seriously.
>> > >
>> > > I've managed to trace the issue to some kind of caching problem in
>> mediawiki core but I'm a bit out of my depth trying to work it out.
>> > >
>> > > Basically under certain circumstances our code is broken and this is
>> why the tests are failing.
>> > >
>> > > The issue is that we are serving old JavaScript along with new
>> JavaScript and these files are naturally incompatible causing the errors as
>> they are trying to access old frontend APIs that no longer exist.
>> > >
>> > > How ResourceLoader deals with caching changed last month and I
>> suspect there is a bug in that code. I suspect Timo would be able to help
>> me chase down the route cause further.
>> > >
>> > > In the meantime I'm not sure if there is a way to stop the email spam
>> till this gets resolved. One idea I had was that when a test build fails it
>> would be great if the email alerts stopped and developers become
>> responsible for restarting them. Greg/Antoine/Dan is this possible? There's
>> little benefit in continuing to run them every day and emailing about
>> faikures until this test is fixed.
>> > >
>> > > Any help on https://phabricator.wikimedia.org/T100542 much
>> appreciated. I'm out of my depth as this issue seems to fall outside the
>> good ship readership and more in the ResourceLoader and quality assurance
>> sea.
>>
>> --
>> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
>> | identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |
>>
>> _______________________________________________
>> QA mailing list
>> QA at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/qa
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20150601/4e77002a/attachment-0001.html>