[QA] [WikimediaMobile] QA: Holding our code to better standards.

[Greg Grossmeier]

Looping in QA list and dropping our team private list.

<quote name="Jon Robson" date="2015-09-03" time="11:45:47 -0700">
> Dear Greg, and anyone else that is involved in deployment
> 
> This is a follow-up from Dan Duvall's talk today during the metrics
> meeting about voting browser tests.
> 
> Background:
> The reading web team this quarter with the help of Dan Duvall has made
> huge strides in our QA infrastructure. The extensions Gather,
> MobileFrontend and now the new extension QuickSurveys are all running
> browser tests on a per commit basis. A selected set of MobileFrontend
> @smoke tests (a selected subset of all he tests) are running in
> 15minutes on every commit and the entire set of Gather browser tests
> is running in around 21minutes. It marginally slows down getting
> patches deployed... but I think this is a good thing. The results
> speak for themselves.
> 
> In the past month (August 4th-September 4th) only 3/33 builds failed
> for MobileFrontend's daily smoke test build [1] (all 3 due to issues
> with the Jenkins infrastructure). For the full set of tests only 10/33
> failed in the Chrome daily build [3], 8 of which were due to tests
> being flakey and needing improvement or issues with the Jenkin
> infrastructure and the two others serious bugs [4,5] brought about by
> work the performance team had been doing that we were able to fix
> shortly after.
> 
> In Firefox [2] there were only 6 failures and only 2 of these were
> serious bugs, again caused by things outside MobileFrontend [4,6]. One
> of these was pretty serious - we had started loading JavaScript for
> users with legacy browsers such as IE6. These were caught prior to the
> daily builds when suddenly our MobileFrontend commits would not merge.
> 
> The future!:
> Given this success:
> 1) I would like to see us run @integration tests on core, but I
> understand given the number of bugs this might not be feasible so far.
> 2) We should run @integration tests prior to deployments to the
> cluster via the train and communicate out when we have failures (and
> make a decision to push broken code)
> 3) I'd like to see other extensions adopt browser test voting on their
> extensions. Please feel free to reach out to me if you need help with
> that. The more coverage across our extensions we have, the better.
> 
> We really have no excuse going forward to push broken code out to our
> users and at the very least we need to be visible to each other when
> we are deploying broken code. We have a responsibility to our users.
> 
> Thoughts? Reactions? Who's with me?!
> 
> [1] https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/
> [2] https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/
> [3] https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/
> [4] https://phabricator.wikimedia.org/T108045
> [5] https://phabricator.wikimedia.org/T108191
> [6] https://phabricator.wikimedia.org/T111233
> 
> _______________________________________________
> Mobile-l mailing list
> Mobile-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- 
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |