Dear Greg, and anyone else that is involved in deployment
This is a follow-up from Dan Duvall's talk today during the metrics
meeting about voting browser tests.
Background:
The reading web team this quarter with the help of Dan Duvall has made
huge strides in our QA infrastructure. The extensions Gather,
MobileFrontend and now the new extension QuickSurveys are all running
browser tests on a per commit basis. A selected set of MobileFrontend
@smoke tests (a selected subset of all he tests) are running in
15minutes on every commit and the entire set of Gather browser tests
is running in around 21minutes. It marginally slows down getting
patches deployed... but I think this is a good thing. The results
speak for themselves.
In the past month (August 4th-September 4th) only 3/33 builds failed
for MobileFrontend's daily smoke test build [1] (all 3 due to issues
with the Jenkins infrastructure). For the full set of tests only 10/33
failed in the Chrome daily build [3], 8 of which were due to tests
being flakey and needing improvement or issues with the Jenkin
infrastructure and the two others serious bugs [4,5] brought about by
work the performance team had been doing that we were able to fix
shortly after.
In Firefox [2] there were only 6 failures and only 2 of these were
serious bugs, again caused by things outside MobileFrontend [4,6]. One
of these was pretty serious - we had started loading JavaScript for
users with legacy browsers such as IE6. These were caught prior to the
daily builds when suddenly our MobileFrontend commits would not merge.
The future!:
Given this success:
1) I would like to see us run @integration tests on core, but I
understand given the number of bugs this might not be feasible so far.
2) We should run @integration tests prior to deployments to the
cluster via the train and communicate out when we have failures (and
make a decision to push broken code)
3) I'd like to see other extensions adopt browser test voting on their
extensions. Please feel free to reach out to me if you need help with
that. The more coverage across our extensions we have, the better.
We really have no excuse going forward to push broken code out to our
users and at the very least we need to be visible to each other when
we are deploying broken code. We have a responsibility to our users.
Thoughts? Reactions? Who's with me?!
[1]
https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFro…
[2]
https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFro…
[3]
https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFro…
[4]
https://phabricator.wikimedia.org/T108045
[5]
https://phabricator.wikimedia.org/T108191
[6]
https://phabricator.wikimedia.org/T111233