Basics: We rolled out 1.21wmf5 to the non-Wikipedia sites today, after a
brief reversion and re-deployment to fix breakage in how we were
displaying some styling. We are on track to deploy 1.21wmf5 to English
Wikipedia on Monday, December 3 per
https://www.mediawiki.org/wiki/MediaWiki_1.21/Roadmap .
Below: why this happened and how it got fixed, and what we should change
to prevent problems like this in the future.
What happened:
https://gerrit.wikimedia.org/r/#/c/30361/ changed the headings in the
Vector skin. The new code didn't take the WMF config into account, as
the author wasn't expecting styles and HTML to be cached in such
different ways.
The headings were changed from "h4"/"h5", but the CSS used those tags
to
identify them (instead of using CSS classes). Which means, as expected,
that the page layout breaks for up to 30 days.
Page cache is controlled by the wiki page content. Unless the page is
modified, the cache is kept for up to 30 days for anonymous users.
Resource modules, however, are served by ResourceLoader which has its
own much more efficient and deployable cache mechanism. But this means
that the resources for the skin are deployed globally and site-wide
within 5 minutes.... whereas the HTML isn't for another 2 weeks.
The issues that caused were visible in beta labs for the last three
days, but none of us realized they were significant, we thought they
were caused by a misconfigured memcache; see
https://bugzilla.wikimedia.org/show_bug.cgi?id=42452 .
We knew that this particular change and the related change
https://gerrit.wikimedia.org/r/#/c/34702/ might be problematic and sent
out a note about it on Monday --
http://lists.wikimedia.org/pipermail/wikitech-ambassadors/2012-November/000…
-- but it looks like we didn't test thoroughly enough on Monday and
Tuesday to catch it before the Wednesday deploy. Only anonymous users
would have been affected. We don't cache logged-in users in Squid. So
logged-in users didn't notice problems on
mediawiki.org and
test2.wikipedia.org after the first deploy.
Problems popped up after the Phase 2 deployment to non-Wikipedia sites,
so we reverted the 1.21wmf5 deployment and then redeployed while fixing,
purging, etc.
Bug:
https://bugzilla.wikimedia.org/show_bug.cgi?id=42452
Gerrit changes:
https://gerrit.wikimedia.org/r/#/c/35819 ,
https://gerrit.wikimedia.org/r/#/c/35815/ ,
https://gerrit.wikimedia.org/r/#/c/35817/
What we should fix for the future:
This is why client resources must always be backwards compatible.
"Don't change the HTML in incompatible ways" is probably a good
general rule to live by--but having an easy way to say "start purging
all pages on $theseWikis from Squid/Varnish" would also be nice.
get more manual testing on
test2.wikipedia.org and
mediawiki.org
immediately after Phase I deployment, including as anonymous reader and
editor to ensure we catch Squid caching issues
train more people to review code well, to reduce backlog and catch
these kinds of problems?
get more people to +2 in core and in important extensions
beta labs needs to be trustworthy enough to make this sort of thing
a blocker immediately
Chris McMahon's take: (for what it's worth, this seems to me to be
a sign that beta labs is becoming more and more trustworthy all the
time. The more we actually use it, the more we'll understand what does
and does not work there. We fixed the memcache problem, which fixed the
ability to login, but didn't investigate the display problems because
we're used to beta not being very reliable. In this case, beta was
reliable, and we didn't understand that. Even with a bug report in
bugzilla with 9 subscribers, no one recognized a real issue.)
Chris McMahon said: I think this could be framed as an issue of signal,
noise, and bandwidth. Beta labs being broken a lot, review backlog in
gerrit, false failures in tests are all noise. Given the constraints of
ongoing projects, it is difficult to pick out the signal from the noise.
We can take steps to reduce the noise so that the signal stands out
more by reducing technical debt: make the tests green, make the test
environment robust, keep up with code review.
(I assembled this just now from IRC & mailing list chatter from several
people, and errors are mine -- sorry for missing attributions here.
Drafting was on
http://etherpad.wmflabs.org/pad/p/nov-28-2012-deploysnafu )
--
Sumana Harihareswara
Engineering Community Manager
Wikimedia Foundation