How’d we do in our strive for operational excellence last month? Read on to find out! Incidents
3 documented incidents last month. That's at the median for the past twelve months, and slightly below the median of 4 over the past five years (Incident stats graphs https://codepen.io/Krinkle/full/wbYMZK).
- 2021-07-14 eventgate latency spike https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-07-14_eventgate-analytics_latency_spike_caused_MW_app_server_overload - Impact: For ~ 10min MediaWiki API clients experienced request failures. - 2021-07-16 codfw-a2 network https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-07-16_asw-a2-codfw_network - Impact: For ~ 1 hour Restbase clients received errors, affecting mobile apps and ContentTranslation. - 2021-07-26 ruwikinews DynamicPageList https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-07-26_ruwikinews_DynamicPageList - Impact: For 30min, 15% of requests from contributors on all wikis failed. There were also brief moments during which no readers could load recently modified or uncached pages.
Learn about past incidents at Incident status https://wikitech.wikimedia.org/wiki/Incident_status on Wikitech. Remember to review and schedule Incident Follow-up https://phabricator.wikimedia.org/project/view/4758/ in Phabricator, which are preventive measures and other action items filed after an incident. Trends
Last month the workboard held 154 non-old unresolved error reports. Over the past thirty days, the collective efforts of our volunteers and engineering teams have closed 14 of those.
In the month of July we've also introduced or discovered thirty-one new error reports (that's an average of one production regression every day!). Of those new error reports, fifteen were resolved and 16 remain unresolved. The workboard now tallies up to 156 tasks.
Over on the backlog, we're continuing to ploddingly present progress on production problems from phantoms of christmases past.
Figure 1, Figure 2: Unresolved error reports stacked by month. https://phabricator.wikimedia.org/phame/post/view/247/production_excellence_34_july_2021/#trends
For more month-over-month numbers refer to the spreadsheet data https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml . Outstanding errors
Take a look at the workboard https://phabricator.wikimedia.org/tag/wikimedia-production-error/ and look for tasks that could use your help.
Below are various older issues that may have fallen by the wayside, taken from somewhat-random stab-in-the-dark queries.
Oldest unresolved errors that are still reproducible (Phab query https://phabricator.wikimedia.org/maniphest/query/07CAHhY.GApw/#R):
- Reported in 2015: Unable to view history of protected Flow board (StructuredDiscussions, Growth team), T118502 https://phabricator.wikimedia.org/T118502. - Reported in 2016: Error when deleting a heading next to a table (VisualEditor, Editing team), T140871 https://phabricator.wikimedia.org/T140871.
Stalled error reports (Phab query https://phabricator.wikimedia.org/maniphest/query/Dmy0AuERAQct/#R):
- Stalled Mar 2021: Constraints check for Q142 France times out (Wikidata, WMDE), T212282 https://phabricator.wikimedia.org/T212282.
Oldest error with a patch for review (Phab query https://phabricator.wikimedia.org/maniphest/query/eb6hYVaKr0Kx/#R):
- Reported in 2016: Maps broken during 2nd live preview (Maps, Product Infra), T151524 https://phabricator.wikimedia.org/T151524. - Reported in 2018: Corrupt connection for cross-wiki db query (Platform team), T193565 https://phabricator.wikimedia.org/T193565.
Jan 2021 (3 of 50 issues https://phabricator.wikimedia.org/maniphest/query/WIP9W8q54HB6/#R left) ⚠️ *Unchanged. Have a look-see!* Feb 2021 (6 of 20 issues https://phabricator.wikimedia.org/maniphest/query/5MzPJAb5oJgv/#R left) ⚠️ *Unchanged. Take a gander!* Mar 2021 (13 of 48 issues https://phabricator.wikimedia.org/maniphest/query/RsVPep46KRY4/#R left) ⚠️ *Unchanged. Check it out!* Apr 2021 (18 of 42 issues https://phabricator.wikimedia.org/maniphest/query/rYyMt_gYYymb/#R left) -1 May 2021 (22 of 54 issues https://phabricator.wikimedia.org/maniphest/query/tmkGqt0C93YG/#R left) -3 June 2021 (11 of 26 issues https://phabricator.wikimedia.org/maniphest/query/roL0TaxtcaLQ/#R left) -4 July 2021 (16 of 31 issues https://phabricator.wikimedia.org/maniphest/query/mUVAD8TJHE3n/#R left) +31; -15 Tally 154 issues open, as of Excellence #33 (June 2021) https://phabricator.wikimedia.org/phame/post/view/240/production_excellence_33_june_2021/ . -14 issues closed, of the previous 154 open issues. +16 new issues that survived July 2021. 156 issues open, as of today Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
🔗 Share or read later via https://phabricator.wikimedia.org/phame/post/view/247/