How’d we do in our strive for operational excellence last month? Read on to
find out!
Read on Phabricator at
https://phabricator.wikimedia.org/phame/post/view/235/
Incidents
6 documented incidents. That's above the historical average
<https://codepen.io/Krinkle/full/wbYMZK> of 3–4 per month.
Learn about recent incidents at Incident status
<https://wikitech.wikimedia.org/wiki/Incident_status> on Wikitech, or
Preventive
measures <https://phabricator.wikimedia.org/project/view/4758/> in
Phabricator.
-------
Trends
In April, we saw a continuation of the healthy trend that started this
January
<https://phabricator.wikimedia.org/phame/post/view/227/production_excellence_28_january_2021/#trends>
— a trend where the back of the line is moving forward at least as quickly
as the front of the line. We did take a little breather in March
<https://phabricator.wikimedia.org/phame/post/view/229/production_excellence_30_march_2021/#trends>
where we almost broke even, but otherwise the trend is going well.
Last month we bade farewell to the production errors we found in July 2019.
This month we cleared out the column for October 2019.
One point of concern is that we did encounter a high number of new
production errors — are errors that we failed to catch during development,
code review, continuous integration, beta testing, or pre-deployment
checks. Where we used to discover about a dozen of those a month, whereas
we found 42 during this month. As of writing, 17 of the 42 April-discovered
errors have been resolved.
The "Old" column (generally tracking pre-2019 tasks) grew for the first
time in six months. This increase can largely be attributed to improved
telemetry of client-side errors uncovering issues in under-resourced
products, such as the old Kaltura video player.
Figure 1, Figure 2: Unresolved error reports stacked by month.
<https://phabricator.wikimedia.org/phame/post/view/235/production_excellence_31_april_2021/#trends>
-------
Outstanding errors
Take a look at the workboard
<https://phabricator.wikimedia.org/tag/wikimedia-production-error/> and
look for tasks that could use your help.
Summary over recent months, per spreadsheet
<https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml>
:
Aug 2019 (1 of 14 left) ⚠️ Unchanged (over one year old).
Oct 2019 (0 of 12 left) ✅ Last three tasks resolved! -3
Jan 2020 (1 of 7 left) ⚠️ Unchanged (over one year old).
Mar 2020 (2 of 2 left) ⚠️ Unchanged (over one year old).
Apr 2020 (5 of 14 left) ⚠️ Unchanged (over one year old).
May 2020 (5 of 14 left) ⏸ —
Jun 2020 (5 of 14 left) ⬇️ One task resolved. -1
Jul 2020 (4 of 24 issues
<https://phabricator.wikimedia.org/maniphest/query/s__D8Kd0xuQH/#R>) ⬇️ One
task resolved. -1
Aug 2020 (13 of 53 issues
<https://phabricator.wikimedia.org/maniphest/query/hu1yhWu4sXkP/#R>) ⬇️ Two
tasks resolved. -2
Sep 2020 (7 of 33 issues
<https://phabricator.wikimedia.org/maniphest/query/CGFQViLShnOY/#R>) ⏸ —
Oct 2020 (20 of 69 issues
<https://phabricator.wikimedia.org/maniphest/query/MYnnBybPTYpd/#R>) ⬇️ Two
tasks resolved. -2
Nov 2020 (9 of 38 issues
<https://phabricator.wikimedia.org/maniphest/query/CkC_VqQq5VC0/#R>) ⏸ —
Dec 2020 (7 of 33 issues
<https://phabricator.wikimedia.org/maniphest/query/10NQy74iKaZJ/#R>) ⬇️
Four tasks resolved. -4
Jan 2021 (1 of 50 issues
<https://phabricator.wikimedia.org/maniphest/query/WIP9W8q54HB6/#R>) ⬇️ One
task resolved. -1
Feb 2021 (8 of 20 issues
<https://phabricator.wikimedia.org/maniphest/query/5MzPJAb5oJgv/#R>) ⬇️ One
task resolved. -1
Mar 2021 (18 of 48 issues
<https://phabricator.wikimedia.org/maniphest/query/RsVPep46KRY4/#R>) ⬇️
Sixteen tasks resolved. -16
*Apr 2021* (25 of 42 issues
<https://phabricator.wikimedia.org/maniphest/query/rYyMt_gYYymb/#R>) 42 new
issues found, of which 25 remained open. +42; -17
-------
Tally
139 issues open, as of Excellence #30
<https://phabricator.wikimedia.org/phame/post/view/229/production_excellence_30_march_2021/>
(March 2021).
-31 issues closed, of the previously open issues.
+25 new issues that survived April 2021.
133 issues open, as of today (12 May 2021).
-------
Thanks
Thank you to everyone who helped by reporting, investigating, or resolving
problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof