How’d we do in our strive for operational excellence last month? Read on to
find out!
Read on Phabricator at
https://phabricator.wikimedia.org/phame/post/view/229/
Incidents
2 documented incidents. That's average for this time of year, when we usually
had <https://codepen.io/Krinkle/full/wbYMZK> 1-4 incidents.
Learn about recent incidents at Incident status
<https://wikitech.wikimedia.org/wiki/Incident_status> on Wikitech, or
Preventive
measures <https://phabricator.wikimedia.org/project/view/4758/> in
Phabricator.
-------
Trends
In March we made significant progress on the outstanding errors of previous
months. Several of the 2020 months are finally starting to empty out. But
with over 30 new tasks from March itself remaining, we did not break even,
and ended up slightly higher than last month. This could be reversing two
positive trends, but I hope not.
Firstly, there was a steep increase in the number of new production errors
that were not resolved within the same month. This is counter the positive
trend we started in November. The past four months typically saw 10-20
errors outlive their month of discovery, and this past month saw 34 of its
48 new errors remain unresolved.
Secondly, we saw the overall number of unresolved errors increase again. This
January
<https://phabricator.wikimedia.org/phame/post/view/227/production_excellence_28_january_2021/>
began a downward trend for the first time in thirteen months, which
continued nicely through February. But, this past month we broke even and
even pushed upward by one task. I hope this is just a breather and we can
continue our way downward.
Figure 1, Figure 2: Unresolved error reports stacked by month.
<https://phabricator.wikimedia.org/phame/post/view/229/production_excellence_30_march_2021/#trends>
-------
Outstanding errors
Take a look at the workboard
<https://phabricator.wikimedia.org/tag/wikimedia-production-error/> and
look for tasks that could use your help.
Summary over recent months, per spreadsheet
<https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml>
:
Jul 2019 (0 of 18 left) ✅ Last two tasks resolved! -2
Aug 2019 (1 of 14 left) ⚠️ *Unchanged (and over one year old).*
Oct 2019 (3 of 12 left) ⬇️ One task resolved. -1
Nov 2019 (0 of 5 left) ✅ Last task resolved! -1
Dec 2019 (0 of 9 left) ✅ Last task resolved! -1
Jan 2020 (2 of 7 left) ⬇️ One task resolved. -1
Feb 2020 (0 of 7 left) ✅ Last task resolved! -1
Mar 2020 (2 of 2 left) ⚠️ *Unchanged (and over one year old).*
Apr 2020 (5 of 14 left) ⬇️ Four tasks resolved. -4
May 2020 (5 of 14 left) ⬇️ One task resolved. -1
Jun 2020 (6 of 14 left) ⬇️ One task resolved. -1
Jul 2020 (5 of 24 issues
<https://phabricator.wikimedia.org/maniphest/query/s__D8Kd0xuQH/#R>) ⬇️
Four tasks resolved. -4
Aug 2020 (15 of 53 issues
<https://phabricator.wikimedia.org/maniphest/query/hu1yhWu4sXkP/#R>) ⬇️
Five tasks resolved. -5
Sep 2020 (7 of 33 issues
<https://phabricator.wikimedia.org/maniphest/query/CGFQViLShnOY/#R>) ⬇️ One
task resolved. -1
Oct 2020 (22 of 69 issues
<https://phabricator.wikimedia.org/maniphest/query/MYnnBybPTYpd/#R>) ⬇️
Four tasks resolved. -4
Nov 2020 (9 of 38 issues
<https://phabricator.wikimedia.org/maniphest/query/CkC_VqQq5VC0/#R>) ⬇️ Two
tasks resolved. -2
Dec 2020 (11 of 33 issues
<https://phabricator.wikimedia.org/maniphest/query/10NQy74iKaZJ/#R>) ⬇️ One
task resolved. -1
Jan 2021 (4 of 50 issues
<https://phabricator.wikimedia.org/maniphest/query/WIP9W8q54HB6/#R>) ⬇️ One
task resolved. -1
Feb 2021 (9 of 20 issues
<https://phabricator.wikimedia.org/maniphest/query/5MzPJAb5oJgv/#R>) ⬇️ Two
tasks resolved. -2
*Mar 2021* (34 of 48 issues
<https://phabricator.wikimedia.org/maniphest/query/RsVPep46KRY4/#R>) 34 new
tasks survived and remain unresolved. +48; -14
-------
Tally
138 issues open, as of Excellence #29
<https://phabricator.wikimedia.org/phame/post/view/228/production_excellence_29_february_2021/>
(6 Mar 2021).
-33 issues closed, of the previous 138 open issues.
+34 new issues that survived March 2021.
139 issues open, as of today (2 Apr 2021).
-------Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving
problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
Footnotes:
Incident status, Wikitech
<https://wikitech.wikimedia.org/wiki/Incident_status>.
Wikimedia incident stats by Krinkle, CodePen
<https://codepen.io/Krinkle/full/wbYMZK>.
Production Excellence: Month-over-month spreadsheet and plot
<https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml>
.
Report charts for Wikimedia-production-error project, Phabricator
<https://phabricator.wikimedia.org/project/reports/1055/>.