How’d we do in our strive for operational excellence last month? Read on to find out!

Read on Phabricator at https://phabricator.wikimedia.org/phame/post/view/229/ Incidents

2 documented incidents. That's average for this time of year, when we usually had https://codepen.io/Krinkle/full/wbYMZK 1-4 incidents.

Learn about recent incidents at Incident status https://wikitech.wikimedia.org/wiki/Incident_status on Wikitech, or Preventive measures https://phabricator.wikimedia.org/project/view/4758/ in Phabricator.

-------

Trends

In March we made significant progress on the outstanding errors of previous months. Several of the 2020 months are finally starting to empty out. But with over 30 new tasks from March itself remaining, we did not break even, and ended up slightly higher than last month. This could be reversing two positive trends, but I hope not.

Firstly, there was a steep increase in the number of new production errors that were not resolved within the same month. This is counter the positive trend we started in November. The past four months typically saw 10-20 errors outlive their month of discovery, and this past month saw 34 of its 48 new errors remain unresolved.

Secondly, we saw the overall number of unresolved errors increase again. This January https://phabricator.wikimedia.org/phame/post/view/227/production_excellence_28_january_2021/ began a downward trend for the first time in thirteen months, which continued nicely through February. But, this past month we broke even and even pushed upward by one task. I hope this is just a breather and we can continue our way downward.

Figure 1, Figure 2: Unresolved error reports stacked by month. https://phabricator.wikimedia.org/phame/post/view/229/production_excellence_30_march_2021/#trends

-------

Outstanding errors

Take a look at the workboard https://phabricator.wikimedia.org/tag/wikimedia-production-error/ and look for tasks that could use your help.

Summary over recent months, per spreadsheet https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml : Jul 2019 (0 of 18 left) ✅ Last two tasks resolved! -2 Aug 2019 (1 of 14 left) ⚠️ *Unchanged (and over one year old).* Oct 2019 (3 of 12 left) ⬇️ One task resolved. -1 Nov 2019 (0 of 5 left) ✅ Last task resolved! -1 Dec 2019 (0 of 9 left) ✅ Last task resolved! -1 Jan 2020 (2 of 7 left) ⬇️ One task resolved. -1 Feb 2020 (0 of 7 left) ✅ Last task resolved! -1 Mar 2020 (2 of 2 left) ⚠️ *Unchanged (and over one year old).* Apr 2020 (5 of 14 left) ⬇️ Four tasks resolved. -4 May 2020 (5 of 14 left) ⬇️ One task resolved. -1 Jun 2020 (6 of 14 left) ⬇️ One task resolved. -1 Jul 2020 (5 of 24 issues https://phabricator.wikimedia.org/maniphest/query/s__D8Kd0xuQH/#R) ⬇️ Four tasks resolved. -4 Aug 2020 (15 of 53 issues https://phabricator.wikimedia.org/maniphest/query/hu1yhWu4sXkP/#R) ⬇️ Five tasks resolved. -5 Sep 2020 (7 of 33 issues https://phabricator.wikimedia.org/maniphest/query/CGFQViLShnOY/#R) ⬇️ One task resolved. -1 Oct 2020 (22 of 69 issues https://phabricator.wikimedia.org/maniphest/query/MYnnBybPTYpd/#R) ⬇️ Four tasks resolved. -4 Nov 2020 (9 of 38 issues https://phabricator.wikimedia.org/maniphest/query/CkC_VqQq5VC0/#R) ⬇️ Two tasks resolved. -2 Dec 2020 (11 of 33 issues https://phabricator.wikimedia.org/maniphest/query/10NQy74iKaZJ/#R) ⬇️ One task resolved. -1 Jan 2021 (4 of 50 issues https://phabricator.wikimedia.org/maniphest/query/WIP9W8q54HB6/#R) ⬇️ One task resolved. -1 Feb 2021 (9 of 20 issues https://phabricator.wikimedia.org/maniphest/query/5MzPJAb5oJgv/#R) ⬇️ Two tasks resolved. -2 *Mar 2021* (34 of 48 issues https://phabricator.wikimedia.org/maniphest/query/RsVPep46KRY4/#R) 34 new tasks survived and remain unresolved. +48; -14

------- Tally 138 issues open, as of Excellence #29 https://phabricator.wikimedia.org/phame/post/view/228/production_excellence_29_february_2021/ (6 Mar 2021). -33 issues closed, of the previous 138 open issues. +34 new issues that survived March 2021. 139 issues open, as of today (2 Apr 2021). -------Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Footnotes:

Incident status, Wikitech https://wikitech.wikimedia.org/wiki/Incident_status. Wikimedia incident stats by Krinkle, CodePen https://codepen.io/Krinkle/full/wbYMZK. Production Excellence: Month-over-month spreadsheet and plot https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml . Report charts for Wikimedia-production-error project, Phabricator https://phabricator.wikimedia.org/project/reports/1055/.