How’d we do in our strive for operational excellence last month? Read on to find out!

Read on Phabricator at https://phabricator.wikimedia.org/phame/post/view/235/

Incidents

6 documented incidents. That's above the historical average https://codepen.io/Krinkle/full/wbYMZK of 3–4 per month.

Learn about recent incidents at Incident status https://wikitech.wikimedia.org/wiki/Incident_status on Wikitech, or Preventive measures https://phabricator.wikimedia.org/project/view/4758/ in Phabricator.

-------

Trends

In April, we saw a continuation of the healthy trend that started this January https://phabricator.wikimedia.org/phame/post/view/227/production_excellence_28_january_2021/#trends — a trend where the back of the line is moving forward at least as quickly as the front of the line. We did take a little breather in March https://phabricator.wikimedia.org/phame/post/view/229/production_excellence_30_march_2021/#trends where we almost broke even, but otherwise the trend is going well.

Last month we bade farewell to the production errors we found in July 2019. This month we cleared out the column for October 2019.

One point of concern is that we did encounter a high number of new production errors — are errors that we failed to catch during development, code review, continuous integration, beta testing, or pre-deployment checks. Where we used to discover about a dozen of those a month, whereas we found 42 during this month. As of writing, 17 of the 42 April-discovered errors have been resolved.

The "Old" column (generally tracking pre-2019 tasks) grew for the first time in six months. This increase can largely be attributed to improved telemetry of client-side errors uncovering issues in under-resourced products, such as the old Kaltura video player.

Figure 1, Figure 2: Unresolved error reports stacked by month. https://phabricator.wikimedia.org/phame/post/view/235/production_excellence_31_april_2021/#trends

-------

Outstanding errors

Take a look at the workboard https://phabricator.wikimedia.org/tag/wikimedia-production-error/ and look for tasks that could use your help.

Summary over recent months, per spreadsheet https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml : Aug 2019 (1 of 14 left) ⚠️ Unchanged (over one year old). Oct 2019 (0 of 12 left) ✅ Last three tasks resolved! -3 Jan 2020 (1 of 7 left) ⚠️ Unchanged (over one year old). Mar 2020 (2 of 2 left) ⚠️ Unchanged (over one year old). Apr 2020 (5 of 14 left) ⚠️ Unchanged (over one year old). May 2020 (5 of 14 left) ⏸ — Jun 2020 (5 of 14 left) ⬇️ One task resolved. -1 Jul 2020 (4 of 24 issues https://phabricator.wikimedia.org/maniphest/query/s__D8Kd0xuQH/#R) ⬇️ One task resolved. -1 Aug 2020 (13 of 53 issues https://phabricator.wikimedia.org/maniphest/query/hu1yhWu4sXkP/#R) ⬇️ Two tasks resolved. -2 Sep 2020 (7 of 33 issues https://phabricator.wikimedia.org/maniphest/query/CGFQViLShnOY/#R) ⏸ — Oct 2020 (20 of 69 issues https://phabricator.wikimedia.org/maniphest/query/MYnnBybPTYpd/#R) ⬇️ Two tasks resolved. -2 Nov 2020 (9 of 38 issues https://phabricator.wikimedia.org/maniphest/query/CkC_VqQq5VC0/#R) ⏸ — Dec 2020 (7 of 33 issues https://phabricator.wikimedia.org/maniphest/query/10NQy74iKaZJ/#R) ⬇️ Four tasks resolved. -4 Jan 2021 (1 of 50 issues https://phabricator.wikimedia.org/maniphest/query/WIP9W8q54HB6/#R) ⬇️ One task resolved. -1 Feb 2021 (8 of 20 issues https://phabricator.wikimedia.org/maniphest/query/5MzPJAb5oJgv/#R) ⬇️ One task resolved. -1 Mar 2021 (18 of 48 issues https://phabricator.wikimedia.org/maniphest/query/RsVPep46KRY4/#R) ⬇️ Sixteen tasks resolved. -16 *Apr 2021* (25 of 42 issues https://phabricator.wikimedia.org/maniphest/query/rYyMt_gYYymb/#R) 42 new issues found, of which 25 remained open. +42; -17 ------- Tally 139 issues open, as of Excellence #30 https://phabricator.wikimedia.org/phame/post/view/229/production_excellence_30_march_2021/ (March 2021). -31 issues closed, of the previously open issues. +25 new issues that survived April 2021. 133 issues open, as of today (12 May 2021). -------

Thanks

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof