How’d we do in our strive for operational excellence last month? Read on to find out!

Read on Phabricator at https://phabricator.wikimedia.org/phame/post/view/228

📈 Incidents

3 documented incidents last month, [1] which is average for the time of year. [2]

Learn about these incidents at Incident status https://wikitech.wikimedia.org/wiki/Incident_status on Wikitech, and their Preventive measures https://phabricator.wikimedia.org/project/view/4758/ in Phabricator.

For those with NDA-restricted access, there may be additional private incident reports https://drive.google.com/drive/u/0/folders/1d2hlER4s_GPe0TWDqNfa2RQcYFuOdhOo available.

💡 *Did you know: Our Incident reports have switched to using the ISO date format in their titles and listings, for improved readability and edit-ability (esp. when publishing on a later date). So long 202010221, and hello 2021-02-21!*

------- 📊 Trends

In February we saw a continuation of the new downward trend that began this January https://phabricator.wikimedia.org/phame/post/view/227/production_excellence_28_january_2021/, which came after twelve months of continued rising. Let's make sure this trend sticks with us as we work our way through the debt, whilst also learning to have a healthy week-to-week iteration where we monitor and follow-up on any new developments such that they don't introduce lasting regressions.

The recent tally (issues filed since we started reporting in March 2019) is down to 138 unresolved errors, from 152 last month. The old backlog (pre-2019 issues) also continued its 5-month streak and is down to 148, from 160 last month. If this progress continues we'll soon have fewer "Old" issues than "Recent" issues, and possibly by the start of 2022 we may be able to report and focus only on our rotation through recent issues as hopefully we are then balancing our work such that issues reported this month are addressed mostly in the same month or otherwise later that quarter within 2-3 months. Visually that would manifest as the colored chunks having a short life on the chart with each drawn at a sharp downwards angle – instead of dragged out where it was building up an ever-taller shortcake. I do like cake, but I prefer the kind I can eat. 🍰

Month-over-month plots based on spreadsheet data https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml. [3] [4]

Figure 1, Figure 2: Unresolved error reports stacked by month. https://phabricator.wikimedia.org/phame/post/view/228/production_excellence_29_february_2021/#trends

-------

📖 Outstanding errors

Summary over recent months:

- ⚠️ July 2019 (2 of 18 issues left): *no change*. - ⚠️ August 2019 (1 of 14 issues): *no change*. - ⚠️ October 2019 (4 of 12 issues): *no change*. - ⚠️ November 2019 (1 of 5 issues): *no change*. - ⚠️ December 2019 (1 of 9 issues): One task resolved (-1). - ⚠️ January 2020 (2 of 7 issues): *no change*. - ⚠️ February 2020 (1 of 7 issues): *no change*. - ⚠️ March 2020 (2 of 2 issues): *no change*. - April 2020 (9 of 14 issues left): *no change*. - May 2020 (6 of 14 issues left): *no change*. - June 2020 (7 of 14 issues left): *no change*. - July 2020 (9 of 24 new issues https://phabricator.wikimedia.org/maniphest/query/s__D8Kd0xuQH/#R): *no change*. - August 2020 (20 of 53 new issues https://phabricator.wikimedia.org/maniphest/query/hu1yhWu4sXkP/#R): Two tasks resolved (-2). - September 2020 (9 of 33 new issues https://phabricator.wikimedia.org/maniphest/query/CGFQViLShnOY/#R): Five tasks resolved (-5). - October 2020 (26 of 69 new issues https://phabricator.wikimedia.org/maniphest/query/MYnnBybPTYpd/#R): Five tasks resolved (-5). - November 2020 (11 of 38 new issues https://phabricator.wikimedia.org/maniphest/query/CkC_VqQq5VC0/#R): Three tasks resolved (-3). - December 2020 (12 of 33 new issues https://phabricator.wikimedia.org/maniphest/query/10NQy74iKaZJ/#R): Seven tasks resolved (-7). - January 2021 (5 of 50 new issues https://phabricator.wikimedia.org/maniphest/query/WIP9W8q54HB6/#R): Two tasks resolved (-2). - *February 2021*: 11 of 20 new issues https://phabricator.wikimedia.org/maniphest/query/5MzPJAb5oJgv/#R survived the month and remained unresolved (+20; -9)

Recent tally 152 issues open, as of Excellence #28 https://phabricator.wikimedia.org/phame/post/view/227/production_excellence_28_january_2021/ (16 Feb 2021). -25 issues closed since, of the previous 152 open issues. +11 new issues that survived Feb 2021. 138 issues open, as of today 5 Mar 2021.

For the on-going month of March 2021, we've got 12 new issues https://phabricator.wikimedia.org/maniphest/query/aDItN1Nm0ron/#R so far.

Take a look at the workboard https://phabricator.wikimedia.org/tag/wikimedia-production-error/ and look for tasks that could use your help!

------- 🎉 Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

-------

Footnotes:

[1] Incident status Wikitech https://wikitech.wikimedia.org/wiki/Incident_status. [2] Wikimedia incident stats by Krinkle, CodePen https://codepen.io/Krinkle/full/wbYMZK. [3] Month-over-month, Production Excellence spreadsheet https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml . [4] Open tasks, Wikimedia-prod-error, Phabricator https://phabricator.wikimedia.org/maniphest/query/Fw3RdXt1Sdxp/#R.