How’d we do in our strive for operational excellence last month? Incidents

3 documented incidents. That's lower than June in the previous five years where the month saw 5-9 incidents. I've added a new panel ⭐️ to the Incident statistics https://codepen.io/Krinkle/full/wbYMZK tool. This one plots monthly statistics on top of previous years, to more easily compare them.

Learn more from the Incident documents https://wikitech.wikimedia.org/wiki/Incident_status on Wikitech, and remember to review and schedule Incident Follow-up https://phabricator.wikimedia.org/project/view/4758/ in Phabricator, which are preventive measures and other action items filed after an incident.

------- Trends

In June, work on production errors appears to have stagnated a bit. Or more precisely, the work only resulted in relatively few tasks being resolved. 15 of the 26 new tasks are still open as of writing.

Of the tasks from previous months, only 11 were resolved, leaving most columns unchanged. See the table further down for a more detailed breakdown and links to Phabricator queries for the tasks in question.

With the 15 remaining new tasks, and the 11 tasks resolved from our backlog, this raises the chart from 150 to 154 tasks.

Figure 1, Figure 2: Unresolved error reports stacked by month. https://phabricator.wikimedia.org/phame/post/view/240/production_excellence_33_june_2021/#trends

Month-over-month plots based on spreadsheet data https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml .

------- Outstanding errors

Take a look at the workboard and look for tasks that could use your help. → https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months: Jan 2020 (1 of 7 left) ⚠️ Unchanged (over one year old). Mar 2020 (2 of 2 left) ⚠️ Unchanged (over one year old). Apr 2020 (4 of 14 left) ⚠️ Unchanged (over one year old). May 2020 (5 of 14 left) ⚠️ Unchanged (over one year old). Jun 2020 (5 of 14 left) ⚠️ Unchanged (over one year old). Jul 2020 (4 of 24 issues) ⚠️ Unchanged (over one year old). Aug 2020 (11 of 53 issues) ⬇️ One task resolved. -1 Sep 2020 (7 of 33 issues) ⚠️ Unchanged (over one year old). Oct 2020 (19 of 69 issues) ⚠️ Unchanged (over one year old). Nov 2020 (8 of 38 issues) ⚠️ Unchanged (over one year old). Dec 2020 (7 of 33 issues) ⚠️ Unchanged (over one year old). Jan 2021 (3 of 50 issues https://phabricator.wikimedia.org/maniphest/query/WIP9W8q54HB6/#R) ⚠️ Unchanged (over one year old). Feb 2021 (6 of 20 issues https://phabricator.wikimedia.org/maniphest/query/5MzPJAb5oJgv/#R) ⬇️ One task resolved. -1 Mar 2021 (13 of 48 issues https://phabricator.wikimedia.org/maniphest/query/RsVPep46KRY4/#R) ⬇️ One task resolved. -1 Apr 2021 (19 of 42 issues https://phabricator.wikimedia.org/maniphest/query/rYyMt_gYYymb/#R) ⬇️ Four tasks resolved. -4 May 2021 (25 of 54 issues https://phabricator.wikimedia.org/maniphest/query/tmkGqt0C93YG/#R) ⬇️ Four tasks resolved. -4 June 2021 (15 of 26 issues https://phabricator.wikimedia.org/maniphest/query/roL0TaxtcaLQ/#R) 📌 26 new issues, of which 11 were closed. +26, -11

------- Tally 150 issues open, as of Excellence #32 (May 2021) https://phabricator.wikimedia.org/phame/post/view/236/production_excellence_32_may_2021/ . -11 issues closed, of the previous 150 open issues. +15 new issues that survived June 2021. 154 issues open as of yesterday.

------- Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

🔗 Share or read later via https://phabricator.wikimedia.org/phame/post/view/240/