How’d we do in our strive for operational excellence last month? Read on to find out!

Read on Phabricator at https://phabricator.wikimedia.org/phame/post/view/236/

Incidents

Zero incidents recorded in the past month. Yay! That's only five months after November 2020, the last month without documented incidents (Incident stats).

Remember to review Preventive measures in Phabricator, which are action items filed after an incident.

-------

Trends

In May, we unfortunately saw a repeat of the worrying pattern we saw in April, but with higher numbers. We found 54 new errors. This is the most new errors in a single month, since the Excellence monthly began three years ago in 2018. About half of these (29 of 54) remain unresolved as of writing, two weeks into the following month.

Figure 1, Figure 2: Unresolved error reports stacked by month.

Month-over-month plots based on spreadsheet data.

-------

New errors in May

Below is a snapshot of just the 54 new issues found last month, listed by their code steward.

Be mindful that the reporting of errors is not itself a negative point per-se. I think it should be celebrated when teams have good telemetry, detect their issues early, and address them within their development cycle. It might be more worrisome when teams lack telemetry or time to find such issues, or can't keep up with the pace at which issues are found.

Take a look at the workboard and look for tasks that could use your help.
→  https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

-------

Tally
133	issues open, as of Excellence #31 (12 May 2021).
-12	issues closed, of the previous 133 open issues.
+29	new issues that survived May 2021.
150	issues open, as of today (12 June 2021).

-------

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof