How’d we do in our strive for operational excellence last month? Read on to find out!

Read on Phabricator at https://phabricator.wikimedia.org/phame/post/view/229/

Incidents

2 documented incidents. That's average for this time of year, when we usually had 1-4 incidents.

Learn about recent incidents at Incident status on Wikitech, or Preventive measures in Phabricator.

-------

Trends

In March we made significant progress on the outstanding errors of previous months. Several of the 2020 months are finally starting to empty out. But with over 30 new tasks from March itself remaining, we did not break even, and ended up slightly higher than last month. This could be reversing two positive trends, but I hope not.

Firstly, there was a steep increase in the number of new production errors that were not resolved within the same month. This is counter the positive trend we started in November. The past four months typically saw 10-20 errors outlive their month of discovery, and this past month saw 34 of its 48 new errors remain unresolved.

Secondly, we saw the overall number of unresolved errors increase again. This January began a downward trend for the first time in thirteen months, which continued nicely through February. But, this past month we broke even and even pushed upward by one task. I hope this is just a breather and we can continue our way downward.

Figure 1, Figure 2: Unresolved error reports stacked by month.

-------

Outstanding errors

Take a look at the workboard and look for tasks that could use your help.

Summary over recent months, per spreadsheet:

-------

-------

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Footnotes:

Incident status, Wikitech.
Wikimedia incident stats by Krinkle, CodePen.
Production Excellence: Month-over-month spreadsheet and plot.
Report charts for Wikimedia-production-error project, Phabricator.