📘 Read on Phabricator at https://phabricator.wikimedia.org/phame/post/view/203/ -------

How’d we do in our strive for operational excellence last month? Read on to find out!

## 📈 Month in review

* 4 documented incidents in June. [1] * 37 new production errors were filed and 27 were closed. [2] [3] * 72 recent production errors still open (up from 68). * 203 total Wikimedia-prod-error tasks currently open (up from 192). [4]

For more about recent incidents and pending actionables see Wikitech and Phabricator, at https://wikitech.wikimedia.org/wiki/Incident_documentation and https://phabricator.wikimedia.org/project/view/4758/

-------

## 📖 Outstanding errors

Breakdown of the errors reported in June that are still open today:

* (Needs owner) / Newsletter extension: Unexpected locking SELECT query. https://phabricator.wikimedia.org/T253926 * (Needs owner) / FlaggedRevs extension: Unable to submit review of page due to bad fr_page_id record. https://phabricator.wikimedia.org/T256296 * Editing team / MassMessage extension: Delivery fails due to system user conflict. https://phabricator.wikimedia.org/T171003 * Parsing team / Parsoid: Pagebundle data unavailable due to a bad UTF-8 string. https://phabricator.wikimedia.org/T236866 * Growth team / Recent changes: Update for ActiveUsers data failing due to deadlock. https://phabricator.wikimedia.org/T255059 * Growth team / GrowthExperiments: Issue with question display on personal homepage. https://phabricator.wikimedia.org/T255616 * Language team / Translate extension: Update jobs fail due to invalid function call. https://phabricator.wikimedia.org/T255669 * Language team / ContentTranslation: Save action fails due to duplicate insert query. https://phabricator.wikimedia.org/T256230 * Core Platform team / Content handling: Incompatible content type during content merge/stash. https://phabricator.wikimedia.org/T255700 * Core Platform team / Monolog: API usage logs and error logs sometimes missing due to socket failure. https://phabricator.wikimedia.org/T255578 * Search Platform team / WikibaseCirrus: Elevated error levels from EntitySearchElastic warnings. https://phabricator.wikimedia.org/T255658 * Wikidata / API: Generator query fails due to invalid API result format. https://phabricator.wikimedia.org/T254334 * Wikidata / API: EntityData query emits warning about bad RDF. https://phabricator.wikimedia.org/T255054 * Wikidata / Repo: Entity relation update jobs fail due to deadlock. https://phabricator.wikimedia.org/T255706

-------

## 📊 Trends

Take a look at the workboard and look for tasks that could use your help. → https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

* July 2019 (5 of 18 tasks left): Two tasks closed. * August (1 of 14 tasks left): Another task closed, only one remaining! 🚀 * September (5 of 12 tasks left): Two tasks closed. * October (6 of 12 tasks left), no change. * November (3 of 5 tasks left): Another task closed. * December (5 of 9 tasks left), no change. * January 2020 (5 of 7 tasks lef), no change. * February (4 of 7 tasks left), no change. * March (2 of 2 tasks left), no change. * April (11 of 14 tasks left): Three tasks closed. * May (11 tasks left): Three tasks closed. * June: 14 new tasks survived the month of June. ⚠️

At the end of May the number of open production errors over recent months was 68. Of those, 10 got closed, but with 14 new tasks from June still open, the total has grown further to 72.

The workboard had 192 open tasks last month, which saw another increase, to now 203 open tasks (this includes tasks from 2019 and earlier).

-------

## 🎉 Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

-------

Footnotes:

[1] Incidents. – https://wikitech.wikimedia.org/wiki/Incident_documentation#2020

[2] Tasks created. – https://phabricator.wikimedia.org/maniphest/query/VTpmvaJLYVL1/#R

[3] Tasks closed. – https://phabricator.wikimedia.org/maniphest/query/qn5yeURqyl3D/#R

[4] Open tasks. – https://phabricator.wikimedia.org/maniphest/query/Fw3RdXt1Sdxp/#R