📘 Read on Phabricator at
https://phabricator.wikimedia.org/phame/live/1/post/163/
-------
How’d we do in our strive for operational excellence last month? Read on to
find out!
## 📊 Month in numbers
* ⚠️11 documented incidents. [1]
* 39 new Wikimedia-prod-error reports. [2]
* 25 Wikimedia-prod-error reports closed. [3]
The number of incidents in June was high compared to previous years. At 11
incidents, this is higher than this year’s median (5), the 2018 median (4),
and the 2017 median (5). It is also higher than any month of June in the
last 4 years. – More data at <https://codepen.io/Krinkle/full/wbYMZK>.
To read more about these incidents, their investigations, and pending
actionables; check <
https://wikitech.wikimedia.org/wiki/Incident_documentation#2019>gt;.
There are currently 204 open Wikimedia-prod-error reports (up from 186 in
April, and 201 in May). [4]
-------
## *️⃣ [Op-ed] Integrated maintenance cost
Hereby a shoutout to the Wikidata and Core Platform teams, at WMDE and WMF
respectively. They both recently established a rotating subteam that
focuses on incidental work. Such as maintenance, and other work that might
otherwise hinder feature development.
I expect this to improve efficiency by avoiding context switches between
feature and incidental work. The rotational aspect should distribute the
work more evenly among team members (avoiding burnout). And, it may
increase exposure to other teams, and lesser-known areas of our code; which
provide opportunities for personal growth and to retain institutional
knowledge.
-------
## 📉 Current problems
Take a look at the workboard and look for tasks that might need your help.
The workboard lists known issues, grouped by the month in which they were
first observed.
→
https://phabricator.wikimedia.org/tag/wikimedia-production-error/
Breakdown of recent months (past two weeks not included):
* November: 1 issue got fixed! (1 issue left).
* December: 3 issues left (unchanged). ⚠️
* January: 1 issue left (unchanged). ⚠️
* February: 2 issues left (unchanged). ⚠️
* March: 4 issues left (unchanged). ⚠️
* April: 2 issues got fixed! (10 of 14 issues, that survived April, remain
open).
* May: 4 issues got fixed! (6 of 10 issues, that survived May, are left).
* June: 11 new issues from last month remain unresolved.
By steward and software component, the unresolved issues that survived June:
* CPT / MW Auth (PHP fatal):
https://phabricator.wikimedia.org/T228717
* CPT / MW Actor (DB contention):
https://phabricator.wikimedia.org/T227739
* CPT or Multimedia / Thumb handler (MultiCurl error):
https://phabricator.wikimedia.org/T225197
* Multimedia / File metadata (PHP error):
https://phabricator.wikimedia.org/T226751
* Wikidata / Commons page view (PHP fatal):
https://phabricator.wikimedia.org/T227360
* Wikidata / Jobrunner (PHP memory fatal):
https://phabricator.wikimedia.org/T227450
* Wikidata / Jobrunner (Trx error):
https://phabricator.wikimedia.org/T225098
* Product-Infra / ReadingList API (PHP fatal):
https://phabricator.wikimedia.org/T226593
* (Unknown?) / Special:ConfirmEmail (PHP fatal):
https://phabricator.wikimedia.org/T226337
* (Unknown?) / Page renaming (DB timeout):
https://phabricator.wikimedia.org/T226898
* (Unknown?) / Page renaming (Bad revision fatal):
https://phabricator.wikimedia.org/T225366
-------
## 🎉 Thanks!
Thank you to everyone who has helped by reporting, investigating, or
resolving problems in Wikimedia production. Including: Brad Jorsch, Brion
Vibber, Roan Kattouw, CScott, Daniel Kinzler, David Causse, DerFussi,
Ebe123, Filippo Giunchedi, James Forrester, Kosta Harlan, Legoktm, Lucas
Werkmeister, Bartosz Dziewoński, Matthias Mullie, Michael Große, Niklas
Laxström, Stephane Bisson, Stas Malyshev, Tchanders, Gergő Tisza, Tpt,
Umherirrender, and Urbanecm.
Thanks!
Until next time,
– Timo Tijhof
🔮 “These are his marbles...” “Ha! He really did lose his marbles, didn't
he?” “Yeah, he lost them good.”
-------------------------------------------------
Footnotes:
[1] Incidents. –
https://wikitech.wikimedia.org/wiki/Special:PrefixIndex?prefix=Incident+doc…
[2] Tasks created. –
https://phabricator.wikimedia.org/maniphest/query/barA9u0RtzE3/#R
[3] Tasks closed. –
https://phabricator.wikimedia.org/maniphest/query/zXZdtEhocO1a/#R
[4] Open tasks. –
https://phabricator.wikimedia.org/maniphest/query/47MGY8BUDvRD/#R