How’d we do in our strive for operational excellence last month? Read on to find out!
Incidents
We've had quite the month, with 8 documented incidents. That's more than double
the two-year median of three a month (Incident graphs
<https://codepen.io/Krinkle/full/wbYMZK>).
2022-03-01 ulsfo network
<https://wikitech.wikimedia.org/wiki/Incidents/2022-03-01_ulsfo_network>
Impact: For 20 minutes, clients normally routed to Ulsfo were unable to reach our
projects. This includes New Zealand, parts of Canada, and the United States west coast.
2022-03-04 esams availability banner sampling
<https://wikitech.wikimedia.org/wiki/Incidents/2022-03-04_esams_availability_banner_sampling>
Impact: For 1.5 hours, all wikis were largely unreachable from Europe (via Esams), with
more limited impact across the globe via other data centers as well.
2022-03-06 wdqs-categories
<https://wikitech.wikimedia.org/wiki/Incidents/2022-03-06_wdqs-categories>
Impact: For 1.5 hours, some requests to the public Wikidata Query Service API were
sporadically blocked.
2022-03-10 site availability
<https://wikitech.wikimedia.org/wiki/Incidents/2022-03-10_MediaWiki_availability>
Impact: For 12 min, all wikis were unreachable to logged-in users, and to unregistered
users trying to access uncached content.
2022-03-27 api <https://wikitech.wikimedia.org/wiki/Incidents/2022-03-27_api>
Impact: For ~4 hours, in three segments of 1-2 hours each over two days, there were higher
levels of failed or slow MediaWiki API requests.
2022-03-27 wdqs outage
<https://wikitech.wikimedia.org/wiki/Incidents/2022-03-27_wdqs_outage>
Impact: For 30 minutes, all WDQS queries failed due to an internal deadlock.
2022-03-29 network
<https://wikitech.wikimedia.org/wiki/Incidents/2022-03-29_network>
Impact: For approximately 5 minutes, Wikipedia and other Wikimedia sites were slow or
inaccessible for many users, mostly in Europe/Africa/Asia. (Details not public at this
time.)
2022-03-31 api errors
<https://wikitech.wikimedia.org/wiki/Incidents/2022-03-31_api_errors>
Impact: For 22 minutes, API server and app server availability were slightly decreased
(~0.1% errors, all for s7-hosted wikis such as Spanish Wikipedia), and the latency of API
servers was elevated as well.
Incident follow-up
Remember to review and schedule Incident Follow-up (Sustainability)
<https://phabricator.wikimedia.org/project/view/4758/> in Phabricator, which are
preventive measures and tech debt mitigations written down after an incident is concluded.
Read more about past incidents at Incident status
<https://wikitech.wikimedia.org/wiki/Incident_status> on Wikitech. Some recently
completed sustainability work:
Add linecard diversity to router-to-router interconnect at Codfw
<https://phabricator.wikimedia.org/T248506>
Filed by Chris (SRE Infra) in 2020 after an incident where all hosts in the Codfw data
center lost connectivity at once. Completed by Arzhel and Cathal (SRE Infra), and Papaul
(DC Ops); including in Esams where the same issue existed.
Expand parser tests to cover language conversation variants in table-of-contents output
<https://phabricator.wikimedia.org/T295187>
Suggested and carried out by CScott (Parsoid) after reviewing an incident in November. The
TOC on wikis that rely on the LanguageConverter service (such as Chinese Wikipedia) were
no longer localized
Fix unquoted URL parameters in Icgina health checks
<https://phabricator.wikimedia.org/T304323>
Suggested by Riccardo (SRE Infra) in response to an early warning signal for TLS
certificate expiry. He realized that automated checks for a related cluster were still
claiming to be in good health, when they in fact should have been firing a similar
warning. Carried out by Filippo and Dzahn.
Provide automation to quickly show replication status when primary is down
<https://phabricator.wikimedia.org/T281249>
Filed in April by Jaime (SRE Data Persistence), carried out by John and Ladsgroup.
Trends
Since the last edition, we resolved 24 of the 301 unresolved errors that carried over from
previous months.
In March, we created 54 new production errors
<https://phabricator.wikimedia.org/maniphest/query/ryOkF_JP6cV1/#R>. That's
quite high compared to the twenty-odd reports we find most months. Of these, 17 remain
open today a month later.
In the month of April, so far, we reported 20 new errors
<https://phabricator.wikimedia.org/maniphest/query/1LEA6jQzf7iU/#R> of which also 17
remain open today.
The production error workboard once again adds up to exactly 298 open tasks (spreadsheet
<https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml>).
Take a look at the workboard and look for tasks that could use your help.
→
https://phabricator.wikimedia.org/tag/wikimedia-production-error/
Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving problems in
Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
🔗 Share or read later via
https://phabricator.wikimedia.org/phame/post/view/283/