How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents We've had quite the month, with 8 documented incidents. That's more than double the two-year median of three a month (Incident graphs https://codepen.io/Krinkle/full/wbYMZK).

2022-03-01 ulsfo network https://wikitech.wikimedia.org/wiki/Incidents/2022-03-01_ulsfo_network Impact: For 20 minutes, clients normally routed to Ulsfo were unable to reach our projects. This includes New Zealand, parts of Canada, and the United States west coast.

2022-03-04 esams availability banner sampling https://wikitech.wikimedia.org/wiki/Incidents/2022-03-04_esams_availability_banner_sampling Impact: For 1.5 hours, all wikis were largely unreachable from Europe (via Esams), with more limited impact across the globe via other data centers as well.

2022-03-06 wdqs-categories https://wikitech.wikimedia.org/wiki/Incidents/2022-03-06_wdqs-categories Impact: For 1.5 hours, some requests to the public Wikidata Query Service API were sporadically blocked.

2022-03-10 site availability https://wikitech.wikimedia.org/wiki/Incidents/2022-03-10_MediaWiki_availability Impact: For 12 min, all wikis were unreachable to logged-in users, and to unregistered users trying to access uncached content.

2022-03-27 api https://wikitech.wikimedia.org/wiki/Incidents/2022-03-27_api Impact: For ~4 hours, in three segments of 1-2 hours each over two days, there were higher levels of failed or slow MediaWiki API requests.

2022-03-27 wdqs outage https://wikitech.wikimedia.org/wiki/Incidents/2022-03-27_wdqs_outage Impact: For 30 minutes, all WDQS queries failed due to an internal deadlock.

2022-03-29 network https://wikitech.wikimedia.org/wiki/Incidents/2022-03-29_network Impact: For approximately 5 minutes, Wikipedia and other Wikimedia sites were slow or inaccessible for many users, mostly in Europe/Africa/Asia. (Details not public at this time.)

2022-03-31 api errors https://wikitech.wikimedia.org/wiki/Incidents/2022-03-31_api_errors Impact: For 22 minutes, API server and app server availability were slightly decreased (~0.1% errors, all for s7-hosted wikis such as Spanish Wikipedia), and the latency of API servers was elevated as well.

Incident follow-up

Remember to review and schedule Incident Follow-up (Sustainability) https://phabricator.wikimedia.org/project/view/4758/ in Phabricator, which are preventive measures and tech debt mitigations written down after an incident is concluded. Read more about past incidents at Incident status https://wikitech.wikimedia.org/wiki/Incident_status on Wikitech. Some recently completed sustainability work:

Add linecard diversity to router-to-router interconnect at Codfw https://phabricator.wikimedia.org/T248506 Filed by Chris (SRE Infra) in 2020 after an incident where all hosts in the Codfw data center lost connectivity at once. Completed by Arzhel and Cathal (SRE Infra), and Papaul (DC Ops); including in Esams where the same issue existed.

Expand parser tests to cover language conversation variants in table-of-contents output https://phabricator.wikimedia.org/T295187 Suggested and carried out by CScott (Parsoid) after reviewing an incident in November. The TOC on wikis that rely on the LanguageConverter service (such as Chinese Wikipedia) were no longer localized

Fix unquoted URL parameters in Icgina health checks https://phabricator.wikimedia.org/T304323 Suggested by Riccardo (SRE Infra) in response to an early warning signal for TLS certificate expiry. He realized that automated checks for a related cluster were still claiming to be in good health, when they in fact should have been firing a similar warning. Carried out by Filippo and Dzahn.

Provide automation to quickly show replication status when primary is down https://phabricator.wikimedia.org/T281249 Filed in April by Jaime (SRE Data Persistence), carried out by John and Ladsgroup.

Trends

Since the last edition, we resolved 24 of the 301 unresolved errors that carried over from previous months. In March, we created 54 new production errors https://phabricator.wikimedia.org/maniphest/query/ryOkF_JP6cV1/#R. That's quite high compared to the twenty-odd reports we find most months. Of these, 17 remain open today a month later.

In the month of April, so far, we reported 20 new errors https://phabricator.wikimedia.org/maniphest/query/1LEA6jQzf7iU/#R of which also 17 remain open today.

The production error workboard once again adds up to exactly 298 open tasks (spreadsheet https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml).

Take a look at the workboard and look for tasks that could use your help. → https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Thanks! Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

🔗 Share or read later via https://phabricator.wikimedia.org/phame/post/view/283/