Small mistake in the above email, on-call is Cathal and Eoghan
On Tue, Aug 22, 2023 at 11:09 AM Clément Goubert cgoubert@wikimedia.org wrote:
Hi y'all,
I'm about to start sending 2% of global traffic to mw-on-k8s https://phabricator.wikimedia.org/T341780#9108656 On-call has been made aware (it's me and slyngs), emergency rollback is:
- Revert https://gerrit.wikimedia.org/r/c/operations/puppet/+/951131
- sudo cumin -b 20 'A:cp-text and P{P:trafficserver::backend}'
'run-puppet-agent -q'
If you want to see what happens, you can check out the mw-on-k8s dashboard https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s This will impact the mw-web and mw-api-ext deployments.
Cheers,
-- Clément Goubert (they/them) Senior SRE Wikimedia Foundation