Cloud-admin October 2020

cloud-admin@lists.wikimedia.org

3 participants
3 discussions

General CloudVPS network maintenance on 2020-10-29
by Arturo Borrero Gonzalez 29 Oct '20

29 Oct '20

Hi! There will be a general CloudVPS network maintenance on 20202-10-29, from 16:00 UTC to 17:00 UTC. During the operation window, all cloud services might be intermittently down, inaccessible. This operation affects all CloudVPS projects, including Toolforge, PAWS and Quarry. Services running in the cloud might fail to contact external entities, and connections to ToolsDB, NFS, wiki-replicas or LDAP might be affected as well. In the best case scenario, the changes (and downtime) will be barely noticed. The maintenance consist on introducing new hardware equipment in to the CloudVPS edge network. You can find additional details in Phabricator [0]. regards. [0] https://phabricator.wikimedia.org/T265288 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 2

s4 on labsdb on the analytics role.
by Manuel Arostegui 07 Oct '20

07 Oct '20

Hello, This is a heads up about the current situation with s4 (commons) and labsdb. There's been more activity lately on s4, and that had made labsdb1011 (analytics role) start lagging behind. https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=6&orgId=1&from=no… I have tried to ease its weight a couple of days ago, to help it recovering: https://gerrit.wikimedia.org/r/c/operations/puppet/+/630392 https://gerrit.wikimedia.org/r/c/operations/puppet/+/630531 https://gerrit.wikimedia.org/r/c/operations/puppet/+/630770 The last change has (as sort of expected) made labsdb1010 lag: https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=6&orgId=1&from=no… I am going to decrease the pt-kill query time from 3600 to 1800 to see if that helps labsdb1010 to guard the fort a bit. There's not much else we can do at the moment, but just keep all these issues in mind if people complain about lag on s4 (commons) on the analytics role. The web role is doing fine (labsdb1009 isn't lagging). Manuel.

2 6

unscheduled keystone maintenance
by Arturo Borrero Gonzalez 06 Oct '20

06 Oct '20

Hi there, we need to perform some unscheduled keystone maintenance right now. Authentication to some cloud services, in particular Horizon, might be interrupted during this maintenance period. We expect such maintenance to don't last more than 1h. regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin October 2020