- Cloud-admin - lists.wikimedia.org

(DRAFT) The future of /srv
by Andrew Bogott 10 Mar '21

10 Mar '21

Hello all! I've been thinking more about how to move to a future of quota-managed storage, and have written a few scraps of puppet code to support us moving off of the old labs::lvm roles. Attached below is a draft of an announcement email explaining the changes; writing this was the easiest way for me to think through what the changes would feel like to our users. For context: As best I can tell the only automated uses of our lvm puppet classes are via profile::labs::lvm::srv. So, while my announcement draft tries to acknowledge a diversity of uses cases I think there's really only one, which is "mount all extra space in /srv". That workflow will be replaced with a similar but slightly more cumbersome "attach a new cinder volume and then mount all the space on that volume in /srv/" A quick grep suggests that the most affected users workers will be people building new VMs for toolforge, CI, and Quarry. I'm not feeling at all bad about asking those folks to edit their puppet classes but I am feeling kind of bad about adding a manual per-VM horizon step; perhaps someday we'll have instrumentation for that. I welcome your thoughts about both the wording of this email and the overall plan that it implies. Thanks! -A ==== With Cinder storage[0] now available and well-tested on cloud-vps, it's time to move forward to widespread adoption. As of today, use of lvm (e.g. via role::labs::lvm::srv) is deprecated in favor of cinder volumes for all local storage outside of the initial 20GB allocated for for standard OS functions. In support of this move we'll be making several technical changes to our infrastructure over the coming days and weeks: - Default per-project cinder usage quotas will be increased from 10GB to 80GB - Instance flavors with variable disk sizes will be removed. Going forward all flavors will have a default local storage size of 20GB regardless of associated cores or RAM. - The old lvm puppet roles will continue to function as before on legacy VMs. On new VMs this role will detect the lack of available space for partitioning and report failures[1]. - Instance resizing will be enabled in the Horizon UI. This will allow you to adjust the allocated cores and RAM allocated to a given VM without rebuilding. Manual setup of Cinder volumes is documented here: https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_ins… If your workflow involves puppet roles that depend on labs_lvm, you will need to alter both your puppet code and your manual processes. I've provided some helper roles to assist. The simplest[2] is the new 'role::labs::cindermount::srv' which in most cases can be used as a drop-in replacement for role::labs::lvm::srv: once a cinder volume is attached to a VM that role will detect, format, and mount it as /srv. The base class cinderutils::ensure[3] can be reused for other variations on this theme. Please don't hesitate to contact me or other WMCS staff for assistance with puppet patches! In some cases the default quota of 80GB will be insufficient for a full migration to cinder from the old flavor model, especially if you have made use of the old 'large' or 'xlarge' sizes. When it comes time to to rebuild these larger VMs, please open a quota request ticket and detail what VMs (of what flavors) will be deleted and we will try to be quick and generous with quota expansions to assist in migration. [0] https://techblog.wikimedia.org/2021/02/05/cinder-on-cloud-vps/ [1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668567 [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/669958 [3] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668757

2 1

Network change for VMs contacting NFS dumps
by Arturo Borrero Gonzalez 24 Feb '21

24 Feb '21

Hello, today 2021-02-23 in about ~30 minutes (16:00 UTC) we will change how virtual machine instances running in Cloud VPS contact NFS dump servers [0]. There is no action required on your side. We anticipate little to no impact as a result of the network changes. But in case you notice something is not properly working with dumps NFS in Cloud VPS (or Toolforge) please contact us [1] as soon as possible. The relevant phabricator ticket [2] is T272397. regards. [0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Dumps [1] https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction#Commun… [2] https://phabricator.wikimedia.org/T272397 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

Change to how Cloud VPS and Toolforge contact Wikis
by Arturo Borrero Gonzalez 02 Feb '21

02 Feb '21

Hello, we are planning to change how Cloud VPS instances and Toolforge tools contact WMF-hosted wikis, in particular the source IP address for the network connection. The new IP address that wikis will see is 185.15.56.1. The change is scheduled to go live on 2021-02-08. More detailed information in wikitech: https://wikitech.wikimedia.org/wiki/News/CloudVPS_NAT_wikis If you are a Cloud VPS user or Toolforge developer, check your tools after that date to make sure they are properly running. If you detect a block, a rate-limit or similar, please let us know. If you are a WMF SRE or engineer involved with the wikis, be informed that this address could generate a significant traffic volume, perhaps about 30%-40% total wiki edits. We are trying to smooth the change as much as possible, so please send your feedback if you think there is something we didn't account for yet. Thanks, best regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

Toolforge kubernetes maintenance today 2020-12-10 @ 15:30 UTC
by Arturo Borrero Gonzalez 12 Dec '20

12 Dec '20

Hi there! Today 2020-12-10 @ 15:30 UTC we will perform an upgrade of the Toolforge kubernetes cluster [0]. We don't expect any major disruption of the service, but we detected in past upgrades that some components might be restarted, causing brief interruptions of network flows. Given the amount of worker nodes we have, more than 50, the operation will take us at least a couple of hours. Tools maintainers: you don't have to do anything during this operation, but if you detect anything weird please contact us either in the phabricator task [0], in the IRC channel #wikimedia-cloud or in the cloud(a)lists.wikimedia.org [1] mailing list. regards. [0] https://phabricator.wikimedia.org/T263284 [1] https://lists.wikimedia.org/mailman/listinfo/cloud -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

3 3

PAWS kubernetes upgrade
by Arturo Borrero Gonzalez 01 Dec '20

01 Dec '20

Hi there, we are about to upgrade the kubernetes version that runs PAWS, from 1.6 to 1.17. We don't expect any interruptions major on the service, perhaps only some hiccups when pods are restarted/rescheduled. More information is available in this phabricator ticket: https://phabricator.wikimedia.org/T268669 The operation may take something between 30 minutes and 1 hours, and we are starting soon after I finish sending this email. Please, ping us if you see anything wrong. regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

General CloudVPS network maintenance on 2020-11-09 (today)
by Arturo Borrero Gonzalez 09 Nov '20

09 Nov '20

Hi!, There will be a general CloudVPS network maintenance on 2020-10-09 @ 12:30 UTC. The operation window will last for 1h. During the operation, all cloud services will be inaccessible or intermittently down. This operation affects all CloudVPS projects, including Toolforge, PAWS and Quarry. Services running in the cloud might fail to contact external entities, and connections to ToolsDB, NFS, wiki-replicas or LDAP will be affected as well. The operation we are doing today is a followup to what we did two weeks ago [0], and involves changing the IP addressing of the network that connects the CloudVPS network to the internet. Sorry for the short notice, we couldn't avoid scheduling this to today. regards. [0] https://phabricator.wikimedia.org/T265288 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

General CloudVPS network maintenance on 2020-10-29
by Arturo Borrero Gonzalez 30 Oct '20

30 Oct '20

Hi! There will be a general CloudVPS network maintenance on 20202-10-29, from 16:00 UTC to 17:00 UTC. During the operation window, all cloud services might be intermittently down, inaccessible. This operation affects all CloudVPS projects, including Toolforge, PAWS and Quarry. Services running in the cloud might fail to contact external entities, and connections to ToolsDB, NFS, wiki-replicas or LDAP might be affected as well. In the best case scenario, the changes (and downtime) will be barely noticed. The maintenance consist on introducing new hardware equipment in to the CloudVPS edge network. You can find additional details in Phabricator [0]. regards. [0] https://phabricator.wikimedia.org/T265288 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 2

s4 on labsdb on the analytics role.
by Manuel Arostegui 08 Oct '20

08 Oct '20

Hello, This is a heads up about the current situation with s4 (commons) and labsdb. There's been more activity lately on s4, and that had made labsdb1011 (analytics role) start lagging behind. https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=6&orgId=1&from=no… I have tried to ease its weight a couple of days ago, to help it recovering: https://gerrit.wikimedia.org/r/c/operations/puppet/+/630392 https://gerrit.wikimedia.org/r/c/operations/puppet/+/630531 https://gerrit.wikimedia.org/r/c/operations/puppet/+/630770 The last change has (as sort of expected) made labsdb1010 lag: https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=6&orgId=1&from=no… I am going to decrease the pt-kill query time from 3600 to 1800 to see if that helps labsdb1010 to guard the fort a bit. There's not much else we can do at the moment, but just keep all these issues in mind if people complain about lag on s4 (commons) on the analytics role. The web role is doing fine (labsdb1009 isn't lagging). Manuel.

2 6

unscheduled keystone maintenance
by Arturo Borrero Gonzalez 06 Oct '20

06 Oct '20

Hi there, we need to perform some unscheduled keystone maintenance right now. Authentication to some cloud services, in particular Horizon, might be interrupted during this maintenance period. We expect such maintenance to don't last more than 1h. regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

codfw1dev appreciation message
by Arturo Borrero Gonzalez 16 Jul '20

16 Jul '20

Hi there! wanted to share a big THANKS YOU to both Alex Monk (Krenair) and Andrew Bogott for the amazing work they have been doing to support the codfw1dev openstack deployment, including deploying an in-cloud puppetmaster to get it even closer to being a full mirror of our live openstack deployment eqiad1. Note working in getting the codfw1dev deployment in shape is something we don't usually set an explicit goal on, even though, from my point of view, is vital for us to have a proper testing environment. Traditionally I've been using codfw1dev only for testing non-virtual things, like openstack services, etc. With the current status, you can even test virtual stuff (running in VM instances) with an experience very close to eqiad1. This week I've been using the codfw1dev deployment for testing stuff, and for the first time since I can remember, the experience was... mostly magically smooth out of the box. Now the challenge for myself (and possibly others) is to remember to test stuff there first :-) I case you want to try this yourself, I left here some hints on how to use the deployment: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Testing_deployme… regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin