- Cloud-admin - lists.wikimedia.org

PuppetDB and Cumin in Toolforge
by Taavi Väänänen 10 Feb '22

10 Feb '22

Hi all, Toolforge now has a PuppetDB server! Thanks to Krenair's awesome work that makes it easy to add a PuppetDB server to a Puppet master using `role::puppetmaster::standalone`, I've installed tools-puppetdb-1 which seems to be working fine. For now it's only managing SSH host keys between hosts, but I'm sure you all can find some clever and awesome uses for it. In related news, there is now also a Cumin server in Toolforge and Toolsbeta (tools-cumin-1 and toolsbeta-cumin-1, respectively). These servers are hooked up to the per-project PuppetDB servers, allowing use of Cumin queries like this: $ cumin "P{P:toolforge::grid::node::web::generic}" 4 hosts will be targeted: tools-sgewebgen-10-[1-2].tools.eqiad1.wikimedia.cloud,tools-sgewebgrid-generic-[0901-0902].tools.eqiad.wmflabs Enjoy! -- Taavi

2 1

triage
by Andrew Bogott 14 Dec '21

14 Dec '21

Howdy! In response to the general consensus that we are doing too many things, I'm trying to compile a list of things that we are doing (for various definitions of 'doing'). I'd appreciate people reviewing this and adding things that I've forgotten. Please treat this as more of a brainstorming exercise than an official document -- add anything you can think of, and feel free to rename my categories or re-categorize things according to your whim. https://docs.google.com/document/d/1IKtYLYRvNOQraATWTNsAD2mF14XhxrOuzAxlIJx… I confess that I'm doing this partly out of laziness (it feels easier than reading 1000 phab tickets) -- I also have a nagging suspicion that we are doing (and/or distracted by) some things that are largely invisible on phabricator. Thank you! -A

2 2

Re: Placing and enabling m5 eqiad proxy
by Manuel Arostegui 27 Oct '21

27 Oct '21

Hello, I am considering Wednesday 27th (next week) at 14:00 UTC to do this change. If no one says otherwise, I will go ahead and schedule it. Thanks Manuel. On Wed, Oct 20, 2021 at 8:17 AM Manuel Arostegui <marostegui(a)wikimedia.org> wrote: > On Wed, Oct 20, 2021 at 1:23 AM Kunal Mehta <legoktm(a)wikimedia.org> wrote: > >> >> Hi, >> >> On 10/19/21 6:00 AM, Manuel Arostegui wrote: >> > My idea is to decrease the current TTL from m5-master (from 5 minutes >> to >> > 1 minute, so we can revert faster if needed). >> > Once that is in place, I will simply change the CNAME to point to >> > dbproxy1017.eqiad.wmnet. Everything should be transparent (if the >> grants >> > are correct), and connections should smoothly change to the proxy in >> the >> > next few days unless you want to restart the applications right away. >> >> Sounds good. I think it's probably just easier for us to restart mailman >> and mailman-web soon after to find any issues right away than later on. >> >> -- Kunal >> > > Sounds good! I will email this thread and update the task once I've > decided a day/time for this change for next week. >

1 2

Placing and enabling m5 eqiad proxy
by Manuel Arostegui 19 Oct '21

19 Oct '21

Hello everyone, I want to go ahead and finish the dbproxy installation on m5 ( https://phabricator.wikimedia.org/T288093). If you are receiving this email is cause you have databases on that section. Currently we have: +---------------------+ | Database | +---------------------+ | labsdbaccounts | | mailman3 | | mailman3web | | striker | | test_labsdbaccounts | | toolhub | +---------------------+ I have reviewed the grants and I think they are all in place (but it is a very hard task given how messy our grants are at the moment, so I might have missed some) My idea is to decrease the current TTL from m5-master (from 5 minutes to 1 minute, so we can revert faster if needed). Once that is in place, I will simply change the CNAME to point to dbproxy1017.eqiad.wmnet. Everything should be transparent (if the grants are correct), and connections should smoothly change to the proxy in the next few days unless you want to restart the applications right away. I don't think I will have time to do this during this week (as I just came back from holidays) but I am planning to do so next week. I will email once it is done, but please subscribe to the task too and let me know if you see something weird with your applications after the switch. Thanks Manuel.

1 0

WMCS OpenStack logs migration to Common Logging Schema
by Cole White 24 Jun '21

24 Jun '21

Hello! We (Observability) have identified a few OpenStack syslog producers as prime candidates for migration to the Common Logging Schema. We've prepared a patch <https://gerrit.wikimedia.org/r/c/operations/puppet/+/689262> that will duplicate the currently produced OpenStack logs to an ECS-compatible form to demo and prepare ECS-compatible Kibana dashboards. What we need from you: 1. A quick review to see if we're missing anything. 2. A list of Kibana dashboards and saved searches that need translation to use the new format. Thanks in advance! And as always, questions are welcome! -- Cole White Wikimedia Foundation

2 5

CloudVPS / Toolforge edge network maintenance 2021-05-06 @ 15:00 UTC
by Arturo Borrero Gonzalez 06 May '21

06 May '21

Hello there, We will be doing an upgrade to the CloudVPS edge network Thursday 2021-05-06 @ 15:00 UTC that will likely impact user experience, including Toolforge. We scheduled an 1h operation window. During that time, intermittent network interruption, packet loss and other network problems are to be expected. The edge network maintenance will affect how virtual machines (and Toolforge tools) contact NFS, wiki-replicas, wikis API endpoints, and, in general, any network traffic that flows leaving or entering the cloud (also known as north-south traffic). More information on the operation can be found in phabricator [0] and in wikitech [1]. Regards. [0]https://phabricator.wikimedia.org/T270704 [1] https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme… -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 2

Test message, please ignore
by Taavi Väänänen 06 May '21

06 May '21

This is a test message to make sure I can send mail this list. Please ignore. -- Taavi

2 1

(DRAFT) The future of /srv
by Andrew Bogott 10 Mar '21

10 Mar '21

Hello all! I've been thinking more about how to move to a future of quota-managed storage, and have written a few scraps of puppet code to support us moving off of the old labs::lvm roles. Attached below is a draft of an announcement email explaining the changes; writing this was the easiest way for me to think through what the changes would feel like to our users. For context: As best I can tell the only automated uses of our lvm puppet classes are via profile::labs::lvm::srv. So, while my announcement draft tries to acknowledge a diversity of uses cases I think there's really only one, which is "mount all extra space in /srv". That workflow will be replaced with a similar but slightly more cumbersome "attach a new cinder volume and then mount all the space on that volume in /srv/" A quick grep suggests that the most affected users workers will be people building new VMs for toolforge, CI, and Quarry. I'm not feeling at all bad about asking those folks to edit their puppet classes but I am feeling kind of bad about adding a manual per-VM horizon step; perhaps someday we'll have instrumentation for that. I welcome your thoughts about both the wording of this email and the overall plan that it implies. Thanks! -A ==== With Cinder storage[0] now available and well-tested on cloud-vps, it's time to move forward to widespread adoption. As of today, use of lvm (e.g. via role::labs::lvm::srv) is deprecated in favor of cinder volumes for all local storage outside of the initial 20GB allocated for for standard OS functions. In support of this move we'll be making several technical changes to our infrastructure over the coming days and weeks: - Default per-project cinder usage quotas will be increased from 10GB to 80GB - Instance flavors with variable disk sizes will be removed. Going forward all flavors will have a default local storage size of 20GB regardless of associated cores or RAM. - The old lvm puppet roles will continue to function as before on legacy VMs. On new VMs this role will detect the lack of available space for partitioning and report failures[1]. - Instance resizing will be enabled in the Horizon UI. This will allow you to adjust the allocated cores and RAM allocated to a given VM without rebuilding. Manual setup of Cinder volumes is documented here: https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_ins… If your workflow involves puppet roles that depend on labs_lvm, you will need to alter both your puppet code and your manual processes. I've provided some helper roles to assist. The simplest[2] is the new 'role::labs::cindermount::srv' which in most cases can be used as a drop-in replacement for role::labs::lvm::srv: once a cinder volume is attached to a VM that role will detect, format, and mount it as /srv. The base class cinderutils::ensure[3] can be reused for other variations on this theme. Please don't hesitate to contact me or other WMCS staff for assistance with puppet patches! In some cases the default quota of 80GB will be insufficient for a full migration to cinder from the old flavor model, especially if you have made use of the old 'large' or 'xlarge' sizes. When it comes time to to rebuild these larger VMs, please open a quota request ticket and detail what VMs (of what flavors) will be deleted and we will try to be quick and generous with quota expansions to assist in migration. [0] https://techblog.wikimedia.org/2021/02/05/cinder-on-cloud-vps/ [1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668567 [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/669958 [3] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668757

2 1

Network change for VMs contacting NFS dumps
by Arturo Borrero Gonzalez 23 Feb '21

23 Feb '21

Hello, today 2021-02-23 in about ~30 minutes (16:00 UTC) we will change how virtual machine instances running in Cloud VPS contact NFS dump servers [0]. There is no action required on your side. We anticipate little to no impact as a result of the network changes. But in case you notice something is not properly working with dumps NFS in Cloud VPS (or Toolforge) please contact us [1] as soon as possible. The relevant phabricator ticket [2] is T272397. regards. [0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Dumps [1] https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction#Commun… [2] https://phabricator.wikimedia.org/T272397 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

Change to how Cloud VPS and Toolforge contact Wikis
by Arturo Borrero Gonzalez 02 Feb '21

02 Feb '21

Hello, we are planning to change how Cloud VPS instances and Toolforge tools contact WMF-hosted wikis, in particular the source IP address for the network connection. The new IP address that wikis will see is 185.15.56.1. The change is scheduled to go live on 2021-02-08. More detailed information in wikitech: https://wikitech.wikimedia.org/wiki/News/CloudVPS_NAT_wikis If you are a Cloud VPS user or Toolforge developer, check your tools after that date to make sure they are properly running. If you detect a block, a rate-limit or similar, please let us know. If you are a WMF SRE or engineer involved with the wikis, be informed that this address could generate a significant traffic volume, perhaps about 30%-40% total wiki edits. We are trying to smooth the change as much as possible, so please send your feedback if you think there is something we didn't account for yet. Thanks, best regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin