- Cloud-admin - lists.wikimedia.org

Toolforge Futures meeting 2021-12-14
by Andrew Bogott 02 Mar '22

02 Mar '22

This meeting coalesced around a few major topics: * Why not just bring-your-own-container? ** We turn out to have fairly different ideas about what user experiences we want to support. ** General agreement that we could use more research and/or documentation about current and future workflows (although some of that already exists at https://www.mediawiki.org/wiki/Wikimedia_Cloud_Services_team/Our_audiences ** This question hinges on how sophisticated our users are or aren't. ** Komla suggests that many public cloud platforms provide simple deployment that don't require users to understand about container or k8s details; there's general agreement that we would also like to be like that ** Andrew thinks that push-to-deploy should be our 'main priority' and byo doesn't really address that. * How/When will we kill off the grid engine? ** We tend to think of this as blocked by push-to-deploy, but perhaps we should be open to other non-blocking options (e.g. the 'one big container' migration path) * What to do about the Stretch->Buster migration? ** Nicholas isn't convinced that we should migrate to Buster if we're just going to kill the grid eventually anyway. Andrew and Arturo mostly disagree. ** Probably the migration to Buster isn't a lot of hard mental work, just building things and throwing pre-existing switches. ** Main blocker for this is allocating time and tasking someone with doing the work ------ RAW ETHERPAD NOTES ------ == Toolforge next steps meeting 2021-12-14 == The approaching deadlines (from an announcement email on 2021-09-27) is: January 1st, 2022: * Stretch alternatives will be available for tool migration in Toolforge The proposed agenda follows. === Goals (Grid) === * Grid engine deprecation is blocked until users can customize container builds. Buildpacks are intended to address this need. * The blocker is providing equivalent runtime support for tools on the Kubernetes cluster as the current grid engine cluster has. Kubernetes containers are "thin" and tools will need the ability to add libraries and binaries that are custom to them. === Goals (Buildpacks) === ** Why are we doing this? *** Allow users to customize k8s images.. Better able to allow users to migrate off of gridengine * Arturo asks: why not just bring-your-own-container? ** Bryan answers: because bring-your-own container means containers without any toolforge integration (e.g. no ldap) * What about putting that in a base layer? ** That's a build your own container approach, which is what buildpacks is bringing ** But allowing build-your-own today is simpler than buildpacks for SREs ** Adding complexity for end users * Building your own docker image adds more complexity for end users ** Buildpacks also limit what you can put in a docker container, so potentially better security ** * How does buildpacks improve security? ** If you let someone else build a docker image, it could run as a different user and open security holes ** How could it be limited in k8s? The container runs as root on the host ** By building the container in buildpacks, we limit it ** k8s has full control over the runc runtime, so k8s could prevent user spoofing * Public clouds have bring your own container, so it must be possible right? * Would like to see a list of prioritized user workflows ** What workflows are we enabling with buildpacks? ** Consider looking at https://www.mediawiki.org/wiki/Wikimedia_Cloud_Services_team/Our_audiences * What's the long-term vision for TF? ** Push to git repo, and it "just works" ** The heroku workflow is built on buildpacks * What's the concern / fear about buildpacks? ** Complexity, can we find a way to simplify things? ** Lack of adoption (what if we end up the only ones using it) * Bringing your own container isn't a regression, but it's not a replacement for existing workflows (aka, we can't kill the grid by simply adding bring your own container) ** Why? ** Workflow and brainstate. Users will no longer understand how to run workflows. Running a job is much simpler than building and maintaining a container to run a job. Real risk of losing tools * Can we assume complex tool authors are technically capable of building a container? ** Complexity can be easily introduced in the grid; that doesn't mean they could build and maintain a container ** `webservice --backend=gridengine start` supports PHP, Python, and Perl fcgi out of the box. * Google cloud comparision -- industry seems to be using towards containers, but users don't need to build a container or even know that you are running in a container. Build a flask app, run a command, GCE builds a docker container and runs it ** all you need is for the buildpack to 'detect' what runtimes you need (it can be a file, or chekcking for a packages.lock or whatever) * So David et la -- does buildpacks seem complex? ** Buildpacks are easy-ish. Complexity is introduced by putting buildpacks into Toolforge ** Tekton / buildpacks PoC is easy * Build service ** admission controller, some custom resource definitions, hardware. Any docker registry could be used? Harbor presented some issues in running outside of k8s * How much engineering effort to bring that in prod? Is it possible to bring online this year? ** Depends on resources, and how org handles things ** Yes, if we work on it? :-) * Priority wise, a push to deploy solution is the most important / seamless thing we could work on ** Push to deploy toolforge, before grid deprecation even? ** Can't get rid of the grid until reasonable replacement ** Why? ** Because we need to support people today ** Actually need mixed runtime environment deployable on k8s * For example, 1 giant container that contains everything from a grid exec node ** So grid isn't dependent on push to deploy exactly === Goals (Buster) === * Have a plan for Buster on grid engine * Decide whether or not to have a plan for Buster on k8s (and, optionally, have a plan) * Decide what timeline adjustment is realistic; pick someone to communicate this delay to the users ==== Grid Engine ==== * Why do we hate the grid? ** When jobs are being run, little isolation. Uses watcher spawned processes + runtime hacks. ** No longer developed or supported by any upstreams ** In ~2018? looked at modern "grids" that spawned things similar, but could be better managed. (IE, slurm) ** At the time, decided k8s was the future, and decided that slurm or similar wasn't a good idea * Grid is important -- what could we do this year? ** Find someone to build a buster grid ** Make a mad dash at killing the grid asap. * Giant container is unlocked now , previously limited to 2G ** 3.1G container, needed on each k8s exec node ** Only need 1 copy on each node; shared between jobs ** When building new containers, be wary of variants when deploying new containers. Could be N x 3.1G ** Large containers aren't performant on k8s * Buster migration ** Most of the pieces are in place ** Build out nodes, switchover * Why the timeline? ** Organizational timeline ** Grid is senstive to DNS changes <discuss> <decide> <who> ==== k8s ==== <discuss> <decide> <who> ==== timeline ==== <discuss> <decide> <who> * dcaro: I propose delaying any non-urgent decision until we finish with the decision making/tech discussion email thread ** I would argue that the stretch deprecation is starting to become urgent (although I don't have context about that email thread) === Current status, open projects === The list may be the list of open projects: * stretch-to-buster migration final push ** including but not limited to the grid * grid engine deprecation timeline & plans ** draft here https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme… * pending kubernetes upgrades (if any) ** yes, current clusters are at 1.20 and 1.23 is the latest release. 1.21 is simple (and I think it would be a good opportunity for someone to learn the process) but 1.22 is complicated because it removes tons of deprecated things ** 1.20 is still in supported, support policy is last 4 releases * toolforge build service (buildpacks) ** currently we don't really have any visibility to package versions and available (security) upgrades in our images or deployed image versions - can we improve this with buildpack images or otherwise? https://phabricator.wikimedia.org/T291908 * toolforge jobs framework === Next steps, prioritization === * what to do next, and who === Long term future === * Share your ideas of how Toolforge should look in 5 years from now

3 3

Please leave mariadb stopped on cloudcontrol1004
by Andrew Bogott 24 Feb '22

24 Feb '22

Context: "Galera on cloudcontrol1004 going out of sync" https://phabricator.wikimedia.org/T302146 Galera (the database backend for OpenStack) has been very unstable ever since I upgraded the cluster to Bullseye. This is probably an issue with a buggy version of mariadb/galera. I'm trying an experiment: mariadb is currently stopped on cloudcontrol1004, and puppet disabled so it won't get restarted. I want to see if that change (a two-node cluster and/or removing the suspected-cursed cloudcontrol1004 from the cluster) causes things to stop breaking. I've done my best to downtime alerts that will result from this, bug if one leaks through please don't respond by enabling puppet on 1004 -- we want to leave that db node switched off for now. Other services on cloudcontrol1004 should continue to run normally. Thanks! -Andrew

1 0

Toolsdb and linkwatcher
by Arturo Borrero Gonzalez 18 Feb '22

18 Feb '22

Hi there, Apparently linkwatcher is a key tool in fighting vandalism in the wikis, but we cannot afford hosting it on ToolsDB any longer. I requests your comments / suggestions. Timeline: ==== 8< ==== == 2019-05-22 == Brooke detects high storage usage problem by the linkwatcher tool in ToolsDB [0]. [0] https://phabricator.wikimedia.org/T224154 == 2019-06-26 == Bryan warns [1] about the linkwatcher tool potentially becoming a disaster condition for ToolsDB. [1] https://phabricator.wikimedia.org/T224154#5284580 == 2019-07-19 == After some actions are taken to move linkwatcher to its own Cloud VPS project, activity stops and the database data remains in ToolsDB [2]. [2] https://phabricator.wikimedia.org/T227377 == 2022-02-17 == ToolsDB stops replicating to its secondary due to depleted storage [3]. It is discovered [4] that linkwatcher uses 1/3 of total storage of ToolsDB, this is ~1TB out of ~3TB. Data reported in other metrics [5] may not be accurate. While it may not be the primary cause of the problem, the "disaster condition" mentioned by Bryan 3 years earlier is becoming more apparent. [3] https://phabricator.wikimedia.org/T301951 [4] https://phabricator.wikimedia.org/T301967 [5] https://tool-db-usage.toolforge.org/ ==== 8< ==== regards -- Arturo Borrero Gonzalez Site Reliability Engineer Wikimedia Cloud Services Wikimedia Foundation

1 0

PuppetDB and Cumin in Toolforge
by Taavi Väänänen 11 Feb '22

11 Feb '22

Hi all, Toolforge now has a PuppetDB server! Thanks to Krenair's awesome work that makes it easy to add a PuppetDB server to a Puppet master using `role::puppetmaster::standalone`, I've installed tools-puppetdb-1 which seems to be working fine. For now it's only managing SSH host keys between hosts, but I'm sure you all can find some clever and awesome uses for it. In related news, there is now also a Cumin server in Toolforge and Toolsbeta (tools-cumin-1 and toolsbeta-cumin-1, respectively). These servers are hooked up to the per-project PuppetDB servers, allowing use of Cumin queries like this: $ cumin "P{P:toolforge::grid::node::web::generic}" 4 hosts will be targeted: tools-sgewebgen-10-[1-2].tools.eqiad1.wikimedia.cloud,tools-sgewebgrid-generic-[0901-0902].tools.eqiad.wmflabs Enjoy! -- Taavi

2 1

triage
by Andrew Bogott 15 Dec '21

15 Dec '21

Howdy! In response to the general consensus that we are doing too many things, I'm trying to compile a list of things that we are doing (for various definitions of 'doing'). I'd appreciate people reviewing this and adding things that I've forgotten. Please treat this as more of a brainstorming exercise than an official document -- add anything you can think of, and feel free to rename my categories or re-categorize things according to your whim. https://docs.google.com/document/d/1IKtYLYRvNOQraATWTNsAD2mF14XhxrOuzAxlIJx… I confess that I'm doing this partly out of laziness (it feels easier than reading 1000 phab tickets) -- I also have a nagging suspicion that we are doing (and/or distracted by) some things that are largely invisible on phabricator. Thank you! -A

2 2

Re: Placing and enabling m5 eqiad proxy
by Manuel Arostegui 27 Oct '21

27 Oct '21

Hello, I am considering Wednesday 27th (next week) at 14:00 UTC to do this change. If no one says otherwise, I will go ahead and schedule it. Thanks Manuel. On Wed, Oct 20, 2021 at 8:17 AM Manuel Arostegui <marostegui(a)wikimedia.org> wrote: > On Wed, Oct 20, 2021 at 1:23 AM Kunal Mehta <legoktm(a)wikimedia.org> wrote: > >> >> Hi, >> >> On 10/19/21 6:00 AM, Manuel Arostegui wrote: >> > My idea is to decrease the current TTL from m5-master (from 5 minutes >> to >> > 1 minute, so we can revert faster if needed). >> > Once that is in place, I will simply change the CNAME to point to >> > dbproxy1017.eqiad.wmnet. Everything should be transparent (if the >> grants >> > are correct), and connections should smoothly change to the proxy in >> the >> > next few days unless you want to restart the applications right away. >> >> Sounds good. I think it's probably just easier for us to restart mailman >> and mailman-web soon after to find any issues right away than later on. >> >> -- Kunal >> > > Sounds good! I will email this thread and update the task once I've > decided a day/time for this change for next week. >

1 2

Placing and enabling m5 eqiad proxy
by Manuel Arostegui 19 Oct '21

19 Oct '21

Hello everyone, I want to go ahead and finish the dbproxy installation on m5 ( https://phabricator.wikimedia.org/T288093). If you are receiving this email is cause you have databases on that section. Currently we have: +---------------------+ | Database | +---------------------+ | labsdbaccounts | | mailman3 | | mailman3web | | striker | | test_labsdbaccounts | | toolhub | +---------------------+ I have reviewed the grants and I think they are all in place (but it is a very hard task given how messy our grants are at the moment, so I might have missed some) My idea is to decrease the current TTL from m5-master (from 5 minutes to 1 minute, so we can revert faster if needed). Once that is in place, I will simply change the CNAME to point to dbproxy1017.eqiad.wmnet. Everything should be transparent (if the grants are correct), and connections should smoothly change to the proxy in the next few days unless you want to restart the applications right away. I don't think I will have time to do this during this week (as I just came back from holidays) but I am planning to do so next week. I will email once it is done, but please subscribe to the task too and let me know if you see something weird with your applications after the switch. Thanks Manuel.

1 0

WMCS OpenStack logs migration to Common Logging Schema
by Cole White 25 Jun '21

25 Jun '21

Hello! We (Observability) have identified a few OpenStack syslog producers as prime candidates for migration to the Common Logging Schema. We've prepared a patch <https://gerrit.wikimedia.org/r/c/operations/puppet/+/689262> that will duplicate the currently produced OpenStack logs to an ECS-compatible form to demo and prepare ECS-compatible Kibana dashboards. What we need from you: 1. A quick review to see if we're missing anything. 2. A list of Kibana dashboards and saved searches that need translation to use the new format. Thanks in advance! And as always, questions are welcome! -- Cole White Wikimedia Foundation

2 5

CloudVPS / Toolforge edge network maintenance 2021-05-06 @ 15:00 UTC
by Arturo Borrero Gonzalez 07 May '21

07 May '21

Hello there, We will be doing an upgrade to the CloudVPS edge network Thursday 2021-05-06 @ 15:00 UTC that will likely impact user experience, including Toolforge. We scheduled an 1h operation window. During that time, intermittent network interruption, packet loss and other network problems are to be expected. The edge network maintenance will affect how virtual machines (and Toolforge tools) contact NFS, wiki-replicas, wikis API endpoints, and, in general, any network traffic that flows leaving or entering the cloud (also known as north-south traffic). More information on the operation can be found in phabricator [0] and in wikitech [1]. Regards. [0]https://phabricator.wikimedia.org/T270704 [1] https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme… -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 2

Test message, please ignore
by Taavi Väänänen 06 May '21

06 May '21

This is a test message to make sure I can send mail this list. Please ignore. -- Taavi

2 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin