- Cloud-admin - lists.wikimedia.org

Toolforge Build Service Testing Next Steps
by Seyram Komla Sapaty 16 Jun '23

16 Jun '23

Hello Admins, For the past month, we have been testing Toolforge Build Service[0]. We invited about 30 tool maintainers to test out the service. A number of maintainers responded with valuable feedback. From the feedback received, there’s immediate need for the support for the following: 1. Custom environment variable support to avoid having to create configuration files in the NFS/tool home. 2. Package installation for some specific cases in which os/system libraries are needed. 3. Secrets management solution, that work similarly as environment variables, this will remove the need to store them on NFS/tool home. Work is underway to add support for all the above before the next round of invites. - For envvars (and secrets with it), see[1] for details. - For arbitrary packages installation, see[2]. There's also work on adding documentation for other languages (php being the next focus) so we can broaden the target languages.[3] For the next steps, we propose to expand the invitation to a higher number of tool maintainers as soon as the above services/features are available. The target date to gather the new list of invitees is Tuesday, the 20th of June, but will reconsider if the services are not yet ready. On that day, we will send another email to this list with the new list of proposed invites, that will be sent the day after if there's no comments. Please reach out if you have any questions or feedback. For more details about the release timeline see the dedicated task [4] For more details about the beta release itself, please see the following: Current project board [5]. The project page[6] Release discovered tasks[7]: Thanks for your support! [0]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service [1]: https://phabricator.wikimedia.org/T337538 [2]: https://phabricator.wikimedia.org/T336669 [3]: https://phabricator.wikimedia.org/T337397 [4]: https://phabricator.wikimedia.org/T335249 [5]: https://phabricator.wikimedia.org/project/board/5596/ [6]: https://wikitech.wikimedia.org/wPortal:Toolforge/Ongoing_Efforts/Toolforge_… [7]: https://phabricator.wikimedia.org/project/view/6529/ -- Seyram Komla Sapaty Developer Advocate Wikimedia Cloud Services

1 0

Toolforge repository moves
by David Caro 16 Jun '23

16 Jun '23

Hi! I have been moving some of the toolforge related repositories from gerrit to gitlab, you will find all of them under: https://gitlab.wikimedia.org/repos/cloud/toolforge I have renamed also a few of them to make the namings a bit more consistent: * Renamed toolforge-builds-api -> builds-api * Renamed toolforge-envvars-cli -> envvars-cli * Renamed toolforge-envvars-api -> envvars-api So please make sure to change the urls of your repos to the new urls: ``` git remote set-url origin git(a)gitlab.wikimedia.org/repos/cloud/toolforge/ <new_repo_name> ``` This will allow us to better setup CI and CD pipelines to build and deploy toolforge. There's an ongoing decision about the exact process here: https://phabricator.wikimedia.org/T339198 Make sure to write down your ideas there! Have a nice weekend, David

1 0

Access to log of quarry queries
by Hal Triedman 05 Jun '23

05 Jun '23

Hi cloud admins! My name is Hal Triedman — I'm a Privacy Engineer at WMF, but in my spare time I do a lot of work on machine learning. One of the things we've been looking into is the creation of label-query datasets for Mediawiki database queries, with the goal of being able to finetune an AI model to help users write queries with more ease/create embeddings that allow for easier searching of past queries. Quarry is particularly interesting for this project because it has the following qualities: 1) it is entirely on Mediawiki databases 2) it has been used to make hundreds of thousands of queries 3) many of those queries have relatively descriptive titles about what is happening in the SQL Is there any easy way of assembling a database of existing public title-query pairs (i.e. by running a database query that excludes things like "Untitled query", or just pulling published queries)? Please let me know, and thanks. Hal

3 2

Toolforge Build Service beta testing
by Seyram Komla Sapaty 12 May '23

12 May '23

Hello Admins, The build service is ready for the next phase, and we want to get some early users to test it and give early feedback! This next step is going to start this Monday, when around 100 tool maintainers will kindly receive an email asking them to try and test the service. The full list will be made available in the task [1] You can read a draft of the email here[2] Later next week, there will also be a session at the Athens Hackathon (thanks Slavina!) [3] introducing the build service to some new users too, as new user experiences give an important perspective too. We will tentatively stay in this period of feedback for ~1 month. At that point we will sit back, reflect on the feedback (will share the thoughts with you too), and decide if we are ready for a broader announcement (cloud-announcement, blog post, …, to be defined), to start getting wider input. In preparation for this, the team (including volunteers) has been working on setting up some minimal information on wikitech [4] and phabricator [5]. Feel free to add comments to the talk page [6] and/or make changes and fixes to those pages. There's still many things to figure out, and many more will show up during this phase, your help and input will be critical in shaping this service. I encourage you to try it out yourself if you have not, and open any bugs that you find or feature requests that you think would be useful, find the links for those in the feedback page [7]. You can see a more detailed planning in the task [1] (feel free to add comments there too) Thanks for your continued support! [1]: https://phabricator.wikimedia.org/T335249 [2]: https://etherpad.wikimedia.org/p/tQr7TQr20xorkEXXk6Pj [3]: https://phabricator.wikimedia.org/T336055 [4]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service [5]: https://phabricator.wikimedia.org/project/profile/6529/ [6]: https://wikitech.wikimedia.org/wiki/Help_talk:Toolforge/Build_Service [7]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Feedback -- Seyram Komla Sapaty Developer Advocate Wikimedia Cloud Services

3 7

Rethinking how we deploy and offer Cloud VPS (openstack)
by Arturo Borrero Gonzalez 12 May '23

12 May '23

Hi there, in the last quarter, we conducted a research [0] to evaluate and rethink how we deploy and offer Clod VPS (openstack), i.e, our IaaS setup. The research results were written into a wiki page [1] which summary being that the most attractive option for us is to move to a kubernetes + openstack-helm deployment of Cloud VPS / openstack in the future. This project is not trivial, and may be actually a multi-year project, so when / why / how this project will start is yet to be decided. The wiki page [1] contains plenty of details about everything related to the project. Also, a bunch of open questions and unknowns. Feel free to point to gaps or missing information bits in the plans. regards. [0] https://phabricator.wikimedia.org/T326758 [1] https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme… -- Arturo Borrero Gonzalez Senior SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

How and when to notify about maintenance
by Andrew Bogott 17 Apr '23

17 Apr '23

In response to our recent maintenance windows we got some feedback[0] about advance notice of outages. I created this chart to provide us with some internal guidelines about when we should publicize maintenance, and how to do so: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Maintenance_noti… You will notice that at the moment my imagination is limited to 'write to a mailing list.' I encourage people to fill in ideas on that page (or the associated talk page) about other ways we can warn people about these things. If we wind up with so many broadcast channels that it becomes impractical to actually use them all we can invest in automation. I'm also not especially committed to the brackets on that chart; I'd like to have broad categories and low standards, but edits are welcome! One thing that I want to be more mindful about is the distinction between "things that mess with our users" (e.g. quarry or horizon downtime) vs. "things that mess with our users' users" (e.g. web proxy downtime.) I'd love it if someone with better wiki-editing skills spruced up the chart to reflect that difference. -A [0] for example https://phabricator.wikimedia.org/T333477#8764263

2 1

Re: [Cloud-announce] Toolforge Kubernetes upgrade on 2023-04-03 (new date: 2023-04-10)
by Arturo Borrero Gonzalez 10 Apr '23

10 Apr '23

On 3/30/23 12:42, Arturo Borrero Gonzalez wrote: > On 3/28/23 00:13, Taavi Väänänen wrote: >> Hi, >> >> We will be upgrading the Toolforge Kubernetes cluster next Monday (2023-04-03) >> starting at around 10:00 UTC. >> >> The expected impact is that tools running on the Kubernetes cluster will get >> restarted a couple of times over the course of the few hours it takes for us >> to upgrade the entire cluster. The ability to manage tools will remain >> operational. >> >> Since the version we're upgrading to (1.22) removes a bunch of deprecated >> Kubernetes APIs, tools that use kubectl and raw Kubernetes resources directly >> may want to check that they're on the latest available versions. The vast >> majority of tools that are only using the Jobs framework and/or the webservice >> command are not affected by these changes. >> > > This has been rescheduled to Monday 2023-04-10 to leave room for the other > operations we have. > Hi there! This is happening now! https://phabricator.wikimedia.org/T286856 regards. -- Arturo Borrero Gonzalez Senior SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

lima-kilo: heads up, new directory tree
by Arturo Borrero Gonzalez 10 Mar '23

10 Mar '23

Hi there, if you are using lima-kilo [0], I just merged a change [1] to confine all content into a directory (~/.local/toolforge-lima-kilo/) and that includes the use configuration file: * old: ~/.config/toolforge-lima-kilo-userconfig.yaml * new: ~/.local/toolforge-lima-kilo/userconfig.yaml While the update includes some logic to handle this change (it will relocate the old file to the new location if you didn't do it beforehand), if you add new config settings to the old file, it wont have any effect. regards. [0] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/lima-… [1] https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/commit/5d981… -- Arturo Borrero Gonzalez Senior SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

Toolforge: brief network maintenance today 2023-03-06
by Arturo Borrero Gonzalez 06 Mar '23

06 Mar '23

Hi there! Today 2023-03-06, in a few minutes, we will restart the Toolforge internal network, A brief interruption of network communications is expected during the maintenance. This is because we're re-deploying calico to the kubernetes cluster [0]. No action required on your side. regards. [0] https://phabricator.wikimedia.org/T328539 -- Arturo Borrero Gonzalez Senior SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

Some upcoming changes to the Cloud VPS metrics stack
by Taavi Väänänen 06 Feb '23

06 Feb '23

Hi all, There are a couple of major changes to our Cloud VPS o11y stack that I'm planning to make in the near term. Most of this should be visible on Phabricator as well, but I wanted to make everyone aware here regardless since following activity on Phabricator is hard and I don't want to cause any major surprises here. I'm sure some of this will have an effect on our users and I/we need to communicate it beforehand, but I'm not at that stage quite yet. == 1. New Grafana instance: grafana.wmcloud.org == The first and hopefully least impactful change is replacing the current grafana-cloud.wikimedia.org (aka grafana-labs.wikimedia.org) Grafana instance with a new one. The reason is that the current one runs directly on hardware (cloudmetrics*.eqiad.wmnet), and due to upstream Grafana changes it soon won't be able to reach out to Prometheus instances living on cloud-vps VMs. This work is tracked as T307465, and has patches up for review starting from https://gerrit.wikimedia.org/r/c/operations/puppet/+/869210/. == 2. Diamond removal == The Prometheus instance in metricsinfra now scrapes all Cloud VPS VMs. This was the primary blocker for getting rid of Diamond (a Python 2 program that collected node metrics and pushed them to Graphite). I hope that this transition will be mostly invisible to users if we migrate the most used Grafana dashboard (cloud-vps-project-board) to pull the metrics from Prometheus instead. This is tracked as T317032. == 3. Statsd/Graphite removal (once Diamond is gone) == My understanding is that the statsd/Graphite service was originally not intended as a generic service for cloud-vps users (although it certainly is used like one today). Either way we don't really have a good replacement for it except some limited cases that could use node-exporter text files instead. I'm not sure how big of a deal that is if we never claimed to support it anyway? This is tracked as T326266. Any questions or comments on the above? Taavi

2 2

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin