Cloud-admin June 2023

cloud-admin@lists.wikimedia.org

8 participants
5 discussions

Second Round of Build Service Testing
by Seyram Komla Sapaty 04 Sep '23

04 Sep '23

Hello Admins, As communicated earlier, we have put together a list of about 100 tools whose maintainers we propose to invite in the next round of testing. This expanded list now includes tools written in languages other than Python. You can see the list here[0] The feedback and suggestions around custom and secret environment variable support[1] and package installation for buildservice[2] have all now been successfully rolled out to toolforge. If there's no changes requested, the new invites will be sent on the 23rd of Jun. Kindly reach out if you have any questions or feedback. Thank you! [0] https://etherpad.wikimedia.org/p/second-round [1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Install_ap… [2] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Envvars_Service

3 4

Help with a Wikimania presentation on how Cloud Services governance works?
by Bryan Davis 11 Aug '23

11 Aug '23

After talking with both Arturo and Birgit about things we might present at Wikimania, I came up with this abstract for a talk: Co-creating platforms and products: how the Wikimedia Cloud Services team works with the larger Wikimedia technical community to build and maintain Cloud VPS, Toolforge, Quarry, PAWS, and more Did you know that volunteers are involved in planning, building, and maintaining the Cloud VPS and Toolforge projects as co-equals with paid staff from the Wikimedia Foundation? Since the start of the "Labs" project in 2011, one of the guiding principles for WMCS projects has been improving collaboration between Foundation staff and technical volunteers. Learn more about some of the policies and practices that are used to make this collaboration possible. The submission would be under either the "governance" or "technology" tracks. I think it would work best as a panel discussion that is either "hybrid" (some folks in Singapore, some on-line) or pre-recorded video. I think this is something that folks in the community might be interested in learning a bit about. I also think it would be interesting for those of us who have participated in this process to take some time to reflect on how we have worked together in the past and how we might like to see those those processes and practices evolve in the future. To make this talk work well there should be active voices from both the paid and volunteer staff involved. Towards that end, I'm mailing the cloud-admin@ list + 4 of you that I know have been active in the past in helping with Toolforge and/or Cloud VPS admin and features work to gauge your interest in participating. Thoughts? Bryan -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

6 14

Toolforge Build Service Testing Next Steps
by Seyram Komla Sapaty 16 Jun '23

16 Jun '23

Hello Admins, For the past month, we have been testing Toolforge Build Service[0]. We invited about 30 tool maintainers to test out the service. A number of maintainers responded with valuable feedback. From the feedback received, there’s immediate need for the support for the following: 1. Custom environment variable support to avoid having to create configuration files in the NFS/tool home. 2. Package installation for some specific cases in which os/system libraries are needed. 3. Secrets management solution, that work similarly as environment variables, this will remove the need to store them on NFS/tool home. Work is underway to add support for all the above before the next round of invites. - For envvars (and secrets with it), see[1] for details. - For arbitrary packages installation, see[2]. There's also work on adding documentation for other languages (php being the next focus) so we can broaden the target languages.[3] For the next steps, we propose to expand the invitation to a higher number of tool maintainers as soon as the above services/features are available. The target date to gather the new list of invitees is Tuesday, the 20th of June, but will reconsider if the services are not yet ready. On that day, we will send another email to this list with the new list of proposed invites, that will be sent the day after if there's no comments. Please reach out if you have any questions or feedback. For more details about the release timeline see the dedicated task [4] For more details about the beta release itself, please see the following: Current project board [5]. The project page[6] Release discovered tasks[7]: Thanks for your support! [0]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service [1]: https://phabricator.wikimedia.org/T337538 [2]: https://phabricator.wikimedia.org/T336669 [3]: https://phabricator.wikimedia.org/T337397 [4]: https://phabricator.wikimedia.org/T335249 [5]: https://phabricator.wikimedia.org/project/board/5596/ [6]: https://wikitech.wikimedia.org/wPortal:Toolforge/Ongoing_Efforts/Toolforge_… [7]: https://phabricator.wikimedia.org/project/view/6529/ -- Seyram Komla Sapaty Developer Advocate Wikimedia Cloud Services

1 0

Toolforge repository moves
by David Caro 16 Jun '23

16 Jun '23

Hi! I have been moving some of the toolforge related repositories from gerrit to gitlab, you will find all of them under: https://gitlab.wikimedia.org/repos/cloud/toolforge I have renamed also a few of them to make the namings a bit more consistent: * Renamed toolforge-builds-api -> builds-api * Renamed toolforge-envvars-cli -> envvars-cli * Renamed toolforge-envvars-api -> envvars-api So please make sure to change the urls of your repos to the new urls: ``` git remote set-url origin git(a)gitlab.wikimedia.org/repos/cloud/toolforge/ <new_repo_name> ``` This will allow us to better setup CI and CD pipelines to build and deploy toolforge. There's an ongoing decision about the exact process here: https://phabricator.wikimedia.org/T339198 Make sure to write down your ideas there! Have a nice weekend, David

1 0

Access to log of quarry queries
by Hal Triedman 05 Jun '23

05 Jun '23

Hi cloud admins! My name is Hal Triedman — I'm a Privacy Engineer at WMF, but in my spare time I do a lot of work on machine learning. One of the things we've been looking into is the creation of label-query datasets for Mediawiki database queries, with the goal of being able to finetune an AI model to help users write queries with more ease/create embeddings that allow for easier searching of past queries. Quarry is particularly interesting for this project because it has the following qualities: 1) it is entirely on Mediawiki databases 2) it has been used to make hundreds of thousands of queries 3) many of those queries have relatively descriptive titles about what is happening in the SQL Is there any easy way of assembling a database of existing public title-query pairs (i.e. by running a database query that excludes things like "Untitled query", or just pulling published queries)? Please let me know, and thanks. Hal

3 2

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin June 2023