Cloud August 2024

cloud@lists.wikimedia.org

19 participants
16 discussions

javascript tooling?
by Roy Smith 19 Aug '24

19 Aug '24

I'm about to embark on building a client-side javascript tool intended to help with enwiki's [[WP:DYK]] process. JS is not my strength (and what I do know about tooling is quite outdated) so I'm looking for advice on what's in common use in the WMF environment these days. If I'm going to learn some new tools, I figure I might as well learn what folks here are using. If only because it'll make it easier for me to mooch on other people for help :-) As far as testing goes, I used to use JUnit. I gather that's pretty old-hat by now. What are you-all using? And for app frameworks. Angular? React? I hear Vie might be the new hotness? I'm leaning more towards "easy to learn" vs "most powerful".

7 16

[Cloud-announce] New cinder volumes may fail for the next day or two
by Andrew Bogott 16 Aug '24

16 Aug '24

Due to a serious hardware failure[0] we've had to take much of our file storage cluster offline. This should not affect any existing toolforge or cloud-vps usage, but new file-intensive work (e.g. creating a big new database or volume) may fail due to lack of space. We hope to have things restored to full service in a couple of days but right now we're just watching the suspect switch to see if it fails again. Sorry for any inconvenience! -Andrew [0] https://phabricator.wikimedia.org/T371878 _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

How to run golang jobs with Toolforge Build Service?
by Sascha Brawer 12 Aug '24

12 Aug '24

Dear all, Has anyone figured out how to use Toolforge Build Service for a tool (job + webservice) written in golang? I'm trying to switch qrank.toolforge.org to Toolforge Build Service, but I'm struggling with it. To debug this, I wrote a minimal job that just prints a hello message to stdout: https://github.com/brawer/wikidata-qrank/blob/main/cmd/hello/main.go Because Toolforge is emulating Heroku, it appears necessary to tell buildpack which binaries should get packaged into the container. The "// +heroku install" comment on line 14 of the following go.mod file seems to do the trick, together with listing the binaries again in a Heroku Procfile: https://github.com/brawer/wikidata-qrank/blob/main/go.mod#L14 https://github.com/brawer/wikidata-qrank/blob/main/Procfile When running the following command on login.toolforge.org, the project seems to be built just fine: $ toolforge build start https://github.com/brawer/wikidata-qrank Some interesting log message from build service: [step-build] 2024-04-30T18:13:31.170521374Z Building packages: [step-build] 2024-04-30T18:13:31.170585858Z - ./cmd/hello [step-build] 2024-04-30T18:13:31.170599159Z - ./cmd/qrank-builder [step-build] 2024-04-30T18:13:31.170608711Z - ./cmd/webserver [step-build] 2024-04-30T18:13:44.563334902Z [step-build] 2024-04-30T18:13:44.563404532Z [Setting launch table] [step-build] 2024-04-30T18:13:44.563417703Z Detected processes: [step-build] 2024-04-30T18:13:44.563427021Z - hello: hello [step-build] 2024-04-30T18:13:44.563437573Z - qrank-builder: qrank-builder [step-build] 2024-04-30T18:13:44.563446463Z - webserver: webserver [step-build] 2024-04-30T18:13:44.563488376Z - web: hello [step-build] 2024-04-30T18:13:44.577068596Z [step-build] 2024-04-30T18:13:44.577163104Z [Discovering process types] [step-build] 2024-04-30T18:13:44.579933448Z Procfile declares types -> web, qrank-builder, hello The "web: hello" in the launch table looks very suspicious; it's not what my Procfile states (see link above). But at least the build seems to have been successful: $ toolforge build show Build ID: qrank-buildpacks-pipelinerun-vhcrt Start Time: 2024-04-30T18:13:03Z End Time: 2024-04-30T18:14:00Z Status: ok Message: Tasks Completed: 1 (Failed: 0, Cancelled 0), Skipped: 0 Parameters: Source URL: https://github.com/brawer/wikidata-qrank Ref: N/A Envvars: N/A Destination Image: tools-harbor.wmcloud.org/tool-qrank/tool-qrank:latest Now, I'd be happy to just run the "hello" job and see its output log. However, when I do this: $ toolforge jobs run --command hello --image tool-qrank/tool-qrank:latest --mount=all --filelog The job runs for about five minutes (?!?!), and then: $ cat hello.out ERROR: failed to launch: bash exec: argument list too long So clearly something is off... but what? How to run a golang tool on Toolforge when using Build Service? Thanks for any help, — Sascha

3 2

Data Usage at WMF - Interview?
by Andrew Otto 10 Aug '24

10 Aug '24

Hello WMF Cloud Services users! The WMF Data Platform Engineering team <https://www.mediawiki.org/wiki/Data_Platform_Engineering> is strategizing about how people use WMF data. We would like to gather qualitative feedback <https://docs.google.com/document/d/13VEbydiAxvnDtOKrRK3riFPSQPuI4duk2L2NxUs…> from users of data inside and outside of the WMF. We'd like to talk to a few users of Wikimedia data available in Cloud Services. This includes (but is not limited to!) Quarry, Wiki replicas, dumps, WMF APIs, etc. If you are a heavy user of these data sources and would be willing to share your experience (and frustrations!), please talk to us! Reply to this email and we will set something up, hopefully within the next couple of months. Thank you, - Andrew Otto, Principal Software Engineer, WMF

2 1

[Cloud-announce] Toolforge API generally available
by David Caro 06 Aug '24

06 Aug '24

Hi! I'm happy to announce that we have a stable API for the Toolforge platform! You can now access and manage your tools using an API and build your own wrappers around it. This API is currently only available from within Toolforge, including from within your own jobs and webservices (just make sure to use `--mount=all` so the certs are available). We plan to make this API accessible from anywhere in the future. This is the same API that the toolforge clients use directly, so expect to be able to do everything the clients can do. Webservice is not available yet, but will be soon! (https://phabricator.wikimedia.org/T348755) You can find the actual API docs here: https://api-docs.toolforge.org And the wiki page with details on deprecation process, authentication, and a python example here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/API Let us know if you find any issues or if you have any questions through the usual channels. The full list is under "Communication and support" on: https://wikitech.wikimedia.org/wiki/Portal:Toolforge Thanks! David -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment." _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

2 1

[Cloud-announce] Still using tools-login.wmflabs.org?
by Andrew Bogott 02 Aug '24

02 Aug '24

If you are a toolforge user and your workflow still relies on tools-login.wmflabs.org, there are a few things to be aware of: 1) Until yesterday, tools-login.wmflabs.org pointed to a login host running Debian10(buster), which is no longer a supported operation system. I adjusted it to point to a newer bastion running Debian 12 (bookworm). 2) Some ssh key types (e.g. ssh-rsakeys created with puttygen on Windows, RSA keys signed with SHA-1) are no longer supported by modern operating systems for security reasons. If you are encountering a new login failure with toolforge /and/ you were using the .wmflabs.org login host, you probably need to generate modern keys (ECDSA) and upload them to https://idm.wikimedia.org 3) The software available on the new bastion differs somewhat from that on the old bastion. If you find yourself unable to work on the new hosts, please open a phabricator ticketwith the #toolforge tagor respond to this email with specifics. 4) The surprising nature of this change is due to the fact that WMCS has not used the .wmflabs.org domain for many years; anything under that domain is only working by accident and will probably be broken by us without warning. Please update your workflows and login scripts to use the new, preferred login name: login.toolforge.org. 5) If you urgently need the functions of the old login host, you can reach it as login-buster.toolforge.org. Be warned, though, that that host is not likely to exist for more than a few more weeks. Sorry for any inconvenience caused! -Andrew _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud August 2024