- Cloud - lists.wikimedia.org

javascript tooling?
by Roy Smith 18 Aug '24

18 Aug '24

I'm about to embark on building a client-side javascript tool intended to help with enwiki's [[WP:DYK]] process. JS is not my strength (and what I do know about tooling is quite outdated) so I'm looking for advice on what's in common use in the WMF environment these days. If I'm going to learn some new tools, I figure I might as well learn what folks here are using. If only because it'll make it easier for me to mooch on other people for help :-) As far as testing goes, I used to use JUnit. I gather that's pretty old-hat by now. What are you-all using? And for app frameworks. Angular? React? I hear Vie might be the new hotness? I'm leaning more towards "easy to learn" vs "most powerful".

7 16

[Cloud-announce] New cinder volumes may fail for the next day or two
by Andrew Bogott 16 Aug '24

16 Aug '24

Due to a serious hardware failure[0] we've had to take much of our file storage cluster offline. This should not affect any existing toolforge or cloud-vps usage, but new file-intensive work (e.g. creating a big new database or volume) may fail due to lack of space. We hope to have things restored to full service in a couple of days but right now we're just watching the suspect switch to see if it fails again. Sorry for any inconvenience! -Andrew [0] https://phabricator.wikimedia.org/T371878 _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

How to run golang jobs with Toolforge Build Service?
by Sascha Brawer 12 Aug '24

12 Aug '24

Dear all, Has anyone figured out how to use Toolforge Build Service for a tool (job + webservice) written in golang? I'm trying to switch qrank.toolforge.org to Toolforge Build Service, but I'm struggling with it. To debug this, I wrote a minimal job that just prints a hello message to stdout: https://github.com/brawer/wikidata-qrank/blob/main/cmd/hello/main.go Because Toolforge is emulating Heroku, it appears necessary to tell buildpack which binaries should get packaged into the container. The "// +heroku install" comment on line 14 of the following go.mod file seems to do the trick, together with listing the binaries again in a Heroku Procfile: https://github.com/brawer/wikidata-qrank/blob/main/go.mod#L14 https://github.com/brawer/wikidata-qrank/blob/main/Procfile When running the following command on login.toolforge.org, the project seems to be built just fine: $ toolforge build start https://github.com/brawer/wikidata-qrank Some interesting log message from build service: [step-build] 2024-04-30T18:13:31.170521374Z Building packages: [step-build] 2024-04-30T18:13:31.170585858Z - ./cmd/hello [step-build] 2024-04-30T18:13:31.170599159Z - ./cmd/qrank-builder [step-build] 2024-04-30T18:13:31.170608711Z - ./cmd/webserver [step-build] 2024-04-30T18:13:44.563334902Z [step-build] 2024-04-30T18:13:44.563404532Z [Setting launch table] [step-build] 2024-04-30T18:13:44.563417703Z Detected processes: [step-build] 2024-04-30T18:13:44.563427021Z - hello: hello [step-build] 2024-04-30T18:13:44.563437573Z - qrank-builder: qrank-builder [step-build] 2024-04-30T18:13:44.563446463Z - webserver: webserver [step-build] 2024-04-30T18:13:44.563488376Z - web: hello [step-build] 2024-04-30T18:13:44.577068596Z [step-build] 2024-04-30T18:13:44.577163104Z [Discovering process types] [step-build] 2024-04-30T18:13:44.579933448Z Procfile declares types -> web, qrank-builder, hello The "web: hello" in the launch table looks very suspicious; it's not what my Procfile states (see link above). But at least the build seems to have been successful: $ toolforge build show Build ID: qrank-buildpacks-pipelinerun-vhcrt Start Time: 2024-04-30T18:13:03Z End Time: 2024-04-30T18:14:00Z Status: ok Message: Tasks Completed: 1 (Failed: 0, Cancelled 0), Skipped: 0 Parameters: Source URL: https://github.com/brawer/wikidata-qrank Ref: N/A Envvars: N/A Destination Image: tools-harbor.wmcloud.org/tool-qrank/tool-qrank:latest Now, I'd be happy to just run the "hello" job and see its output log. However, when I do this: $ toolforge jobs run --command hello --image tool-qrank/tool-qrank:latest --mount=all --filelog The job runs for about five minutes (?!?!), and then: $ cat hello.out ERROR: failed to launch: bash exec: argument list too long So clearly something is off... but what? How to run a golang tool on Toolforge when using Build Service? Thanks for any help, — Sascha

3 2

Data Usage at WMF - Interview?
by Andrew Otto 09 Aug '24

09 Aug '24

Hello WMF Cloud Services users! The WMF Data Platform Engineering team <https://www.mediawiki.org/wiki/Data_Platform_Engineering> is strategizing about how people use WMF data. We would like to gather qualitative feedback <https://docs.google.com/document/d/13VEbydiAxvnDtOKrRK3riFPSQPuI4duk2L2NxUs…> from users of data inside and outside of the WMF. We'd like to talk to a few users of Wikimedia data available in Cloud Services. This includes (but is not limited to!) Quarry, Wiki replicas, dumps, WMF APIs, etc. If you are a heavy user of these data sources and would be willing to share your experience (and frustrations!), please talk to us! Reply to this email and we will set something up, hopefully within the next couple of months. Thank you, - Andrew Otto, Principal Software Engineer, WMF

2 1

[Cloud-announce] Toolforge API generally available
by David Caro 05 Aug '24

05 Aug '24

Hi! I'm happy to announce that we have a stable API for the Toolforge platform! You can now access and manage your tools using an API and build your own wrappers around it. This API is currently only available from within Toolforge, including from within your own jobs and webservices (just make sure to use `--mount=all` so the certs are available). We plan to make this API accessible from anywhere in the future. This is the same API that the toolforge clients use directly, so expect to be able to do everything the clients can do. Webservice is not available yet, but will be soon! (https://phabricator.wikimedia.org/T348755) You can find the actual API docs here: https://api-docs.toolforge.org And the wiki page with details on deprecation process, authentication, and a python example here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/API Let us know if you find any issues or if you have any questions through the usual channels. The full list is under "Communication and support" on: https://wikitech.wikimedia.org/wiki/Portal:Toolforge Thanks! David -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment." _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

2 1

[Cloud-announce] Still using tools-login.wmflabs.org?
by Andrew Bogott 01 Aug '24

01 Aug '24

If you are a toolforge user and your workflow still relies on tools-login.wmflabs.org, there are a few things to be aware of: 1) Until yesterday, tools-login.wmflabs.org pointed to a login host running Debian10(buster), which is no longer a supported operation system. I adjusted it to point to a newer bastion running Debian 12 (bookworm). 2) Some ssh key types (e.g. ssh-rsakeys created with puttygen on Windows, RSA keys signed with SHA-1) are no longer supported by modern operating systems for security reasons. If you are encountering a new login failure with toolforge /and/ you were using the .wmflabs.org login host, you probably need to generate modern keys (ECDSA) and upload them to https://idm.wikimedia.org 3) The software available on the new bastion differs somewhat from that on the old bastion. If you find yourself unable to work on the new hosts, please open a phabricator ticketwith the #toolforge tagor respond to this email with specifics. 4) The surprising nature of this change is due to the fact that WMCS has not used the .wmflabs.org domain for many years; anything under that domain is only working by accident and will probably be broken by us without warning. Please update your workflows and login scripts to use the new, preferred login name: login.toolforge.org. 5) If you urgently need the functions of the old login host, you can reach it as login-buster.toolforge.org. Be warned, though, that that host is not likely to exist for more than a few more weeks. Sorry for any inconvenience caused! -Andrew _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

[Cloud-announce] Brief toolforge build service outage Monday, 2024-07-29 @ 14:00 UTC
by Andrew Bogott 29 Jul '24

29 Jul '24

On Monday I'll be upgrading the database that backs toolforge's Harbor service. Any attempts to download new container images during this window will fail. Existing, running tools will not be affected, and periodically scheduled jobs should also be fine, but any brand new jobs may fail, depending on what is or isn't cache at the moment. The outage should be brief; if you see a cryptic failure around the scheduled time just take a 10 minute break and try again. -Andrew _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

[Cloud-announce] toolforge build service outage in progress
by Slavina Stefanova 24 Jul '24

24 Jul '24

There is currently an issue with Toolforge's image registry (Harbor) due to its database being out of space. This affects the Toolforge build service, meaning no new builds can be started, and no images can currently be pulled from the registry. Currently running tools should be unaffected. We are actively working on the issue and will update you here. Sorry for the downtime! -- Slavina Stefanova (she/her) Software Engineer | Developer Experience Wikimedia Foundation _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

[Cloud-announce] Trove database upgrades + downtime next week
by Andrew Bogott 23 Jul '24

23 Jul '24

Next week (starting 2024-07-22) I'll be updating Trove database servers[0] to use more up-to-date VM images and guest agents. This will involve a minute or two of downtime (in the best case) and possibly an hour or more of downtime for troublesome (and unpredictable) edge cases. The complete list of database servers can be found on the phabricator task below[1]. If you would prefer to schedule upgrades for your database for a particular time, please follow up with specifics on the task and we will find a time that works for everyone. If I don't hear otherwise, I'll perform the maintenance on your database at my convenience. This maintenance will affect the following cloud-vps projects: adiutor baglama2 citefix copypatrol deployment-prep discordbots dumps glamwikidashboard hoiscript huma library-upgrader linkwatcher maps mwoffliner mwstake osmit pm20database quarry reading-web-staging spacemedia wikisp wmcz-stats xtools Thank you! -Andrew [0] Trove is the software behind Horizons 'database' tab. Databases on VMs that were setup up manually or via puppet are unaffected by this maintenance. [1] https://phabricator.wikimedia.org/T369723 _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

Re: [IMPORTANT] Action Required for Your Wikitech Account Migration
by Travis Briggs 23 Jul '24

23 Jul '24

Hello, Just got this email, but the referenced site doesn't exist ( idm.wikipedia.org). Wondering if this was sent by mistake? Thanks, -Travis On Mon, Jul 22, 2024 at 12:14 AM Komla Sapaty <ssapaty(a)wikimedia.org> wrote: > Hello Audiodude > > We are reaching out to inform you about an important upcoming change > regarding your Wikitech account. > As part of our ongoing efforts to streamline account management and > improve security, we will be > migrating all old developer Wikitech accounts to the MediaWiki account > system (also referred to as global account or Wikimedia global account). > This means that in the future you will be able to access Wikitech using > your MediaWiki/Wikipedia account. > > ==What You Need to Do== > To ensure a smooth transition, we request that you manually link your > Wikitech account to > your MediaWiki account before September 16th. Please follow these steps: > *Log in to idm.wikipedia.org with the same username and password you > would use on wikitech.[0] > *Follow the instructions provided to link your Wikitech (LDAP) account to > your MediaWiki account. > > ==Why This Is Important== > Currently, Wikitech accounts are stored in LDAP, which has been > historically used for managing > technical community accounts across Wikimedia projects. We are now > consolidating these accounts > under the MediaWiki account system. > This change will help us better manage the Wikitech wiki, which also > contains essential technical > documentation. > > ==Migration Details== > *Linked Accounts: If you link your accounts by the deadline, your Wikitech > account will be migrated > to your Mediawiki account, retaining your full edit history. > *Unlinked Accounts: If you do not link your accounts by the deadline, your > Wikitech account will be > migrated to a default scheme (e.g., wikitech_OLDWIKITECHUSERNAME). Your > old edits will still be visible, > but new edits will be made under a new MediaWiki account with no immediate > link to your past edits. > > Your prompt action will help us ensure a seamless transition. > If you have already linked your accounts, no further action is required. > If you have any questions or need assistance, please do not hesitate to > contact us[1] > We are also maintaining a FAQ page here[2] that hopefully answers some > questions you may have. > > [0]: https://idm.wikipedia.org > [1]: > https://wikitech.wikimedia.org/wiki/Portal:Toolforge/About_Toolforge#Commun… > [2]: > https://wikitech.wikimedia.org/wiki/News/Migrating_Wikitech_Account_to_SUL > > Thanks > Komla Sapaty > (Developer Experience) > > >

8 13

2024

2023

2022

2021

2020

2019

2018

2017

Cloud