June 2020 - Cloud - lists.wikimedia.org

Toolforge: new domain toolforge.org
by Arturo Borrero Gonzalez 07 Jul '20

07 Jul '20

Hi! We are happy to announce the new domain 'toolforge.org' is now ready to be adopted by our Toolforge community. There is a lot of information related to this change in a wikitech page we have for this: https://wikitech.wikimedia.org/wiki/News/Toolforge.org The most important change you will see happening is a new domain/scheme for Toolforge-hosted webservices: * from https://tools.wmflabs.org/<toolname>/ * to https://<toolname>.toolforge.org/ A live example of this change can be found in our internal openstack-browser webservice tool: * legacy URL: https://tools.wmflabs.org/openstack-browser/ * new URL: https://openstack-browser.toolforge.org This domain change is something we have been working on for months previous to this announcement. Part of our work has been to ensure we have a smooth transition from the old domain (and URL scheme) to the new canonical one. However, we acknowledge the ride might be bumpy for some folks, due to technical challenges or cases we didn't consider when planning this migration. Please reach out intermediately if you find any limitation or failure anywhere related to this change. The wikitech page also contains a section with information for common problems. You can check now if your webservice needs any specific change by creating a temporal redirection to the new canonical URL: $ webservice --canonical --backend=kubernetes start [..] $ webservice --canonical --backend=gridengine start [..] The --canonical switch will create a temporal redirect that you can turn on/off. Please use this to check how your webservice behaves with the new domain/URL scheme. If you start the webservice without --canonical, the temporal redirect will be removed. We aim to introduce permanent redirects for the legacy URLs on 2020-06-15. We expect to keep serving legacy URLs forever, by means of redirections to the new URLs. More information on the redirections can also be found in the wikitech page. The toolforge.org domain is finally here! <3 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

14 29

Do you use Rust on Toolforge?
by Kunal Mehta 30 Jun '20

30 Jun '20

Hi, Do you use, or want to use Rust on Toolforge? While it's not officially supported, some of us are discussing on Phabricator what the current pain points are, how to work around them and what we'd like to see in the future. If you have comments/suggestions/feedback, we'd appreciate it either the task[1] or just as a reply on this mailing list. I've also started a documentation page that may help people out[2] - let us know if you'd like to see something added to or clarified on that page. Additionally, if you're currently using Rust, please add your name & tool to the list I've started on the wikitech page[2] so we can gauge what the usage and interest is. [1] https://phabricator.wikimedia.org/T194953 [2] https://wikitech.wikimedia.org/wiki/User:Legoktm/Rust_on_Toolforge Thanks, -- Legoktm

3 2

Gerrit upgrade on Saturday, 27th of June
by Christian Aistleitner 28 Jun '20

28 Jun '20

[Apologies for cross-posting] Hi, TL;DR: We will take our Gerrit instance at gerrit.wikimeda.org offline on Saturday, 27th of June from 17:00--22:00 UTC (hopefully much shorter) to upgrade to the latest Gerrit v3.2.2. (10:00--15:00 San Francisco time, and 19:00--24:00 Central European time) --------------------------------- We're currently running Gerrit v2.15, which is no longer supported and we will upgrade to Gerrit v3.2 on Saturday 27th of June. This new version brings lots of improvements. The most noticeable change is probably the UI overhaul. The (default) "old UI" gets removed and the "new UI" has been completely revamped. It's much more modern looking and easier to use. And it also became far more usable on mobile phones. But the new version of course also comes with fixes and additions around the ssh interface, and the HTTP API. Since this is such a big upgrade, we want to give you a chance to look at it beforehand. You can test at https://gerrit-test.wikimedia.org/ It holds a copy of our Gerrit data as of 15th of June, and it runs the new Gerrit v3.2.2. We'll keep tuning things there during this week, so it might get an occasional restart. But look around and test the new UI, the HTTP API, etc. If you run into issues or discover bugs, please open tickets in Phabricator and set the Gerrit 3.x upgrade task (T254158) as parent. Note that this test Gerrit instance is already hooked up with our Phabricator. If you upload / merge / abandon / etc changes with a `Bug: Tnnnn` footer, you'll get comments on the corresponding tasks in Gerrit. So please use our test task T775 for your experiments. As everything, also this upgrade comes with a few downsides: * The new UI no longer works on Internet Explorer (currently ~0.02% of Gerrit's web requests). Microsoft's Edge browser works though. So if you are still using Internet Explorer, please consider switching to a different browser. * Users of git-review need to use at least version 1.27 (released September 2018). Otherwise uploading changes will fail. * Your git should be at least version 2.2.0 (released November 2014) to take advantage of the new, much simpler commit-msg hook. Keeping my fingers crossed for a painless update, Christian with lots of helping hands from RelEng, and SRE

4 11

Slow uWSGI logging
by Roy Smith 25 Jun '20

25 Jun '20

I'm running a web server with "webservice --backend=kubernetes python3.7". As I tail the uwsgi.log file, requests to my server get logged with very long delays. I just timed one at about a minute and a half between when the request was served (03:07:57 UTC 2020) and when it showed up in the log file (03:09:23 UTC 2020). What is going on here? Is there some way to make it not do this?

2 7

Need help with selectize.js
by Roy Smith 24 Jun '20

24 Jun '20

I'm mostly a python guy, who only dabbles in javascript when I can't avoid it :-) I've got a django-based tool (https://tools.wmflabs.org/spi-tools-dev/spi/ <https://tools.wmflabs.org/spi-tools-dev/spi/>) which uses selectize.js. For simple cases, everything works fine. But, if I click on the drop-down, delete the contents, type in a new value, and submit the form (Sock Info button), the submitted form data has case_name blank. Is there anybody here who is familiar with javascript and/or selectize.js who would be willing to point me in the right direction?

6 9

Screen sessions
by Isaac Johnson 24 Jun '20

24 Jun '20

I'm interested in running some long-ish scripts that loop through the dump replicas on Toolforge. Eventually, this sort of thing might move to crontab, but for now it would be nice to run a screen session as we test / debug the scripts. The problem is that if I run the scripts from my tool account (i.e. after "become <tool-name>"), I get the following error: Cannot open your terminal '/dev/pts/28' - please check. The internet indicates that this is because there is a user mismatch because of the "become" command [1]. The suggested solutions weren't clearly applicable: * I can't just skip the "become" step and run the screen session as my personal username as that's explicitly against the rules [2] (though it is possible to start a screen session from my personal username without getting the above error). * It's not clear to me that running the screen session as /dev/null is possible / smart / secure etc. Essentially, I don't have a great read on whether that actually is the correct solution or just another bad hack. So is it possible to run screen sessions from my tool account? If so, does anyone know how I might get around this issue? [1] https://stackoverflow.com/questions/21328140/screen-cannot-open-your-termin… [2] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules Best, Isaac -- Isaac Johnson (he/him/his) -- Research Scientist -- Wikimedia Foundation

4 5

[Cloud-announce] Toolforge email server now enforcing ratelimiting
by Arturo Borrero Gonzalez 24 Jun '20

24 Jun '20

Hi, we just enabled email ratelimiting in our MTA server [0] in Toolforge. Please, report any problem or issue you may find related to this. The current limit is 100 messages per hour per sender address. We may tune the value as we observe the behavior of the system and the users. regards. [0] https://en.wikipedia.org/wiki/Message_transfer_agent -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

[Mediawiki-api-announce] Planned removal of api_urls field in /page/summary endpoint
by Adam Baso 17 Jun '20

17 Jun '20

This message is a notice about forthcoming removal of a response field in the REST API’s /page/summary endpoint. [1] This message was posted to wikitech-l and is being cross-posted on mediawiki-api-announce. The endpoint is used in the Page Previews (“hovercards”) functionality on the classic web (desktop) and Android & iOS experiences for Wikipedia, in addition to numerous external experiences. We don't anticipate impacts on the mainline Wikipedia experiences from this forthcoming field removal, as the endpoint will still be operational. The REST API's /page/summary endpoint provides an api_urls field in its response with links to several other REST API endpoints supported by the Page Content Service. Three of the api_urls subfields refer to experimental endpoints that have been removed from the REST API altogether in favor of newer API endpoints. 1. media (/page/media) 2. references (/page/references) 3. metadata (/page/metadata). api_urls also contains subfields referring to stable endpoints that continue to exist in the REST API: 1. summary (which is self referential) 2. edit_html (/page/html - the Parsoid HTML) 3. talk_html (/page/html/Talk:<title> - the Parsoid HTML for the corresponding Talk page) Wikimedia’s Product Infrastructure team intends to remove the api_urls field of the /page/summary response. A cursory review at https://codesearch.wmflabs.org/ and Wikimedia Git mirrors suggests api_urls isn’t in use in consuming code. Review of web logs suggests traffic for the following endpoints: - The media endpoint is at about 5% of its original traffic before its decommission and it appears to be from old Wikipedia for Android clients. This is expected. - The references endpoint’s Wikipedia for Android traffic does not seem present and its other traffic appears to be from non-user application software based on User-Agent header components. -The metadata endpoint’s traffic seems to have all but stopped. This change is being announced in advance of the change because the endpoint is advertised as stable. [2] Please update your clients if you rely on the presence of the api_urls field. If this change poses a problem for your clients, please do let us know as soon as possible at the tracking task: https://phabricator.wikimedia.org/T247991 We plan to remove the api_urls field described here on or after 14 July 2020. Thank you. Adam Baso Director of Engineering Wikimedia Foundation [1] https://en.wikipedia.org/api/rest_v1/#/Page%20content/get_page_summary__tit… [2] Refer to https://www.mediawiki.org/wiki/API_versioning#End_point_stability and https://www.mediawiki.org/wiki/Wikimedia_Product/Wikimedia_Product_Infrastr… for more information on stability designations. _______________________________________________ Mediawiki-api-announce mailing list Mediawiki-api-announce(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce

1 0

[Cloud-announce] NFS maintenance tomorrow 2020-06-11
by Brooke Storm 11 Jun '20

11 Jun '20

Tomorrow (June 11th) at 1600 UTC, we will be failing over the primary NFS server to do maintenance and upgrades on it. The secondary partner in the cluster is already upgraded and ready, and recent changes *should* make it a fairly straightforward failover with a brief period of high load. If it doesn't proceed smoothly, it will be a slightly longer period of high load and NFS lockup as failover completes (10-20 min or so). After maintenance it will be failed back, which will also, hopefully, be quick and painless. -- Brooke Storm SRE Wikimedia Cloud Services bstorm(a)wikimedia.org IRC: bstorm_ _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 2

[Cloud-announce] DNS updates next Tuesday 2020-06-09, 14:00-16:00 UTC
by Andrew Bogott 09 Jun '20

09 Jun '20

Hello! Next week we'll be rebuilding and upgrading the hardware that provides DNS service to cloud-vps and toolforge. These rebuilds will start at 14:00 UTC and the whole process may take 2-3 hours. It's likely that DNS lookups will be somewhat slower as clients fail over between the in-progress and the working server. In theory there should be few other user-facing effects from these upgrades. In practice, though, this isn't something that we've done for quite a while, and touching DNS is always risky since it underlies pretty much everything. Here are some things to be ready for: - As a precaution we'll be disabling Horizon during the window to prevent new VMs or DNS changes landing in an inconsistent state. - Some badly-behaved DNS clients won't fail over properly and will report errors when their primary DNS server is down. - Puppet will almost certainly experience transient failures, since Puppet is known to be one of those badly-behaved clients. - If things go very badly there may be periods of total DNS outage which will result in many WMCS-hosted services failing. There's no particular reason that this /should/ happen, but this is the worst-case scenario. For additional context, the phabricator task for this work is https://phabricator.wikimedia.org/T253780 - Andrew + the WMCS team _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

2 4

2024

2023

2022

2021

2020

2019

2018

2017

Cloud June 2020