Cloud November 2021

cloud@lists.wikimedia.org

11 participants
7 discussions

[Cloud-announce] Eliminating Debian Stretch in Cloud VPS
by Andrew Bogott 13 Oct '22

13 Oct '22

Debian Stretch's security support ends in mid 2022, and the Foundation's OS policy already discourages use of existing Stretch machines. That means that it's time for all project admins to start rebuilding your VMs with Bullseye (or, if you must, Buster.) Any webservices running in Kubernetes created in the last year or two are most likely using Buster images already, so there's no action needed for those. Older kubernetes jobs should be refreshed to use more modern images whenever possible. If you are still using the grid engine for webservices, we strongly encourage you to migrate your jobs to Kubernetes. For other grid uses, watch this space for future announcements about grid engine migration; we don't yet have a solution prepared for that. Details about the what and why for this process can be found here: https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation Here is the deprecation timeline: March 2021: Stretch VM creation disabled in most projects July 6, 2021: Active support of Stretch ends, Stretch moves into LTS <- You are Here -> January 1st, 2022: Stretch VM creation disabled in all projects, deprecation nagging begins in earnest. Stretch alternatives will be available for tool migration in Toolforge May 1, 2022: All active Stretch VMs will be shut down (but not deleted) by WMCS admins. This includes Toolforge grid exec nodes. June 30, 2022: LTS support for Debian Stretch ends, all Stretch VMs will be deleted by WMCS admins _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

Source command not working with zsh
by Huji Lee 05 Dec '21

05 Dec '21

Hi all, Starting Nov 7, a number of the jobs I would run through Toolforge grid have stopped working. Each job consists of a .sh file like this <https://github.com/PersianWikipedia/fawikibot/blob/master/HujiBot/grid/jobs…> on the first line of which I use the source command to activate a python virtual environment. When I run source by hand, subsequent lines work. But when I call the .sh file and it tries to run the source command, I get a "source: not found" message, the virtual environment does not get activated and indeed running *which python* returns */usr/bin/python* which is bad. All my scripts depend on pip packages that are installed in the virtual env and not available with the system python. The main thing I did on Nov 7 was to add a line at the end of my too's account's .bash_profile as below: exec zsh This is because when I manually log into toolforge, I would like zsh to be my shell, and since tool accounts don't support chsh, I thought executing zsh directly from bash would be okay. But apparently, that now breaks the source command somehow. So I wonder: (a) Is there a way to properly change the default shell of tool accounts? (b) Is there a way to make *source* work under zsh? Importantly, I know the problem is with *exec zsh* because once I removed it and logged out and back in, all scripts worked correctly. Thanks, Huji

4 20

Search engine indexing of webapplication
by Finn Årup Nielsen 23 Nov '21

23 Nov '21

Dear Toolforge cloud people, I am running the Scholia web application on Toolforge and interested in have some of the pages indexed by search engines. We have '<meta name="robots" content="index, nofollow">' which should index but not crawl the Scholia website. We have 3 kinds of content on the webpages generated by Scholia: 1) "Static" content generated from Flask jinja2 templates. These gets indexed (but not that much). 2) Dynamic jQuery-based content based on the Wikidata API service. This does not seem to get indexed by some search engines. 3) Dynamic Wikidata Query Serviece-based content. This does not get indexed. I can understand 1) and 3), but not 2). https://query.wikidata.org/robots.txt is blocking bots request for 3), but as far as I can see https://www.wikidata.org/robots.txt does not block Wikidata API requests for 2). On a webpage on the public web, I have a link to https://scholia.toolforge.org/author/Q20980928, so I would think that that Scholia page would be indexed, and that the h1 tag that is set via the Wikidata API would be indexed. As far as I can determine the page is indexed at Bing and Quant, but not Duckduckgo and not Google. I am wondering whether there is anyone that can explain the discrepancy? As far as I understand Google does indeed index jQuery javascript-generated content. Should we refrain from having bots getting into Scholia and define a restrictive robots.txt to avoid burdening the Toolforge infrastructure too much? In Scholia, we at the moment have a pull request that implement serverside calls to the Wikidata Query Service to generate some metadata for the search engines that can be reached without hitting the WDQS robots.txt restriction. I have been reluctant to merge that pull request due to the extra load on the Toolforge as well as the extra time the request takes blocking the Scholia web application. We have around 100'000 requests per day according to Toolviews, - how much bot activity I do not know. I am wondering whether there is anyone who can give us advise here? best regards Finn Årup Nielsen

1 0

Shared Rust toolchain available on Toolforge
by Kunal Mehta 04 Nov '21

04 Nov '21

Hi, I have setup a "rustup" tool on Toolforge that contains the latest Rust toolchain for use by other tools so it doesn't need to be installed individually in each tool. Documentation on how to use it in your tool is available at <https://wikitech.wikimedia.org/wiki/Tool:Rustup>, all you have to do is add two lines to your tool's ~/.profile. If you need an older version or some other components, let me know and we can install it. Hopefully this makes developing Rust tools just a bit easier :) -- Legoktm

1 0

qsub waiting time on Toolforge
by Anis Elebiary 02 Nov '21

02 Nov '21

Hi all, I have a python script that creates a tsv for the redirect table. I submitted the job through qsub and it's been waiting for more than a day now. Is it normal for the waiting time to be this long ? And, if there's a better way to do the aforementioned task, please let me know. Thank you in advance. Best, Anis

4 8

[Cloud-announce] 2021-11-02: Cloud VPS network outage
by Arturo Borrero Gonzalez 02 Nov '21

02 Nov '21

Hi, Today 2021-11-02 we had a severe network outage on Cloud VPS and Toolforge. Several network connections were affected from 11:40 UTC to 13:20 UTC (1h40m duration). As of this writing the problem has been corrected. Detailed information can be seen in Phabricator: https://phabricator.wikimedia.org/T294853 Sorry for the inconvenience. regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

toolforge down?
by Roy Smith 02 Nov '21

02 Nov '21

https://spi-tools.toolforge.org/spi/ <https://spi-tools.toolforge.org/spi/> and https://xtools.wmflabs.org/ <https://xtools.wmflabs.org/> are loading extremely slowly. Actually, on both, it looks like you eventually get the HTML, but none of the statics (CSS, images, etc) load. https://bullseye.toolforge.org/ <https://bullseye.toolforge.org/> isn't responding at all. Makes me think there's something going on that's affecting all of toolforge.

2 2

2024

2023

2022

2021

2020

2019

2018

2017

Cloud November 2021