Debian Stretch's security support ends in mid 2022, and the Foundation's
OS policy already discourages use of existing Stretch machines. That
means that it's time for all project admins to start rebuilding your VMs
with Bullseye (or, if you must, Buster.)
Any webservices running in Kubernetes created in the last year or two
are most likely using Buster images already, so there's no action needed
for those. Older kubernetes jobs should be refreshed to use more modern
images whenever possible.
If you are still using the grid engine for webservices, we strongly
encourage you to migrate your jobs to Kubernetes. For other grid uses,
watch this space for future announcements about grid engine migration;
we don't yet have a solution prepared for that.
Details about the what and why for this process can be found here:
https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation
Here is the deprecation timeline:
March 2021: Stretch VM creation disabled in most projects
July 6, 2021: Active support of Stretch ends, Stretch moves into LTS
<- You are Here ->
January 1st, 2022: Stretch VM creation disabled in all projects,
deprecation nagging begins in earnest. Stretch alternatives will be
available for tool migration in Toolforge
May 1, 2022: All active Stretch VMs will be shut down (but not deleted)
by WMCS admins. This includes Toolforge grid exec nodes.
June 30, 2022: LTS support for Debian Stretch ends, all Stretch VMs will
be deleted by WMCS admins
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Hi all,
Starting Nov 7, a number of the jobs I would run through Toolforge grid
have stopped working. Each job consists of a .sh file like this
<https://github.com/PersianWikipedia/fawikibot/blob/master/HujiBot/grid/jobs…>
on the first line of which I use the source command to activate a python
virtual environment. When I run source by hand, subsequent lines work. But
when I call the .sh file and it tries to run the source command, I get a
"source: not found" message, the virtual environment does not get activated
and indeed running *which python* returns */usr/bin/python* which is bad.
All my scripts depend on pip packages that are installed in the virtual env
and not available with the system python.
The main thing I did on Nov 7 was to add a line at the end of my too's
account's .bash_profile as below:
exec zsh
This is because when I manually log into toolforge, I would like zsh to be
my shell, and since tool accounts don't support chsh, I thought executing
zsh directly from bash would be okay. But apparently, that now breaks the
source command somehow.
So I wonder:
(a) Is there a way to properly change the default shell of tool accounts?
(b) Is there a way to make *source* work under zsh?
Importantly, I know the problem is with *exec zsh* because once I removed
it and logged out and back in, all scripts worked correctly.
Thanks,
Huji
Dear Toolforge cloud people,
I am running the Scholia web application on Toolforge and interested in
have some of the pages indexed by search engines. We have '<meta
name="robots" content="index, nofollow">' which should index but not
crawl the Scholia website.
We have 3 kinds of content on the webpages generated by Scholia:
1) "Static" content generated from Flask jinja2 templates. These gets
indexed (but not that much).
2) Dynamic jQuery-based content based on the Wikidata API service. This
does not seem to get indexed by some search engines.
3) Dynamic Wikidata Query Serviece-based content. This does not get
indexed.
I can understand 1) and 3), but not 2).
https://query.wikidata.org/robots.txt is blocking bots request for 3),
but as far as I can see https://www.wikidata.org/robots.txt does not
block Wikidata API requests for 2).
On a webpage on the public web, I have a link to
https://scholia.toolforge.org/author/Q20980928, so I would think that
that Scholia page would be indexed, and that the h1 tag that is set via
the Wikidata API would be indexed. As far as I can determine the page is
indexed at Bing and Quant, but not Duckduckgo and not Google.
I am wondering whether there is anyone that can explain the discrepancy?
As far as I understand Google does indeed index jQuery
javascript-generated content.
Should we refrain from having bots getting into Scholia and define a
restrictive robots.txt to avoid burdening the Toolforge infrastructure
too much?
In Scholia, we at the moment have a pull request that implement
serverside calls to the Wikidata Query Service to generate some metadata
for the search engines that can be reached without hitting the WDQS
robots.txt restriction. I have been reluctant to merge that pull request
due to the extra load on the Toolforge as well as the extra time the
request takes blocking the Scholia web application. We have around
100'000 requests per day according to Toolviews, - how much bot activity
I do not know. I am wondering whether there is anyone who can give us
advise here?
best regards
Finn Årup Nielsen
Hi,
I have setup a "rustup" tool on Toolforge that contains the latest Rust
toolchain for use by other tools so it doesn't need to be installed
individually in each tool.
Documentation on how to use it in your tool is available at
<https://wikitech.wikimedia.org/wiki/Tool:Rustup>, all you have to do is
add two lines to your tool's ~/.profile.
If you need an older version or some other components, let me know and
we can install it.
Hopefully this makes developing Rust tools just a bit easier :)
-- Legoktm
Hi all,
I have a python script that creates a tsv for the redirect table. I submitted the job through qsub and it's been waiting for more than a day now. Is it normal for the waiting time to be this long ? And, if there's a better way to do the aforementioned task, please let me know. Thank you in advance.
Best,
Anis
Hi,
Today 2021-11-02 we had a severe network outage on Cloud VPS and Toolforge.
Several network connections were affected from 11:40 UTC to 13:20 UTC (1h40m
duration). As of this writing the problem has been corrected.
Detailed information can be seen in Phabricator:
https://phabricator.wikimedia.org/T294853
Sorry for the inconvenience.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…