Starting Nov 7, a number of the jobs I would run through Toolforge grid
have stopped working. Each job consists of a .sh file like this
on the first line of which I use the source command to activate a python
virtual environment. When I run source by hand, subsequent lines work. But
when I call the .sh file and it tries to run the source command, I get a
"source: not found" message, the virtual environment does not get activated
and indeed running *which python* returns */usr/bin/python* which is bad.
All my scripts depend on pip packages that are installed in the virtual env
and not available with the system python.
The main thing I did on Nov 7 was to add a line at the end of my too's
account's .bash_profile as below:
This is because when I manually log into toolforge, I would like zsh to be
my shell, and since tool accounts don't support chsh, I thought executing
zsh directly from bash would be okay. But apparently, that now breaks the
source command somehow.
So I wonder:
(a) Is there a way to properly change the default shell of tool accounts?
(b) Is there a way to make *source* work under zsh?
Importantly, I know the problem is with *exec zsh* because once I removed
it and logged out and back in, all scripts worked correctly.
Dear Toolforge cloud people,
I am running the Scholia web application on Toolforge and interested in
have some of the pages indexed by search engines. We have '<meta
name="robots" content="index, nofollow">' which should index but not
crawl the Scholia website.
We have 3 kinds of content on the webpages generated by Scholia:
1) "Static" content generated from Flask jinja2 templates. These gets
indexed (but not that much).
2) Dynamic jQuery-based content based on the Wikidata API service. This
does not seem to get indexed by some search engines.
3) Dynamic Wikidata Query Serviece-based content. This does not get
I can understand 1) and 3), but not 2).
https://query.wikidata.org/robots.txt is blocking bots request for 3),
but as far as I can see https://www.wikidata.org/robots.txt does not
block Wikidata API requests for 2).
On a webpage on the public web, I have a link to
https://scholia.toolforge.org/author/Q20980928, so I would think that
that Scholia page would be indexed, and that the h1 tag that is set via
the Wikidata API would be indexed. As far as I can determine the page is
indexed at Bing and Quant, but not Duckduckgo and not Google.
I am wondering whether there is anyone that can explain the discrepancy?
As far as I understand Google does indeed index jQuery
Should we refrain from having bots getting into Scholia and define a
restrictive robots.txt to avoid burdening the Toolforge infrastructure
In Scholia, we at the moment have a pull request that implement
serverside calls to the Wikidata Query Service to generate some metadata
for the search engines that can be reached without hitting the WDQS
robots.txt restriction. I have been reluctant to merge that pull request
due to the extra load on the Toolforge as well as the extra time the
request takes blocking the Scholia web application. We have around
100'000 requests per day according to Toolviews, - how much bot activity
I do not know. I am wondering whether there is anyone who can give us
Finn Årup Nielsen
I have setup a "rustup" tool on Toolforge that contains the latest Rust
toolchain for use by other tools so it doesn't need to be installed
individually in each tool.
Documentation on how to use it in your tool is available at
<https://wikitech.wikimedia.org/wiki/Tool:Rustup>, all you have to do is
add two lines to your tool's ~/.profile.
If you need an older version or some other components, let me know and
we can install it.
Hopefully this makes developing Rust tools just a bit easier :)
I have a python script that creates a tsv for the redirect table. I submitted the job through qsub and it's been waiting for more than a day now. Is it normal for the waiting time to be this long ? And, if there's a better way to do the aforementioned task, please let me know. Thank you in advance.
Today 2021-11-02 we had a severe network outage on Cloud VPS and Toolforge.
Several network connections were affected from 11:40 UTC to 13:20 UTC (1h40m
duration). As of this writing the problem has been corrected.
Detailed information can be seen in Phabricator:
Sorry for the inconvenience.
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…