Hello, all!
We are in the process of re-engineering and virtualizing[0] the NFS
service provided to Toolforge and VMs. The transition will be rocky and
involve some service interruption... I'm still running tests to
determine exactly host much disruption will be required.
The first volume that I'd like to replace is 'scratch,' typically
mounted as /mnt/nfs/secondary-scratch. I'm seeking feedback about how
vital scratch uptime is to your current workflow, and how disruptive it
would be to lose data there.
If you have a project or tool that uses scratch, please respond with
your thoughts! My preference would be to wipe out all existing data on
scratch and also subject users to several unannounced periods of
downtime, but I also don't want anyone to suffer. If you have
important/persistent data on that volume then the WMCS team will work
with you to migrate that data somewhere safer, and if you have an
important workflow that will break due to Scratch downtime then I'll
work harder on devising a low-impact roll-out.
Thank you!
-Andrew
[0] https://phabricator.wikimedia.org/T291405
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
I'm exploring using elasticsearch in Toolforge. I'm running Python 3.7, with elasticsearch==7.16.2 installed. When I run:
def main():
es = Elasticsearch(['elasticsearch.svc.tools.eqiad1.wikimedia.cloud:80'])
print(es.info())
I get:
> elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch
Any idea what's wrong? Are we running some very old version of elasticsearch server?
I'm starting on a project which will process every edit comment in the enwiki dumps using k8s jobs on toolforge. There's 736
enwiki-20211201-pages-meta-history*.xml*bz2 files. Would kicking off 736 jobs (one per file) be a reasonable thing to do from the resource consumption point of view, or would that lead to a WTF response from the sysadmins?
Hello cloud-vps users,
There are still about 84 unclaimed projects at
https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge
Please take a moment to look at that page and mark projects that you are
using.
Unclaimed projects will be in danger of shutdown on February 1st, 2022.
Thank you to those of you who have already acted on this.
Thank you!
- Komla
-------- Forwarded Message --------
Subject: Cloud VPS users, please claim your projects (and, introducing
Komla)
Date: Thu, 2 Dec 2021 14:42:08 -0600
From: Andrew Bogott <abogott(a)wikimedia.org> <abogott(a)wikimedia.org>
Reply-To: abogott(a)wikimedia.org
Organization: The Wikimedia Foundation
To: Cloud-announce(a)lists.wikimedia.org
CC: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>
<wikitech-l(a)lists.wikimedia.org>
Hello cloud-vps users!
It's time for our annual cleanup of unused projects and resources. Our new
developer advocate Komla Sapaty will be guiding this process; please
respond promptly to his emails and do your best to make him feel welcome!
Every year or so the Cloud Services team tries to identify and clean up
unused projects and VMs. We do this via an opt-in process: anyone can mark
a project as 'in use,' and that project will be preserved for another year.
I've created a wiki page that lists all existing projects, here:
https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge
If you are a VPS user, please visit that page and mark any projects that
you use as {{Used}}. Note that it's not necessary for you to be a project
admin to mark something -- if you know that you're currently using a
resource and want to keep using it, go ahead and mark it accordingly. If
you /are/ a project admin, please take a moment to mark which VMs are or
aren't used in your projects.
When February arrives, I will shut down and begin the process of reclaiming
resources from unused projects.
If you think you use a VPS project but aren't sure which, I encourage you
to poke around on https://tools.wmflabs.org/openstack-browser/ to see what
looks familiar. Worst case, just email cloud(a)lists.wikimedia.org with a
description of your use case and we'll sort it out there.
Exclusive toolforge users are free to ignore this email and future related
things.
Thank you!
-Andrew and the WMCS team
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Hi all,
I had previously found a way to mount the home directory of my tool on
Toolforge onto my local Linux machine using a *mount* command. It was so
handy that I turned it into a bash alias and used it on the daily.
Sadly, due to a hardware malfunction, I lost my machine and my
.bash_aliases file with it. I cannot find the mount command anywhere on
Wikitech wiki either. Do you happen to know what the command should look
like?
It is just so much more convenient to mount the drive and use local IDEs to
modify code.
PS: I promise to add it to the wiki so others can use it
Thanks,
Huji
Dear Wikimedia cloud support
What storage options does the Wikimedia cloud have? Can external developers
(i.e. people not employed by the Wikimedia foundation) write to Cinder
and/or Swift? Either from Toolforge or from Cloud VPS?
See below for context. (Actually, is this the right list, or should I ask
elsewhere?)
For Wikidata QRank [https://qrank.toolforge.org/], I run a cronjob on the
toolforge Kubernetes cluster. The cronjob mainly works on Wikidata dumps
and anonymized Wikimedia access logs, which it reads from the NFS-mounted
/public/dumps/public directory. Currently, the job produces 40 internal
files with a total size of 21G; these files need to be preserved between
individual cronjob runs. (In a forthcoming version of the cronjob, this
will grow to ~200 files with a total size of ~40G). For storing these
intermediate files, Cinder might be a good solution. However, afaik Cinder
isn’t available on Toolforge. Therefore, I’m currently storing the
intermediate files in the account’s home directory on NFS. Presumably (but
not sure, but speculating because I’ve seen NFS crumbling elsewhere)
Wikimedia’s NFS server will be easily overloaded; in any case, Wikimedia’s
NFS server seems to protect itself by throttling access. Because of the
throttling, the cronjob is slow when working with its intermediate files.
* Will Cinder be made available to Toolforge users? When?
* Or should I move from Toolforge to Cloud-VPS, so I can store my
intermediate files on Cinder?
* Or should I store my intermediate files in some object storage? Swift?
Ceph? Something else?
* Is access to Cinder and Swift subject to the same throttling as NFS? Or
will moving away from NFS increase the available I/O throughput?
The final output of the QRank system is a single file, currently ~100M in
size but eventually growing to ~1G. When the cronjob has computed a fresh
version of its output, it deletes any old outputs from previous runs (with
the exception of the previous last two versions, which are kept around
internally for debugging). Typical users are other bots or external
pipelines who need a signal for prioritizing Wikidata entities, not end
users on the web. Users typically check for updates with HTTP HEAD, or with
conditional HTTP GET requests (using the standard If-Modified-Since and
If-None-Match headers). Currently, I’m serving the output file with a
custom-written HTTP server that runs as a web service on
Toolforge behind Toolforge’s nginx instance. My server reads its content
from the NFS-mounted home directory that’s getting populated by the
cronjob. Now, it’s not exactly a great idea to serve large data files from
NFS, but afaik it’s the only option available in the Wikimedia cloud, at
least for Toolforge users. Of course I might be wrong.
* Should I move from Toolforge to Cloud-VPS, so I can serve my final output
files from Cinder instead of NFS?
* Or should I rather store my final output files in some object storage?
Swift? Ceph? Something else?
* Or is NFS just fine, even if the size of my data grows from 100M to 1G+?
The cronjob also uses ~5G of temporary files in /tmp, which it deletes
towards the end of each run. The temp files are used for external sorting,
so all access is sequential. I’m not sure where these temporary files
currently sit when running on Toolforge Kubernetes. Given their volume, I
presume that the tmpfs of the Kubernetes nodes will eventually run out of
memory and then fall back to disk, but I wouldn’t know how to find this
out. _If_ the backing store disk for tmpfs eventually ends up being mounted
on NFS, it sounds wasteful for the poor NFS server;, especially since the
files get deleted at job completion. In that case, I’d love to save common
resources by using a local disk. (It doesn’t have to be an SSD; a spinning
hard drive would be fine, given the sequential access pattern). But I’m not
sure how to set this up on Toolforge Kubernetes, and I couldn’t find docs
on wikitech. Actually, this might be a micro-optimization, so perhaps not
worth the trouble. But then, I’d like to be nice with the precious shared
resources in the Wikimedia cloud.
Sorry that I couldn’t find the answers online. While searching, I came
across the following pointers:
– https://wikitech.wikimedia.org/wiki/Ceph: This page has a warning that
it’s probably “no longer true”. If the warning is correct, perhaps the page
could be deleted entirely? Or maybe it could link to the current docs?
– https://wikitech.wikimedia.org/wiki/Swift: This sounds perfect, but the
page doesn’t mention how the files are getting populated, what the ACLs are
managed, and if Wikimedia’s Swift cluster is even accessible to external
developers.
– https://wikitech.wikimedia.org/wiki/Media_storage: This seems current (I
guess?), but the page doesn’t mention if/how external Toolforge/Cloud-VPS
users may upload objects, or if this is just for the current users.
Thanks for your help, and happy holidays,
— Sascha, sascha(a)brawer.ch
Hello everyone
We have noticed that since Septembrer 2020, our WLM-related site
www.wikilm.es has been receiving a couple of meta=siteinfo api queries per
day from Cloud VPS NAT egress 185.15.56.1 with a user agent of...
${user_agent}
(yes, literally that. Probably a case of single quotes when double were
needed)
It's probably some kind of Wiki Loves Monuments statistics tool. I have
looked on some suspects with no luck.
Anyone has an idea which tool may be doing that?
Regards