Dear Wikimedia cloud support
What storage options does the Wikimedia cloud have? Can external developers (i.e. people not employed by the Wikimedia foundation) write to Cinder and/or Swift? Either from Toolforge or from Cloud VPS?
See below for context. (Actually, is this the right list, or should I ask elsewhere?)
For Wikidata QRank [https://qrank.toolforge.org/], I run a cronjob on the toolforge Kubernetes cluster. The cronjob mainly works on Wikidata dumps and anonymized Wikimedia access logs, which it reads from the NFS-mounted /public/dumps/public directory. Currently, the job produces 40 internal files with a total size of 21G; these files need to be preserved between individual cronjob runs. (In a forthcoming version of the cronjob, this will grow to ~200 files with a total size of ~40G). For storing these intermediate files, Cinder might be a good solution. However, afaik Cinder isn’t available on Toolforge. Therefore, I’m currently storing the intermediate files in the account’s home directory on NFS. Presumably (but not sure, but speculating because I’ve seen NFS crumbling elsewhere) Wikimedia’s NFS server will be easily overloaded; in any case, Wikimedia’s NFS server seems to protect itself by throttling access. Because of the throttling, the cronjob is slow when working with its intermediate files. * Will Cinder be made available to Toolforge users? When? * Or should I move from Toolforge to Cloud-VPS, so I can store my intermediate files on Cinder? * Or should I store my intermediate files in some object storage? Swift? Ceph? Something else? * Is access to Cinder and Swift subject to the same throttling as NFS? Or will moving away from NFS increase the available I/O throughput?
The final output of the QRank system is a single file, currently ~100M in size but eventually growing to ~1G. When the cronjob has computed a fresh version of its output, it deletes any old outputs from previous runs (with the exception of the previous last two versions, which are kept around internally for debugging). Typical users are other bots or external pipelines who need a signal for prioritizing Wikidata entities, not end users on the web. Users typically check for updates with HTTP HEAD, or with conditional HTTP GET requests (using the standard If-Modified-Since and If-None-Match headers). Currently, I’m serving the output file with a custom-written HTTP server that runs as a web service on Toolforge behind Toolforge’s nginx instance. My server reads its content from the NFS-mounted home directory that’s getting populated by the cronjob. Now, it’s not exactly a great idea to serve large data files from NFS, but afaik it’s the only option available in the Wikimedia cloud, at least for Toolforge users. Of course I might be wrong. * Should I move from Toolforge to Cloud-VPS, so I can serve my final output files from Cinder instead of NFS? * Or should I rather store my final output files in some object storage? Swift? Ceph? Something else? * Or is NFS just fine, even if the size of my data grows from 100M to 1G+?
The cronjob also uses ~5G of temporary files in /tmp, which it deletes towards the end of each run. The temp files are used for external sorting, so all access is sequential. I’m not sure where these temporary files currently sit when running on Toolforge Kubernetes. Given their volume, I presume that the tmpfs of the Kubernetes nodes will eventually run out of memory and then fall back to disk, but I wouldn’t know how to find this out. _If_ the backing store disk for tmpfs eventually ends up being mounted on NFS, it sounds wasteful for the poor NFS server;, especially since the files get deleted at job completion. In that case, I’d love to save common resources by using a local disk. (It doesn’t have to be an SSD; a spinning hard drive would be fine, given the sequential access pattern). But I’m not sure how to set this up on Toolforge Kubernetes, and I couldn’t find docs on wikitech. Actually, this might be a micro-optimization, so perhaps not worth the trouble. But then, I’d like to be nice with the precious shared resources in the Wikimedia cloud.
Sorry that I couldn’t find the answers online. While searching, I came across the following pointers: – https://wikitech.wikimedia.org/wiki/Ceph: This page has a warning that it’s probably “no longer true”. If the warning is correct, perhaps the page could be deleted entirely? Or maybe it could link to the current docs? – https://wikitech.wikimedia.org/wiki/Swift: This sounds perfect, but the page doesn’t mention how the files are getting populated, what the ACLs are managed, and if Wikimedia’s Swift cluster is even accessible to external developers. – https://wikitech.wikimedia.org/wiki/Media_storage: This seems current (I guess?), but the page doesn’t mention if/how external Toolforge/Cloud-VPS users may upload objects, or if this is just for the current users.
Thanks for your help, and happy holidays,
— Sascha, sascha@brawer.ch
Hi!
On 12/22/21 18:29, Sascha Brawer wrote:
What storage options does the Wikimedia cloud have? Can external developers (i.e. people not employed by the Wikimedia foundation) write to Cinder and/or Swift? Either from Toolforge or from Cloud VPS?
I've left more detailed replies inline. tl;dr: Currently Toolforge doesn't really have any other options than NFS. Cloud VPS additionally gives you the option to use Cinder (extra volumes you can attach to a VM and move from a VM to another).
See below for context. (Actually, is this the right list, or should I ask elsewhere?)
For Wikidata QRank [https://qrank.toolforge.org/ https://qrank.toolforge.org/], I run a cronjob on the toolforge Kubernetes cluster. The cronjob mainly works on Wikidata dumps and anonymized Wikimedia access logs, which it reads from the NFS-mounted /public/dumps/public directory. Currently, the job produces 40 internal files with a total size of 21G; these files need to be preserved between individual cronjob runs. (In a forthcoming version of the cronjob, this will grow to ~200 files with a total size of ~40G). For storing these intermediate files, Cinder might be a good solution. However, afaik Cinder isn’t available on Toolforge. Therefore, I’m currently storing the intermediate files in the account’s home directory on NFS. Presumably (but not sure, but speculating because I’ve seen NFS crumbling elsewhere) Wikimedia’s NFS server will be easily overloaded; in any case, Wikimedia’s NFS server seems to protect itself by throttling access. Because of the throttling, the cronjob is slow when working with its intermediate files.
- Will Cinder be made available to Toolforge users? When?
We're interested in it, but no-one has time or interest to work on making it a reality yet. This is tracked on Phabricator: https://phabricator.wikimedia.org/T275555.
As a reminder: if anyone is interested in working on this or other parts of the WMCS infrastructure, please talk to us!
- Or should I move from Toolforge to Cloud-VPS, so I can store my
intermediate files on Cinder?
~40G is in the range where Cinder/Cloud VPS might indeed be a better solution than NFS. While we don't currently have any official numbers on what is acceptable on NFS and what's not, for context the Toolforge project NFS cluster has currently about 8T of storage for about 3,000 tools.
- Or should I store my intermediate files in some object storage? Swift?
Ceph? Something else?
WMCS currently doesn't offer direct access to any object storage service. This is something we're likely to work on in the mid-term (next 6-12 months is the last estimate I've heard). This project is currently stalled on some network design work: https://phabricator.wikimedia.org/T289882.
- Is access to Cinder and Swift subject to the same throttling as
NFS? Or will moving away from NFS increase the available I/O throughput?
No, NFS is subject to completely separate throttling and Ceph-backed storage methods (local VM disks and Cinder volumes) have much higher amount of bandwidth available.
The final output of the QRank system is a single file, currently ~100M in size but eventually growing to ~1G. When the cronjob has computed a fresh version of its output, it deletes any old outputs from previous runs (with the exception of the previous last two versions, which are kept around internally for debugging). Typical users are other bots or external pipelines who need a signal for prioritizing Wikidata entities, not end users on the web. Users typically check for updates with HTTP HEAD, or with conditional HTTP GET requests (using the standard If-Modified-Since and If-None-Match headers). Currently, I’m serving the output file with a custom-written HTTP server that runs as a web service on Toolforge behind Toolforge’s nginx instance. My server reads its content from the NFS-mounted home directory that’s getting populated by the cronjob. Now, it’s not exactly a great idea to serve large data files from NFS, but afaik it’s the only option available in the Wikimedia cloud, at least for Toolforge users. Of course I might be wrong.
- Should I move from Toolforge to Cloud-VPS, so I can serve my final
output files from Cinder instead of NFS?
- Or should I rather store my final output files in some object storage?
Swift? Ceph? Something else?
- Or is NFS just fine, even if the size of my data grows from 100M to 1G+?
When we offer object storage, yes, storing your files in it is a good idea. I think you should be fine NFS for now (please don't quote me on that). Cloud VPS is an option too if you prefer it.
The cronjob also uses ~5G of temporary files in /tmp, which it deletes towards the end of each run. The temp files are used for external sorting, so all access is sequenyoutial. I’m not sure where these temporary files currently sit when running on Toolforge Kubernetes. Given their volume, I presume that the tmpfs of the Kubernetes nodes will eventually run out of memory and then fall back to disk, but I wouldn’t know how to find this out. _If_ the backing store disk for tmpfs eventually ends up being mounted on NFS, it sounds wasteful for the poor NFS server;, especially since the files get deleted at job completion. In that case, I’d love to save common resources by using a local disk. (It doesn’t have to be an SSD; a spinning hard drive would be fine, given the sequential access pattern). But I’m not sure how to set this up on Toolforge Kubernetes, and I couldn’t find docs on wikitech. Actually, this might be a micro-optimization, so perhaps not worth the trouble. But then, I’d like to be nice with the precious shared resources in the Wikimedia cloud.
Good question, I'm not sure either if tmpfs for Kubernetes containers is on Ceph (SSDs) or on RAM. At least it's not on NFS.
Sorry that I couldn’t find the answers online. While searching, I came across the following pointers: – https://wikitech.wikimedia.org/wiki/Ceph https://wikitech.wikimedia.org/wiki/Ceph: This page has a warning that it’s probably “no longer true”. If the warning is correct, perhaps the page could be deleted entirely? Or maybe it could link to the current docs? – https://wikitech.wikimedia.org/wiki/Swift https://wikitech.wikimedia.org/wiki/Swift: This sounds perfect, but the page doesn’t mention how the files are getting populated, what the ACLs are managed, and if Wikimedia’s Swift cluster is even accessible to external developers. – https://wikitech.wikimedia.org/wiki/Media_storage https://wikitech.wikimedia.org/wiki/Media_storage: This seems current (I guess?), but the page doesn’t mention if/how external Toolforge/Cloud-VPS users may upload objects, or if this is just for the current users.
Those pages document the media storage systems used to store uploads for the production MediaWiki projects (Wikipedia and friends). Those are not accessible from WMCS and should be treated as completely separate systems, and any future WMCS (object) storage services will not use them.
Documentation about the Ceph cluster powering Cloud VPS is on a separate Wikitech page: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Ceph.
Hi Taavi, thanks for your reply, it’s super helpful! I’ll give Cloud VPS a try.
Documentation about the Ceph cluster powering Cloud VPS is on a separate
Wikitech page:
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Ceph.
Curious, if Cloud VPS already has a working Ceph cluster, might it be possible to run Ceph’s built-in object store? https://docs.ceph.com/en/pacific/radosgw/
— Sascha
Am Mi., 22. Dez. 2021 um 18:41 Uhr schrieb Taavi Väänänen hi@taavi.wtf:
Hi!
On 12/22/21 18:29, Sascha Brawer wrote:
What storage options does the Wikimedia cloud have? Can external developers (i.e. people not employed by the Wikimedia foundation) write to Cinder and/or Swift? Either from Toolforge or from Cloud VPS?
I've left more detailed replies inline. tl;dr: Currently Toolforge doesn't really have any other options than NFS. Cloud VPS additionally gives you the option to use Cinder (extra volumes you can attach to a VM and move from a VM to another).
See below for context. (Actually, is this the right list, or should I ask elsewhere?)
For Wikidata QRank [https://qrank.toolforge.org/ https://qrank.toolforge.org/], I run a cronjob on the toolforge Kubernetes cluster. The cronjob mainly works on Wikidata dumps and anonymized Wikimedia access logs, which it reads from the NFS-mounted /public/dumps/public directory. Currently, the job produces 40 internal files with a total size of 21G; these files need to be preserved between individual cronjob runs. (In a forthcoming version of the cronjob, this will grow to ~200 files with a total size of ~40G). For storing these intermediate files, Cinder might be a good solution. However, afaik Cinder isn’t available on Toolforge. Therefore, I’m currently storing the intermediate files in the account’s home directory on NFS. Presumably (but not sure, but speculating because I’ve seen NFS crumbling elsewhere) Wikimedia’s NFS server will be easily overloaded; in any case, Wikimedia’s NFS server seems to protect itself by throttling access. Because of the throttling, the cronjob is slow when working with its intermediate files.
- Will Cinder be made available to Toolforge users? When?
We're interested in it, but no-one has time or interest to work on making it a reality yet. This is tracked on Phabricator: https://phabricator.wikimedia.org/T275555.
As a reminder: if anyone is interested in working on this or other parts of the WMCS infrastructure, please talk to us!
- Or should I move from Toolforge to Cloud-VPS, so I can store my
intermediate files on Cinder?
~40G is in the range where Cinder/Cloud VPS might indeed be a better solution than NFS. While we don't currently have any official numbers on what is acceptable on NFS and what's not, for context the Toolforge project NFS cluster has currently about 8T of storage for about 3,000 tools.
- Or should I store my intermediate files in some object storage? Swift?
Ceph? Something else?
WMCS currently doesn't offer direct access to any object storage service. This is something we're likely to work on in the mid-term (next 6-12 months is the last estimate I've heard). This project is currently stalled on some network design work: https://phabricator.wikimedia.org/T289882.
- Is access to Cinder and Swift subject to the same throttling as
NFS? Or will moving away from NFS increase the available I/O throughput?
No, NFS is subject to completely separate throttling and Ceph-backed storage methods (local VM disks and Cinder volumes) have much higher amount of bandwidth available.
The final output of the QRank system is a single file, currently ~100M in size but eventually growing to ~1G. When the cronjob has computed a fresh version of its output, it deletes any old outputs from previous runs (with the exception of the previous last two versions, which are kept around internally for debugging). Typical users are other bots or external pipelines who need a signal for prioritizing Wikidata entities, not end users on the web. Users typically check for updates with HTTP HEAD, or with conditional HTTP GET requests (using the standard If-Modified-Since and If-None-Match headers). Currently, I’m serving the output file with a custom-written HTTP server that runs as a web service on Toolforge behind Toolforge’s nginx instance. My server reads its content from the NFS-mounted home directory that’s getting populated by the cronjob. Now, it’s not exactly a great idea to serve large data files from NFS, but afaik it’s the only option available in the Wikimedia cloud, at least for Toolforge users. Of course I might be
wrong.
- Should I move from Toolforge to Cloud-VPS, so I can serve my final
output files from Cinder instead of NFS?
- Or should I rather store my final output files in some object storage?
Swift? Ceph? Something else?
- Or is NFS just fine, even if the size of my data grows from 100M to
1G+?
When we offer object storage, yes, storing your files in it is a good idea. I think you should be fine NFS for now (please don't quote me on that). Cloud VPS is an option too if you prefer it.
The cronjob also uses ~5G of temporary files in /tmp, which it deletes towards the end of each run. The temp files are used for external sorting, so all access is sequenyoutial. I’m not sure where these
temporary
files currently sit when running on Toolforge Kubernetes. Given their volume, I presume that the tmpfs of the Kubernetes nodes will eventually run out of memory and then fall back to disk, but I wouldn’t know how to find this out. _If_ the backing store disk for tmpfs eventually ends up being mounted on NFS, it sounds wasteful for the poor NFS server;, especially since the files get deleted at job completion. In that case, I’d love to save common resources by using a local disk. (It doesn’t have to be an SSD; a spinning hard drive would be fine, given the sequential access pattern). But I’m not sure how to set this up on Toolforge Kubernetes, and I couldn’t find docs on wikitech. Actually, this might be a micro-optimization, so perhaps not worth the trouble. But then, I’d like to be nice with the precious shared resources in the Wikimedia cloud.
Good question, I'm not sure either if tmpfs for Kubernetes containers is on Ceph (SSDs) or on RAM. At least it's not on NFS.
Sorry that I couldn’t find the answers online. While searching, I came across the following pointers: – https://wikitech.wikimedia.org/wiki/Ceph https://wikitech.wikimedia.org/wiki/Ceph: This page has a warning
that
it’s probably “no longer true”. If the warning is correct, perhaps the page could be deleted entirely? Or maybe it could link to the current docs? – https://wikitech.wikimedia.org/wiki/Swift https://wikitech.wikimedia.org/wiki/Swift: This sounds perfect, but the page doesn’t mention how the files are getting populated, what the ACLs are managed, and if Wikimedia’s Swift cluster is even accessible to external developers. – https://wikitech.wikimedia.org/wiki/Media_storage https://wikitech.wikimedia.org/wiki/Media_storage: This seems current (I guess?), but the page doesn’t mention if/how external Toolforge/Cloud-VPS users may upload objects, or if this is just for the current users.
Those pages document the media storage systems used to store uploads for the production MediaWiki projects (Wikipedia and friends). Those are not accessible from WMCS and should be treated as completely separate systems, and any future WMCS (object) storage services will not use them.
Documentation about the Ceph cluster powering Cloud VPS is on a separate Wikitech page: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Ceph.
-- Taavi (User:Majavah) volunteer Toolforge/Cloud VPS admin
Hi Sasha,
As of right now it's not possible to access the ceph cluster from the Cloud VPS instances directly, that includes Rados Gateways (rgw). That is the reason why we are trying to setup swift on top of it instead, but that's still on the works, and got de-prioitized due to changes in the team and the foundation that made us focus on maintenance work instead. We have it on our minds, but it's not likely that in the short term we'd be able to get it going.
Please add a comment on the tasks that Taavï mentioned with your use cases, that will help prioritizing and making sure it stays in the focus.
Thanks!
On Thu, 23 Dec 2021, 07:38 Sascha Brawer, sascha@brawer.ch wrote:
Hi Taavi, thanks for your reply, it’s super helpful! I’ll give Cloud VPS a try.
Documentation about the Ceph cluster powering Cloud VPS is on a separate
Wikitech page:
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Ceph.
Curious, if Cloud VPS already has a working Ceph cluster, might it be possible to run Ceph’s built-in object store? https://docs.ceph.com/en/pacific/radosgw/
— Sascha
Am Mi., 22. Dez. 2021 um 18:41 Uhr schrieb Taavi Väänänen hi@taavi.wtf:
Hi!
On 12/22/21 18:29, Sascha Brawer wrote:
What storage options does the Wikimedia cloud have? Can external developers (i.e. people not employed by the Wikimedia foundation) write to Cinder and/or Swift? Either from Toolforge or from Cloud VPS?
I've left more detailed replies inline. tl;dr: Currently Toolforge doesn't really have any other options than NFS. Cloud VPS additionally gives you the option to use Cinder (extra volumes you can attach to a VM and move from a VM to another).
See below for context. (Actually, is this the right list, or should I ask elsewhere?)
For Wikidata QRank [https://qrank.toolforge.org/ https://qrank.toolforge.org/], I run a cronjob on the toolforge Kubernetes cluster. The cronjob mainly works on Wikidata dumps and anonymized Wikimedia access logs, which it reads from the NFS-mounted /public/dumps/public directory. Currently, the job produces 40 internal files with a total size of 21G; these files need to be preserved between individual cronjob runs. (In a forthcoming version of the cronjob, this will grow to ~200 files with a total size of ~40G). For storing these intermediate files, Cinder might be a good solution. However, afaik Cinder isn’t available on Toolforge. Therefore, I’m currently storing the intermediate files in the account’s home
directory
on NFS. Presumably (but not sure, but speculating because I’ve seen NFS crumbling elsewhere) Wikimedia’s NFS server will be easily overloaded; in any case, Wikimedia’s NFS server seems to protect itself by throttling access. Because of the throttling, the cronjob is slow when working with its intermediate files.
- Will Cinder be made available to Toolforge users? When?
We're interested in it, but no-one has time or interest to work on making it a reality yet. This is tracked on Phabricator: https://phabricator.wikimedia.org/T275555.
As a reminder: if anyone is interested in working on this or other parts of the WMCS infrastructure, please talk to us!
- Or should I move from Toolforge to Cloud-VPS, so I can store my
intermediate files on Cinder?
~40G is in the range where Cinder/Cloud VPS might indeed be a better solution than NFS. While we don't currently have any official numbers on what is acceptable on NFS and what's not, for context the Toolforge project NFS cluster has currently about 8T of storage for about 3,000 tools.
- Or should I store my intermediate files in some object storage?
Swift?
Ceph? Something else?
WMCS currently doesn't offer direct access to any object storage service. This is something we're likely to work on in the mid-term (next 6-12 months is the last estimate I've heard). This project is currently stalled on some network design work: https://phabricator.wikimedia.org/T289882.
- Is access to Cinder and Swift subject to the same throttling as
NFS? Or will moving away from NFS increase the available I/O throughput?
No, NFS is subject to completely separate throttling and Ceph-backed storage methods (local VM disks and Cinder volumes) have much higher amount of bandwidth available.
The final output of the QRank system is a single file, currently ~100M in size but eventually growing to ~1G. When the cronjob has computed a fresh version of its output, it deletes any old outputs from previous runs (with the exception of the previous last two versions, which are kept around internally for debugging). Typical users are other bots or external pipelines who need a signal for prioritizing Wikidata
entities,
not end users on the web. Users typically check for updates with HTTP HEAD, or with conditional HTTP GET requests (using the standard If-Modified-Since and If-None-Match headers). Currently, I’m serving
the
output file with a custom-written HTTP server that runs as a web
service
on Toolforge behind Toolforge’s nginx instance. My server reads its content from the NFS-mounted home directory that’s getting populated by the cronjob. Now, it’s not exactly a great idea to serve large data files from NFS, but afaik it’s the only option available in the Wikimedia cloud, at least for Toolforge users. Of course I might be
wrong.
- Should I move from Toolforge to Cloud-VPS, so I can serve my final
output files from Cinder instead of NFS?
- Or should I rather store my final output files in some object
storage?
Swift? Ceph? Something else?
- Or is NFS just fine, even if the size of my data grows from 100M to
1G+?
When we offer object storage, yes, storing your files in it is a good idea. I think you should be fine NFS for now (please don't quote me on that). Cloud VPS is an option too if you prefer it.
The cronjob also uses ~5G of temporary files in /tmp, which it deletes towards the end of each run. The temp files are used for external sorting, so all access is sequenyoutial. I’m not sure where these
temporary
files currently sit when running on Toolforge Kubernetes. Given their volume, I presume that the tmpfs of the Kubernetes nodes will
eventually
run out of memory and then fall back to disk, but I wouldn’t know how
to
find this out. _If_ the backing store disk for tmpfs eventually ends up being mounted on NFS, it sounds wasteful for the poor NFS server;, especially since the files get deleted at job completion. In that case, I’d love to save common resources by using a local disk. (It doesn’t have to be an SSD; a spinning hard drive would be fine, given the sequential access pattern). But I’m not sure how to set this up on Toolforge Kubernetes, and I couldn’t find docs on wikitech. Actually, this might be a micro-optimization, so perhaps not worth the trouble. But then, I’d like to be nice with the precious shared resources in the Wikimedia cloud.
Good question, I'm not sure either if tmpfs for Kubernetes containers is on Ceph (SSDs) or on RAM. At least it's not on NFS.
Sorry that I couldn’t find the answers online. While searching, I came across the following pointers: – https://wikitech.wikimedia.org/wiki/Ceph https://wikitech.wikimedia.org/wiki/Ceph: This page has a warning
that
it’s probably “no longer true”. If the warning is correct, perhaps the page could be deleted entirely? Or maybe it could link to the current docs? – https://wikitech.wikimedia.org/wiki/Swift https://wikitech.wikimedia.org/wiki/Swift: This sounds perfect, but the page doesn’t mention how the files are getting populated, what the ACLs are managed, and if Wikimedia’s Swift cluster is even accessible
to
external developers. – https://wikitech.wikimedia.org/wiki/Media_storage https://wikitech.wikimedia.org/wiki/Media_storage: This seems current (I guess?), but the page doesn’t mention if/how external Toolforge/Cloud-VPS users may upload objects, or if this is just for
the
current users.
Those pages document the media storage systems used to store uploads for the production MediaWiki projects (Wikipedia and friends). Those are not accessible from WMCS and should be treated as completely separate systems, and any future WMCS (object) storage services will not use them.
Documentation about the Ceph cluster powering Cloud VPS is on a separate Wikitech page: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Ceph.
-- Taavi (User:Majavah) volunteer Toolforge/Cloud VPS admin
Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/