My job crunches /public/dumps to build a file (an LMDB database) which it publishes on behalf of other tools. However, on Toolforge Kubernetes, the tool runs about 20 times slower than on my laptop or in a commercial cloud. I wonder how come Toolforge is so awfully slow, and I/O seems like an obvious reason. Basically, my code inserts many millions of keys in random order into a btree in an mmap-ed file. This works super well on local disk or also on a remote Cinder or Ceph volume, but not so well over NFS.

— Sascha