Hello!

Maps is a special case, being super gigantic[0] and basically on its own private NFS server already, so I'm not sure we're ready to move its storage onto Cinder.  That said, I'm going to answer your questions here for others who are following along:

- Is resizing straightforward?

Expanding storage pretty simple, there's a Horizon UI for resizing the volume and then you'd need to run resize2fs to expand the file system. Shrinking is not something we're likely to support.

- Will upfront allocation tie up resources?

Mostly not, cinder volumes are sparse so shouldn't consume physical resources until the space is consumed with actual files.

- Is the performance of Cinder substantially better than the /data/projects NFS?

We don't have a lot of experience to go on yet, but it should be much faster: roughly the same as any other 'local' VM storage as it's backed by the same Ceph cluster.

If you have a known non-terrifyingly-big subset of your data that you'd like to move off of NFS, please open a phabricator quota request and we'll see what we can do!  https://phabricator.wikimedia.org/project/view/2880/


[0] So big that I don't know how big it is because I ran 'du' and gave up after it sat there for 40 minutes


On 2/5/21 5:51 PM, Daniel Schwen wrote:
Neat,
I see the maps project has a 10GB quota on volumes. The tiles for the WikiMiniAtlas take up considerably more space and are currently held on /data/project. The appeal of /data/project is that it is a "infinite" pit to store data in, whereas it seems that with Cinder volumes I'd have to commit to a certain size a priori. Is resizing straightforward? Will upfront allocation tie up resources? Is the performance of Cinder substantially better than the /data/projects NFS?
Cheers,
Daniel

On Fri, Feb 5, 2021 at 4:32 PM Amir Sarabadani <ladsgroup@gmail.com> wrote:
That's amazing!!!! Thank you so much.

On Fri, Feb 5, 2021 at 5:02 PM Andrew Bogott <abogott@wikimedia.org> wrote:
Wikimedia Cloud Services now supports attachable block storage via the
OpenStack Cinder project. Attachable block storage is a flexible storage
option that allows you to create volumes local to your project but not
coupled to a particular VM; they can be moved between different
instances and persist after their associated volume is deleted. Project
admins can access this feature via the 'Volumes' tab in Horizon.

I encourage all of you to start using Cinder storage for your new
databases and large data sets. Over the next few months we'll be working
to move various use cases onto Cinder volumes and off of NFS or LVM;
soon I hope to deprecate large-storage flavor types entirely and support
all new non-root file storage with Cinder.


The default storage quota is quite small, but we plan to be generous
with quota increases. To request additional storage, open a phabricator
ticket here:

https://phabricator.wikimedia.org/project/view/2880/

For more details about this feature, I've written a blog post, here:

https://techblog.wikimedia.org/2021/02/05/cinder-on-cloud-vps/

And, technical documentation can be found here:

https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_instances#Cinder:_Attachable_Block_Storage_for_cloud-vps


Like any new feature, our implementation almost certainly includes bugs
and missteps. Please provide feedback or feature requests via
phabricator or on the cloud mailing list.

-Andrew + the WMCS Team


_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud


--
Amir (he/him)

_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud