-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
I'm currently copying user-store to a new disk array; this should be finished in a couple of days. I hope the new array will be faster, as well as much larger. There will be a separate maintenance notice for the actual switch later.
Once the move is done, in additional to having more storage now (about 14TB), it will be much easier to add additional storage in the future. However, I don't think having a single 14+ TB filesystem is a good idea (the initial volume will be 5TB), so I'm considering new ways to split up the storage.
The most obvious is to have one volume for miscellaneous files, like the current user-store, and then one volume per large project: stats, dumps, etc. People (or MMTs) who wanted additional storage for their project could request a new volume of a particular size. As well as making it easier for us to account for space, this would mean you were guaranteed to get the amount of storage you asked for, unlike the current situation where the filesystem can (and does) fill up.
Does this seem like something that would be useful to users? Otherwise, does anyone have another suggestions for how to allocate the space?
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
River Tarnell:
I'm currently copying user-store to a new disk array; this should be finished in a couple of days. I hope the new array will be faster, as well as much larger. There will be a separate maintenance notice for the actual switch later.
This is done now; in the end I did the switch without a maintenance. Some data is still copying (should be done in about 4 hours), so it'll be slower until then.
Once the move is done, in additional to having more storage now (about 14TB), it will be much easier to add additional storage in the future. However, I don't think having a single 14+ TB filesystem is a good idea (the initial volume will be 5TB), so I'm considering new ways to split up the storage.
Well, no one commented on this, so I propose the following:
* Dumps will go in /store/dumps/<wiki>/ * Page view stats will go in /store/stats/ * Everything else will go in /store/misc/
Symlinks from the old locations will prevent breaking tools which access data in its current location. Projects which need a lot of disk space (say, 100GB+) will be able to request a volume at /store/<whatever>/ for their files.
- river.
On 06/03/11 05:17, River Tarnell wrote:
Once the move is done, in additional to having more storage now (about 14TB), it will be much easier to add additional storage in the future. However, I don't think having a single 14+ TB filesystem is a good idea (the initial volume will be 5TB), so I'm considering new ways to split up the storage.
Well, no one commented on this, so I propose the following:
Sorry, I was off the Internet for the past few days.
- Dumps will go in /store/dumps/<wiki>/
- Page view stats will go in /store/stats/
- Everything else will go in /store/misc/
Symlinks from the old locations will prevent breaking tools which access data in its current location. Projects which need a lot of disk space (say, 100GB+) will be able to request a volume at /store/<whatever>/ for their files.
This looks good to me. I am ready to populate the stats system as soon as it is ready to be used. Have you already decided the size ? It could do with 3 Tb...
Frédéric
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
In article 4D74C74B.4010503@mathgen.ch, Frederic Schutz schutz@mathgen.ch wrote:
- Dumps will go in /store/dumps/<wiki>/
- Page view stats will go in /store/stats/
- Everything else will go in /store/misc/
This looks good to me. I am ready to populate the stats system as soon as it is ready to be used. Have you already decided the size ? It could do with 3 Tb...
Can you estimate the rate of growth for the stats? That would help in planning the size.
The change probably won't happen immediately, since our current (free) VxVM license only allows 4 volumes.
- river.
toolserver-l@lists.wikimedia.org