Going under the assumption that we will be buying a new db server to replace geoffrin, please everyone take a look at: http://www.siliconmechanics.com/
and help me decide on a suitable configuration. Assume a budget of $9,000-$10,000, but don't hesitate to go under (of course!) or over (if for a good reason!).
There have been several requests (from more than one person, or just from one person) for 15k spindle drives. Is that a universal consensus? I find it hard to imagine that faster drives is really pertinent, because just about any problem that can be solved with faster drives can be solved better with more ram?
But, that's a very simplistic view on my part, so I could easily be convinced otherwise.
RAID 10? That's what geoffrin had, and it was sweet. But RAID 5 would give more disk space, right?
What are our disk requirements for the db server? Let's be generous with ourselves here. But, also let's think about whether disk space needs might in some cases be better handled by a big slow cheap SATA storage server?
If we're in need of other new hardware, now is a good time to order that as well.
We have a crucial responsibility to be good stewards of our donor's money. That means two things -- first, we don't buy hardware just because it sounds cool, but to meet specific needs. But second, the money wasn't given to us to sit in the bank forever, it's to give the donors what they want -- a fast fast wikipedia. So we should not be cheapskates, either.
--Jimbo
p.s. Quotes or recommendations for other vendors are welcome, but I'd like to take action quickly, so it would be good to talk about configurations from a single site (siliconmechanics) and then also look around for better pricing or service or ??? elsewhere.
Hi,
There have been several requests (from more than one person, or just from one person) for 15k spindle drives. Is that a universal consensus? I find it hard to imagine that faster drives is really pertinent, because just about any problem that can be solved with faster drives can be solved better with more ram?
In theory: yes. But if more RAM or faster UPM are better should be tested, this belongs to the kind of application.
Don't you/we have a dealer who allows to test this on a new server? This could help us to proof our suggestions instead of speculating.
RAID 10? That's what geoffrin had, and it was sweet. But RAID 5 would give more disk space, right?
Yes. RAID 10 gives you 50% of the hard disc capacity while RAID 5 gives you 66%. That sounds not too much difference but it is :-) 8 drives of 150 GB -> 1200 GB gross capacity. RAID 10: 600 GB net capacity. RAID 5: about 792 GB net capacity. That's more than one extra hard drive for "free".
RAID 10 is a bit faster than RAID 5.
I would recommend two or three spare hard drives, not only one. If one's dying in the RAID and the only hotspare drives begins to rebuild the data (which could take quite a long time) and before someone is able to replace the broken one a second hard drive gets corrupted then the RAID is lost. This is of no importance if someone is 24/7 near the server room.
What are our disk requirements for the db server? Let's be generous with ourselves here. But, also let's think about whether disk space needs might in some cases be better handled by a big slow cheap SATA storage server?
SATA-RAIDs are as fast as SCSI-RAIDs concerning the sustained read benchmarks but not when accessing random data - this is what I measured for our firm here.
BTW: I don't know siliconmechanics (and can't reach their website now) but I was very content with Fujitsu-Siemens Primergy servers. We're running a lot of them. No, I don't have stock options or something like that from them and we don't sell them (we don't sell something computer related at all).
YMMV.
Sorry for my bad English, I'm not a native speaker.
Regards Götz
On Wed, 7 Apr 2004, [ISO-8859-1] G�tz Hoffart wrote:
pertinent, because just about any problem that can be solved with faster drives can be solved better with more ram?
In theory: yes. But if more RAM or faster UPM are better should be tested, this belongs to the kind of application.
Actually, NO. This is a DATABASE SERVER. Databases flush data to disk to ensure data integrity. No ammount of RAM can speed up the process of an fsync() as the data must physically be moved from system memory to the physical drives.
Yes. RAID 10 gives you 50% of the hard disc capacity while RAID 5 gives you 66%. That sounds not too much difference but it is :-) 8 drives of 150 GB -> 1200 GB gross capacity. RAID 10: 600 GB net capacity. RAID 5: about 792 GB net capacity. That's more than one extra hard drive for "free".
Your math is off... RAID 10 == 50% always. RAID 5 == N-1 (minus any spares) So, for 8 drives, at best, that's 8-1 == 7 usable drives which is just over 1TB. With a spare, that drops to 900GB.
RAID 10 is a bit faster than RAID 5.
For pure reads, mirroring is better. But in write performance, mirroring sucks big time. RAID 5 is somewhere in between... it takes less drive space in trade for a marginal cost in speed. (reads are at single drive speeds, writes require a block read and two block writes for every block written -- read the parity block, calculate, and the store the data and parity blocks)
I would recommend two or three spare hard drives, not only one.
...
This is merely a waste of drives. If a second drive fails before the recovery is complete, the array will fail. A RAID 5 array can only function with a SINGLE drive failure ("degraded".) Until recovery is complete, the array is in a degraded state.
Multiple drive failure recovery is part of RAID 6 (n-m + spares) which is still experimental in linux -- and I don't think anyone has tried it with more than 2 parity strips.
SATA-RAIDs are as fast as SCSI-RAIDs concerning the sustained read benchmarks but not when accessing random data - this is what I measured for our firm here.
No they aren't. At least not the one's generally available (read: the cheap crap found at retail outlets.) People often forget about Fibre Channel. I have 5 year old FC (FC-1 1G) drives that hands down beat the fastest, modern IDE and SATA drives. (Cost-wise (new), they aren't worth it.) Plus, FC can expand to include >100 drives without any further hardware (aside from the drives and shelves, of course.)
Again, I'll add my strong recommendation for 3Ware raid contoller cards (esp. if you're using IDE or SATA drives.)
--Ricky
PS: I still have the bonnie++ results from my testing of raid5 + various filesystems on my "junk" -- a qla2100, a eurologic FC7 FC shelf, and 7 10k RPM drives (seagate ST118202FC) If anyone is interested, I can post the numbers.
On Thu, 08 Apr 2004 14:40:34 -0400, Ricky Beam wrote:
Actually, NO. This is a DATABASE SERVER. Databases flush data to disk to ensure data integrity. No ammount of RAM can speed up the process of an fsync() as the data must physically be moved from system memory to the physical drives.
Most disk access on suda is repeated reading. A bi of >6000 is common. Bo is much lower. Cpus are mostly idle, the thing is completely i/o bound currently.
On Thu, 8 Apr 2004, Gabriel Wicke wrote:
On Thu, 08 Apr 2004 14:40:34 -0400, Ricky Beam wrote:
Actually, NO. This is a DATABASE SERVER. Databases flush data to disk to ensure data integrity. No ammount of RAM can speed up the process of an fsync() as the data must physically be moved from system memory to the physical drives.
Most disk access on suda is repeated reading. A bi of >6000 is common. Bo is much lower. Cpus are mostly idle, the thing is completely i/o bound currently.
[[meta:Wikimedia_servers]] says that suda is a dual opteron with 72 GB SCSI raid (level 1 I suppose). Is this information correct? It doesn't seem a slow disk configuration
Alfio
On Fri, 9 Apr 2004 12:12:14 +0200 (CEST) Alfio Puglisi puglisi@arcetri.astro.it wrote:
On Thu, 8 Apr 2004, Gabriel Wicke wrote:
On Thu, 08 Apr 2004 14:40:34 -0400, Ricky Beam wrote:
Actually, NO. This is a DATABASE SERVER. Databases flush data to disk to ensure data integrity. No ammount of RAM can speed up the process of an fsync() as the data must physically be moved from system memory to the physical drives.
Most disk access on suda is repeated reading. A bi of >6000 is common. Bo is much lower. Cpus are mostly idle, the thing is completely i/o bound currently.
[[meta:Wikimedia_servers]] says that suda is a dual opteron with 72 GB SCSI raid (level 1 I suppose). Is this information correct? It doesn't seem a slow disk configuration
it's raid 5, 3 x 36GB drive. A small and slow disk configuration
Shaihulud
On Fri, 9 Apr 2004, Camille Constans wrote:
On Fri, 9 Apr 2004 12:12:14 +0200 (CEST) Alfio Puglisi puglisi@arcetri.astro.it wrote:
On Thu, 8 Apr 2004, Gabriel Wicke wrote:
On Thu, 08 Apr 2004 14:40:34 -0400, Ricky Beam wrote:
Actually, NO. This is a DATABASE SERVER. Databases flush data to disk to ensure data integrity. No ammount of RAM can speed up the process of an fsync() as the data must physically be moved from system memory to the physical drives.
Most disk access on suda is repeated reading. A bi of >6000 is common. Bo is much lower. Cpus are mostly idle, the thing is completely i/o bound currently.
[[meta:Wikimedia_servers]] says that suda is a dual opteron with 72 GB SCSI raid (level 1 I suppose). Is this information correct? It doesn't seem a slow disk configuration
it's raid 5, 3 x 36GB drive. A small and slow disk configuration
Ouch. If the current controller is capable, i would invest the money in multiple 73GB 15k rpm disks, say 5 of them in raid5 to have around 280 GB of fast storage. Or, 4 of them in raid 1 to have 140 GB of safer storage.
Alfio
On Tue, 6 Apr 2004 22:31:02 -0700 Jimmy Wales jwales@bomis.com wrote:
Going under the assumption that we will be buying a new db server to replace geoffrin, please everyone take a look at: http://www.siliconmechanics.com/
and help me decide on a suitable configuration. Assume a budget of $9,000-$10,000, but don't hesitate to go under (of course!) or over (if for a good reason!).
There have been several requests (from more than one person, or just from one person) for 15k spindle drives. Is that a universal consensus? I find it hard to imagine that faster drives is really pertinent, because just about any problem that can be solved with faster drives can be solved better with more ram?
I hope I'm not the only one person would wish 15000 rpm disks :) I understand that they are more expensive, but our main problem in database is access time, and I dont think we can cache all the db in memory. On Suda, mysql use only 1,5G of memory (cause 32bits adressing pb IIRC), and it depends heavily on access time of disks. Cause of its slow disks, developper had to stop SQL query, search query and others stuffs. We're back in time before Geoffrin :( Having 15000 rpm disks, could let us some quiet time before thinking about a new upgrade. And it's not a cpu pb, cpu is idle most of time on Suda.
But, that's a very simplistic view on my part, so I could easily be convinced otherwise.
RAID 10? That's what geoffrin had, and it was sweet. But RAID 5 would give more disk space, right?
Raid10 is faster. I calculate, that the db is growing at around 1GBytes per week, So it could be an idea to have raid5 but with more than 3 disks (4 or 5 ?). Raid 10 with 15000 rpms is a pb cause we wont be able to get more space than 140 G with 4 disks (which should be enough for a year). So we could choose to have a similar setup than Geoffrin, with 4 10 000 rpm disks, 146GB for each disks. We would have 280GB of space, enough for some years.And we could use Suda as replicated server to search function or sql query (need to add more disks on Suda in all of case).
What are our disk requirements for the db server? Let's be generous with ourselves here. But, also let's think about whether disk space needs might in some cases be better handled by a big slow cheap SATA storage server?
We could think about SATA, but I think at it as a SAN or a NAS like the promise Vtrak 15100 :
http://www.promise.com/product/product_detail_eng.asp?productId=115&fami...
And Western Digital produce SATA disks at 10000 RPM. It would be a nice idea, we could use an inexpensive raid 50, which must be fast and safety.
Shaihulud
On Apr 7, 2004, at 5:22 AM, Camille Constans wrote:
Raid10 is faster. I calculate, that the db is growing at around 1GBytes per week, So it could be an idea to have raid5 but with more than 3 disks (4 or 5 ?). Raid 10 with 15000 rpms is a pb cause we wont be able to get more space than 140 G with 4 disks (which should be enough for a year).
if disk space becomes a problem you could allways split the wikipedias onto different database servers. As the project grows this step is inevitable, eventually (we hope) each wikipedia will have it's own database server and slave. the disk-space will not be a problem and performance shouldn't either since they will be completely independent from each other.
Lightning
On Tue, 6 Apr 2004 22:31:02 -0700 Jimmy Wales jwales@bomis.com wrote:
Going under the assumption that we will be buying a new db server to replace geoffrin, please everyone take a look at: http://www.siliconmechanics.com/
and help me decide on a suitable configuration. Assume a budget of $9,000-$10,000, but don't hesitate to go under (of course!) or over (if for a good reason!).
There have been several requests (from more than one person, or just from one person) for 15k spindle drives. Is that a universal consensus? I find it hard to imagine that faster drives is really pertinent, because just about any problem that can be solved with faster drives can be solved better with more ram?
I hope I'm not the only one person would wish 15000 rpm disks :) I understand that they are more expensive, but our main problem in database is access time, and I dont think we can cache all the db in memory. On Suda, mysql use only 1,5G of memory (cause 32bits adressing pb IIRC), and it depends heavily on access time of disks. Cause of its slow disks, developper had to stop SQL query, search query and others stuffs. We're back in time before Geoffrin :( Having 15000 rpm disks, could let us some quiet time before thinking about a new upgrade. And it's not a cpu pb, cpu is idle most of time on Suda.
But, that's a very simplistic view on my part, so I could easily be convinced otherwise.
RAID 10? That's what geoffrin had, and it was sweet. But RAID 5 would give more disk space, right?
Raid10 is faster. I calculate, that the db is growing at around 1GBytes per week, So it could be an idea to have raid5 but with more than 3 disks (4 or 5 ?). Raid 10 with 15000 rpms is a pb cause we wont be able to get more space than 140 G with 4 disks (which should be enough for a year). So we could choose to have a similar setup than Geoffrin, with 4 10 000 rpm disks, 146GB for each disks. We would have 280GB of space, enough for some years.And we could use Suda as replicated server to search function or sql query (need to add more disks on Suda in all of case).
What are our disk requirements for the db server? Let's be generous with ourselves here. But, also let's think about whether disk space needs might in some cases be better handled by a big slow cheap SATA storage server?
We could think about SATA, but I think at it as a SAN or a NAS like the promise Vtrak 15100 :
http://www.promise.com/product/product_detail_eng.asp?productId=115&fami...
And Western Digital produce SATA disks at 10000 RPM. It would be a nice idea, we could use an inexpensive raid 50, which must be fast and safety.
Shaihulud
On Tue, 6 Apr 2004, Jimmy Wales wrote:
Going under the assumption that we will be buying a new db server to replace geoffrin, please everyone take a look at: http://www.siliconmechanics.com/
and help me decide on a suitable configuration. Assume a budget of $9,000-$10,000, but don't hesitate to go under (of course!) or over (if for a good reason!).
Here's a quick configuration:
SM-1280S 1U server with dual Opterons
- dual Opteron 246 2.2 Ghz - 8 GB ECC ram - 3x Seagate Cheetah 15000 rpm 73GB - LSI 20320 1 channel SCSI controller - CD-RW/DVD combo - floppy - Suse linux enterprise for AMD64 - 1U 2 piece rail kit - 3 years warranty, 24x7 helpdesk, next business dayon site
Total $10142.00
Disks would be about 140 GB in RAID5. I have no idea of the actual space requirements for the db server. If this is too low, one can go down to 10.000 rpm disks (which should still give good performance) and get 146GB disks that will double the capacity (280GB in RAID5) for an additional $400.
If that's too much, have just two disks and 73GB of space in RAID1 (15.000 rpm) or 140GB (10.000 rpm) saving around $700.
RAM can be expanded to 16GB in the future buying 2GB DIMMs (that are not available right now). Dual gigabit ethernet is integrated.
Anything I missed?
Alfio
On Tue, 6 Apr 2004 22:31:02 -0700 Jimmy Wales jwales@bomis.com wrote:
Going under the assumption that we will be buying a new db server to replace geoffrin, please everyone take a look at: http://www.siliconmechanics.com/
and help me decide on a suitable configuration. Assume a budget of $9,000-$10,000, but don't hesitate to go under (of course!) or over (if for a good reason!).
There have been several requests (from more than one person, or just from one person) for 15k spindle drives. Is that a universal consensus? I find it hard to imagine that faster drives is really pertinent, because just about any problem that can be solved with faster drives can be solved better with more ram?
I hope I'm not the only one person would wish 15000 rpm disks :) I understand that they are more expensive, but our main problem in database is access time, and I dont think we can cache all the db in memory. On Suda, mysql use only 1,5G of memory (cause 32bits adressing pb IIRC), and it depends heavily on access time of disks. Cause of its slow disks, developper had to stop SQL query, search query and others stuffs. We're back in time before Geoffrin :( Having 15000 rpm disks, could let us some quiet time before thinking about a new upgrade. And it's not a cpu pb, cpu is idle most of time on Suda.
But, that's a very simplistic view on my part, so I could easily be convinced otherwise.
RAID 10? That's what geoffrin had, and it was sweet. But RAID 5 would give more disk space, right?
Raid10 is faster. I calculate, that the db is growing at around 1GBytes per week, So it could be an idea to have raid5 but with more than 3 disks (4 or 5 ?). Raid 10 with 15000 rpms is a pb cause we wont be able to get more space than 140 G with 4 disks (which should be enough for a year). So we could choose to have a similar setup than Geoffrin, with 4 10 000 rpm disks, 146GB for each disks. We would have 280GB of space, enough for some years.And we could use Suda as replicated server to search function or sql query (need to add more disks on Suda in all of case).
What are our disk requirements for the db server? Let's be generous with ourselves here. But, also let's think about whether disk space needs might in some cases be better handled by a big slow cheap SATA storage server?
We could think about SATA, but I think at it as a SAN or a NAS like the promise Vtrak 15100 :
http://www.promise.com/product/product_detail_eng.asp?productId=115&fami...
And Western Digital produce SATA disks at 10000 RPM. It would be a nice idea, we could use an inexpensive raid 50, which must be fast and safety.
Shaihulud
On Tue, 06 Apr 2004 22:31:02 -0700, Jimmy Wales wrote:
Going under the assumption that we will be buying a new db server to replace geoffrin, please everyone take a look at: http://www.siliconmechanics.com/
and help me decide on a suitable configuration. Assume a budget of $9,000-$10,000, but don't hesitate to go under (of course!) or over (if for a good reason!).
There have been several requests (from more than one person, or just from one person) for 15k spindle drives. Is that a universal consensus? I find it hard to imagine that faster drives is really pertinent, because just about any problem that can be solved with faster drives can be solved better with more ram?
I would clearly favour ram, maxing out the disk i/o isn't fast, even with the best hd available- plus it should wear the hd's down in time. An idea (didn't really look into the scsi/disk requirements):
name and description price subtotal SM-2280S CPU: Dual AMD Opteron 240 - 1.4 GHz - 1MB L2 Cache Memory: 8GB (8 x 1GB) PC2700 Registered ECC HDD 1: Seagate Cheetah 10K.6 146GB U320 10KRPM SCA SCSI HDD 2: Seagate Cheetah 10K.6 146GB U320 10KRPM SCA SCSI HDD 3: Seagate Cheetah 10K.6 146GB U320 10KRPM SCA SCSI HDD 4: Seagate Cheetah 10K.6 146GB U320 10KRPM SCA SCSI HDD 5: Seagate Cheetah 10K.6 146GB U320 10KRPM SCA SCSI HDD 6: Seagate Cheetah 10K.6 146GB U320 10KRPM SCA SCSI SCSI Controller: LSI MegaRAID SCSI 320-2 - 2 Channel U320 RAID 64MB CACHE w/BBU Low Profile CD-ROM: Slimline 24X CD-ROM Floppy: 1.44MB Floppy Power Supply: 460W Standard Rail Kit: 2U 3 Piece Sliding Rail Kit 24"-31" WARRANTY: Standard 3 Year - Return to Depot $10644.00 $10644.00
On Tue, Apr 06, 2004 at 10:31:02PM -0700, Jimmy Wales wrote:
Going under the assumption that we will be buying a new db server to replace geoffrin, please everyone take a look at: http://www.siliconmechanics.com/
and help me decide on a suitable configuration. Assume a budget of $9,000-$10,000, but don't hesitate to go under (of course!) or over (if for a good reason!).
RAID 10? That's what geoffrin had, and it was sweet. But RAID 5 would give more disk space, right?
We have two needs from what I see: * Fast database disks and * large disks to store log files, dumps, etc
As it is hard to cover these needs with only one type of disks, we should use two types of disks for these needs.
What are our disk requirements for the db server? Let's be generous with ourselves here. But, also let's think about whether disk space needs might in some cases be better handled by a big slow cheap SATA storage server?
Small faster SATAs for the DB, big slower disks for the dumps.
We have a crucial responsibility to be good stewards of our donor's money. That means two things -- first, we don't buy hardware just because it sounds cool, but to meet specific needs. But second, the money wasn't given to us to sit in the bank forever, it's to give the donors what they want -- a fast fast wikipedia. So we should not be cheapskates, either.
Proposed config:
SM 2880SATA 2* Opteron 240 # CPU is idle, I/O is the bottle neck # Would one CPU be sufficient to cover # current needs? 8 GB RAM # You can't replace RAM by anything but RAM 3Ware 8 Port SATA RAID 4* WD 10Krpm 74 Gig # the faster ones for the data base. # no 15Krpm ones. But we have RAM to handle # this. RAID 10. 148 Gig net capacity 2* WD 7.2Krpm 250 Gig # The big ones for the dumps. Hotswap Power Supply # I'm a paranoid coward. And having a RAID # for the disks but only one PS seems # pointless.
Total price: 7,576 US$.
Maintenance upgrade to 5*8 next business day on site is at 415 US$. Perhaps service is better when they have to come to us instead of us having to send them the server.
Overall CapEx: 7,991 US$
Regards,
JeLuF
On Thu, Apr 08, 2004 at 12:01:09PM +0900, Guillaume Blanchard wrote:
From: "Jens Frank"
We have two needs from what I see:
- Fast database disks and
- large disks to store log files, dumps, etc
What about put "cur", "user", etc on fasts disk and "old" on large disks ?
Data and index directories can only be specified for MyISAM tables, not for InnoDB ones.
JeLuF
On Thu, 8 Apr 2004 02:00:25 +0200 Jens Frank JeLuF@gmx.de wrote:
On Tue, Apr 06, 2004 at 10:31:02PM -0700, Jimmy Wales wrote:
Going under the assumption that we will be buying a new db server to replace geoffrin, please everyone take a look at: http://www.siliconmechanics.com/
and help me decide on a suitable configuration. Assume a budget of $9,000-$10,000, but don't hesitate to go under (of course!) or over (if for a good reason!).
RAID 10? That's what geoffrin had, and it was sweet. But RAID 5 would give more disk space, right?
We have two needs from what I see:
- Fast database disks and
- large disks to store log files, dumps, etc
Large disks to store dump are needed on nfs server, zwinger. Not on db server. Dump are put on another server (maybe cause safety?)
wikitech-l@lists.wikimedia.org