Lee Daniel Crocker wrote:
Absolutely; lies, damned lies, and benchmarks, and all that.
Improving benchmarks which apply to Wikipedia db will hopefully improve the situation out "in the wild". So yeah, the rest is just lies.
Disk I/O may well be a major culprit. Memory/CPU usage probably isn't. I'll also run some tests for things like having the database on a separate machine ... I'd lso appreciate suggestions for other benchmarks (specific MySQL settings, for example).
Even if your system has plenty of memory, MySQL may not be configured to use it. What do your settings in my.cnf look like? These settings will also differ for MyISAM and InnoDB tables.
Improving disk throughput usually translates -> new hardware. You can try a different file system or block size. XFS for Linux is improving. You may want to compare it to ReiserFS. If you are going to test different block sizes for the db, partition accordingly with the db on a separate partition from the OS, Apache, PHP and MySQL binaries. This way, you can leave the binary partitions at a smaller block size and adjust the db partition without affecting the others. When installing your db on a second machine do the same; isolate your binaries from your data.
Monitoring with mytop could be interesting: http://jeremy.zawodny.com/mysql/mytop/
Improving disk throughput usually translates -> new hardware. You can try a different file system or block size. XFS for Linux is improving. You may want to compare it to ReiserFS. If you are going to test different block sizes for the db, partition accordingly with the db on a separate partition from the OS, Apache, PHP and MySQL binaries. This way, you can leave the binary partitions at a smaller block size and adjust the db partition without affecting the others. When installing your db on a second machine do the same; isolate your binaries from your data.
I'm sure I'll end up doing some of that. Right now, I'm using an old Compaq with a small (8Gb) disk for the test installation, mainly because it's trashable. But the software is relatively stable and safe now, so I'll install it on my main development box with the nice 10,000 RPM SCSI and a gig of ram, and run the test suite from the Compaq instead.
I'm a big fan of ReiserFS in general. That's what the MySQL folks recommend as well, and I run that at Piclab (which is a small machine but runs the testsuite faster than my Compaq). I'm not sure that block sizes are that flexible for Resier, but I'll look into it. At any rate, it would be good to find an optimal arrangement for the database before we get the new server to install it on.
On Thu, Apr 17, 2003 at 05:12:37PM -0500, Lee Daniel Crocker wrote:
Improving disk throughput usually translates -> new hardware. You can try a different file system or block size. XFS for Linux is improving. You may want to compare it to ReiserFS. If you are going to test different block sizes for the db, partition accordingly with the db on a separate partition from the OS, Apache, PHP and MySQL binaries. This way, you can leave the binary partitions at a smaller block size and adjust the db partition without affecting the others. When installing your db on a second machine do the same; isolate your binaries from your data.
I'm sure I'll end up doing some of that. Right now, I'm using an old Compaq with a small (8Gb) disk for the test installation, mainly because it's trashable. But the software is relatively stable and safe now, so I'll install it on my main development box with the nice 10,000 RPM SCSI and a gig of ram, and run the test suite from the Compaq instead.
I'm a big fan of ReiserFS in general. That's what the MySQL folks recommend as well, and I run that at Piclab (which is a small machine but runs the testsuite faster than my Compaq). I'm not sure that block sizes are that flexible for Resier, but I'll look into it. At any rate, it would be good to find an optimal arrangement for the database before we get the new server to install it on. http://www.wikipedia.org/mailman/listinfo/wikitech-l
AFAIK, ReiserFS block sizes are stuck at 4KB unless someone changed that while I wasn't looking.
On Thu, Apr 17, 2003 at 04:56:28PM -0500, Dreyer, Jason wrote:
Lee Daniel Crocker wrote:
Absolutely; lies, damned lies, and benchmarks, and all that.
Improving benchmarks which apply to Wikipedia db will hopefully improve the situation out "in the wild". So yeah, the rest is just lies.
Disk I/O may well be a major culprit. Memory/CPU usage probably isn't. I'll also run some tests for things like having the database on a separate machine ... I'd lso appreciate suggestions for other benchmarks (specific MySQL settings, for example).
Even if your system has plenty of memory, MySQL may not be configured to use it. What do your settings in my.cnf look like? These settings will also differ for MyISAM and InnoDB tables.
Improving disk throughput usually translates -> new hardware. You can try a different file system or block size. XFS for Linux is improving. You may want to compare it to ReiserFS. If you are going to test different block sizes for the db, partition accordingly with the db on a separate partition from the OS, Apache, PHP and MySQL binaries. This way, you can leave the binary partitions at a smaller block size and adjust the db partition without affecting the others. When installing your db on a second machine do the same; isolate your binaries from your data.
Also, I still think we need to (badly) upgrade the kernel - AFAIK, it is still running 2.4.6 (Brion hasn't said otherwise). A newer kernel could really help us out here (w/ the ganked and reganked VM subsystem).
On Thu, 17 Apr 2003, Nick Reinking wrote:
Also, I still think we need to (badly) upgrade the kernel - AFAIK, it is still running 2.4.6 (Brion hasn't said otherwise). A newer kernel could really help us out here (w/ the ganked and reganked VM subsystem).
It's 2.4.7.
I've been reluctant to do a kernel upgrade for fear of making it unbootable and subjecting Jason to another 3-hour journey. :) Had I thought of it at the time, we should have gone ahead and done it when he was there dealing with the recent crash.
If we just install the RedHat RPMs it should in theory be fairly painless, but I'm not too familiar with how the red hat boot goodies are set up or how to make it fall back to the previous kernel after the first reboot if the new one doesn't work.
-- brion vibber (brion @ pobox.com)
(Brion Vibber vibber@aludra.usc.edu):
If we just install the RedHat RPMs it should in theory be fairly painless, but I'm not too familiar with how the red hat boot goodies are set up or how to make it fall back to the previous kernel after the first reboot if the new one doesn't work.
I'm fairly familiar with how to do that (just having done it on my test machine a few times), but it requires a boot floppy or CD (and therefore the 3-hour trip for Jason).
On Thu, Apr 17, 2003 at 05:50:35PM -0500, Lee Daniel Crocker wrote:
(Brion Vibber vibber@aludra.usc.edu):
If we just install the RedHat RPMs it should in theory be fairly painless, but I'm not too familiar with how the red hat boot goodies are set up or how to make it fall back to the previous kernel after the first reboot if the new one doesn't work.
I'm fairly familiar with how to do that (just having done it on my test machine a few times), but it requires a boot floppy or CD (and therefore the 3-hour trip for Jason).
I'm pretty sure if it breaks, you can still select the old kernel from a list (RH uses GRUB, I believe). So, you still have to be there once to boot it to the old kernel, and then you can change the boot configuration. Still, are we really going to want to jump to 2.6 right away after it comes out? 2.4 had quite a bit of teething problems until fairly late in the game.
On Thu, 17 Apr 2003, Nick Reinking wrote:
I'm pretty sure if it breaks, you can still select the old kernel from a list (RH uses GRUB, I believe). So, you still have to be there once to boot it to the old kernel, and then you can change the boot configuration.
Sure, but we don't have console without somebody driving to San Diego...
I recall that LILO had an option to set one image as the default for the next boot, and then revert to the lilo.conf-specified one for subsequent boots. GRUB probably has similar functionality, but it scares me and I've never really tried to figure it out.
Still, are we really going to want to jump to 2.6 right away after it comes out? 2.4 had quite a bit of teething problems until fairly late in the game.
I don't see any real need to go 2.6, but a more recent 2.4 ought to be a help (fixed VM probs, some local security loopholes).
-- brion vibber (brion @ pobox.com)
Our servers use LILO. I've never taken the time to learn about GRUB, so I can't abide to have it in use on a server that I end up working on. If the server is, in fact, using GRUB, I would be surprised.
I am intrigued by this LILI option you mention. That's new to me, and it sounds pretty useful.
Jason
P.S. I will eventually be making another trip to San Diego, and I could always update the kernel while I'm there.
Brion Vibber wrote:
On Thu, 17 Apr 2003, Nick Reinking wrote:
I'm pretty sure if it breaks, you can still select the old kernel from a list (RH uses GRUB, I believe). So, you still have to be there once to boot it to the old kernel, and then you can change the boot configuration.
Sure, but we don't have console without somebody driving to San Diego...
I recall that LILO had an option to set one image as the default for the next boot, and then revert to the lilo.conf-specified one for subsequent boots. GRUB probably has similar functionality, but it scares me and I've never really tried to figure it out.
Still, are we really going to want to jump to 2.6 right away after it comes out? 2.4 had quite a bit of teething problems until fairly late in the game.
I don't see any real need to go 2.6, but a more recent 2.4 ought to be a help (fixed VM probs, some local security loopholes).
-- brion vibber (brion @ pobox.com)
Wikitech-l mailing list Wikitech-l@wikipedia.org http://www.wikipedia.org/mailman/listinfo/wikitech-l
On Thu, 17 Apr 2003, Jason Richey wrote:
Our servers use LILO. I've never taken the time to learn about GRUB, so I can't abide to have it in use on a server that I end up working on. If the server is, in fact, using GRUB, I would be surprised.
I don't know which is sitting on the boot sector, but both LILO and GRUB are in fact available on the system, so it should be elementary to use one or the other. In theory. :)
I am intrigued by this LILI option you mention. That's new to me, and it sounds pretty useful.
Lemme look it up....
/sbin/lilo -R - set default command line for next reboot
-R command line This option sets the default command for the boot loader the next time it executes. The boot loader will then erase this line: this is a once-only com mand. It is typically used in reboot scripts, just before calling `shutdown -r'.
So, you could set up separate entries linux-knowngood and linux-scarynew, set linux-knowngood as the default in lilo.conf, then:
/sbin/lilo -R linux-scarynew /sbin/shutdown -r now
If the machine can't boot up under linux-scarynew, it can be remotely rebooted, and the second boot should bring up linux-knowngood. (Or if linux-scarynew crashes on its own later.) If it proves reliable, the default can be changed later on.
-- brion vibber (brion @ pobox.com)
(Nick Reinking nick@twoevils.org):
Also, I still think we need to (badly) upgrade the kernel - AFAIK, it is still running 2.4.6 (Brion hasn't said otherwise). A newer kernel could really help us out here (w/ the ganked and reganked VM subsystem).
Now seems like an awkward time for that, though, since it's my impression that 2.5.X is going to become stable 2.6 Real Soon Now, with new VMs, new threading, etc. Reiser 4 is just around the corner as well.
Here's a thought: if we do plan to install a brand new server for the database, take advantage of that by doing something similar to what we did for the present server. Installing the latest stuff, making lots of tweaks, trashing it a few times if needed, until we get a setup that runs reliably and fast, test the hell out of it, then make a backup, move the database, and bring it up, prepared to switch back if all hell breaks loose.
I'm thinking a machine that's dedicated to running one program (mysqld), attached to one client, can afford to be a little more on the bleeding edge and still be stable than one that has to serve hundreds of clients with dozens of apps like a typical ISP.
wikitech-l@lists.wikimedia.org