Jens Frank wrote:
do we have any numbers related to the CPU usage of the machine? Throwing hardware at a performance issue is a solution that's chosen very often, but it's a) most of the time expensive b) too often not the solution
In case of a high lag situation, which process is blocking the CPU? Is it apache (that is: PHP) or mysql? How is the memory utilization? What is the I/O rate?
The CPUs seem to have a fair amount of idle time whenever I check... I'm not sure how to measure CPU usage usefully (I can pop into 'top', but that corrupts the data -- top itself is usually the biggest user of CPU time!)
Load average when things are running smoothly runs between 1 and 2 (2 is ideal usage for a 2-CPU system). Right now there's an old zombie process that bumps up the load average by 1. During busier times during the day, 3-5 is not uncommon. The busiest periods can push us into the teens or very occasionally more.
Very roughly from top:
Apache processes take about 15-25 MB, with 10-25 MB of shared memory. MySQL has a resident memory size of around 234 MB, plus ~88MB of shared memory. I think most of this is shared between processes.
And more generally: ~14 MB buffers 800-900 MB disk cache (dips to 100-300 during very high load times) ~150-200 MB free (dips much lower during very high load times)
That probably doesn't quite cover everything, as we've got 2GB to fill up with those figures.
And: 120-180 MB of rarely used stuff sitting in swap
If you want to see load and memory stats from "uptime" and "free" updated every 10 minutes starting mid-day yesterday, see http://www.wikipedia.org/tools/uptimelog
How can we measure i/o rate?
-- brion vibber (brion @ pobox.com)