To give some idea of what makes a difference, here are some of the things discovered over the last two weeks:
1. Storing the PHP files on NFS doubled the page load time, from around 180ms to around 360ms. So, that's no longer being done and effectively no complicated programming was needed to double performance.
2. Squid using the disk reached cache hit rates of 78% and still rising, compared to 60% without the disk, but...
3. Squid using synchronous I/O blocked on disk and would sometimes result in timeouts. So, disk was turned off for the last week.
4. The Apaches started slowing down at peak load times earlier this week, so more Squid investigation started, using asynchronous disk I/O this time. Given the past disk caching experience, this has the potential to cut the load the Apaches see by around 25-50% (they see 40% of load without disk, disk cut that to 22%).
5. There's a parameter in Squid which tells it when to ignore the disk, based on the number of file descriptors in use. If the limit is exceeded, Squid ignores the disk and just passes the request directly to the Apaches, or skips saving the page to the disk. Tuning of this parameter is currently ongoing. When set correctly it should let Squid deliver all it can from combined disk and RAM, but only up to the point where it doesn't start to block waiting for the disk.
So, there's no need to get too enthusiastic about tuning the code. The new server setup still isn't tuned fully yet... and it probably won't be before we get a nice fast database server, a second Squid and some more Apaches to spread the load around.