Visitors to my wiki are complaining of regularly-occurring 503 "Service Temporarily Unavailable" errors, or just simply the site not loading through their browsers. I investigated the errors and they seem to occur unpredictably every minute or so and only last for 10 seconds or so. If I open a different browser while the first is getting 503s, the other browser IS able to connect.
I looked at my load numbers and the site seems to be under load from several search engine bots. The actual load number is frequently over 20k (which I've heard is high), sometimes spiking to 40, and a few times I cough it at over 80!?? My host said I had 200 hits per minute when I asked about the issue.
My hardware is a quad core 5 series xenon at 2.3ghz, with 6GB of RAM. I have php's memory limit set to 256mb. My ram never seems to be more than 25% full.
Could this be causing the random 503s? I am not using a robots.txt file, could bots be indexing stuff they shouldn't be and in turn slowing the site?
Thanks!