Visitors to my wiki are complaining of regularly-occurring 503 "Service Temporarily Unavailable" errors, or just simply the site not loading through their browsers. I investigated the errors and they seem to occur unpredictably every minute or so and only last for 10 seconds or so. If I open a different browser while the first is getting 503s, the other browser IS able to connect.
I looked at my load numbers and the site seems to be under load from several search engine bots. The actual load number is frequently over 20k (which I've heard is high), sometimes spiking to 40, and a few times I cough it at over 80!?? My host said I had 200 hits per minute when I asked about the issue.
My hardware is a quad core 5 series xenon at 2.3ghz, with 6GB of RAM. I have php's memory limit set to 256mb. My ram never seems to be more than 25% full.
Could this be causing the random 503s? I am not using a robots.txt file, could bots be indexing stuff they shouldn't be and in turn slowing the site?
Thanks!
Philip, could you perhaps be under the influence of Denial-of-Service attacks?
if these accesses are from bots that seem legitimate, you could, depending on your OS, prevent their access via (for example) iptables.
I would suggest turning off access for the bots in iptables (depending on your OS) and then checking the situation. then you can determine what will be the final solution.
On Feb 1, 2009, at 2:03 AM, Philip Beach wrote:
Visitors to my wiki are complaining of regularly-occurring 503 "Service Temporarily Unavailable" errors, or just simply the site not loading through
-- Rob Lingelbach rob@colorist.org
You should check the access logs for which is causing the error.
I already have checked the access logs. It appears that Google and Yahoo are indeed generating a lot of traffic. Good idea Rob, I've been working on this for a while.
Just out of curiosity, what should my robots.txt look like for Mediawiki. Does anything need to be disallowed?
On Sat, Jan 31, 2009 at 8:30 PM, Platonides Platonides@gmail.com wrote:
You should check the access logs for which is causing the error.
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Assuming you have Wikimedia-style URLs: User-agent: * Disallow: /w/ Disallow: /wiki/Special:Search Disallow: /wiki/Special:Random
Your server will be able to handle a lot more if you set up as much caching as you can http://www.mediawiki.org/wiki/Manual:Cache. No sense letting all that spare RAM rot. :)
On Sat, Jan 31, 2009 at 10:02 PM, Philip Beach beachboy4231@gmail.comwrote:
I already have checked the access logs. It appears that Google and Yahoo are indeed generating a lot of traffic. Good idea Rob, I've been working on this for a while.
Just out of curiosity, what should my robots.txt look like for Mediawiki. Does anything need to be disallowed?
On Sat, Jan 31, 2009 at 8:30 PM, Platonides Platonides@gmail.com wrote:
You should check the access logs for which is causing the error.
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Haha true about rotting RAM, I'll look into that. I am not using wikimedia style URL's, sadly :( it just didn't happen when the site was first set up and I can't move it now, for various reasons. All of my files are in the web-root /. However, through an apache alias, my url is mywiki.com/Pagename.
How would robots.txt look for that? Would I simply drop the preceeding /wiki, like this?
User-agent: * Disallow: /Special:Search Disallow: /Special:Random
Thanks a ton!
On Sun, Feb 1, 2009 at 1:47 AM, Benjamin Lees emufarmers@gmail.com wrote:
Assuming you have Wikimedia-style URLs: User-agent: * Disallow: /w/ Disallow: /wiki/Special:Search Disallow: /wiki/Special:Random
Your server will be able to handle a lot more if you set up as much caching as you can http://www.mediawiki.org/wiki/Manual:Cache. No sense letting all that spare RAM rot. :)
On Sat, Jan 31, 2009 at 10:02 PM, Philip Beach <beachboy4231@gmail.com
wrote:
I already have checked the access logs. It appears that Google and Yahoo are indeed generating a lot of traffic. Good idea Rob, I've been working on this for a while.
Just out of curiosity, what should my robots.txt look like for Mediawiki. Does anything need to be disallowed?
On Sat, Jan 31, 2009 at 8:30 PM, Platonides Platonides@gmail.com
wrote:
You should check the access logs for which is causing the error.
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
TryUser-agent: * Disallow: /index.php Disallow: /skins/ Disallow: /Special:Search Disallow: /Special:Random
Some other good rules to include are Disallow: /MediaWiki: Disallow: /Template:
and maybe Disallow: /Category:
On Sun, Feb 1, 2009 at 2:06 AM, Philip Beach beachboy4231@gmail.com wrote:
Haha true about rotting RAM, I'll look into that. I am not using wikimedia style URL's, sadly :( it just didn't happen when the site was first set up and I can't move it now, for various reasons. All of my files are in the web-root /. However, through an apache alias, my url is mywiki.com/Pagename.
How would robots.txt look for that? Would I simply drop the preceeding /wiki, like this?
User-agent: * Disallow: /Special:Search Disallow: /Special:Random
Thanks a ton!
On Sun, Feb 1, 2009 at 1:47 AM, Benjamin Lees emufarmers@gmail.com wrote:
Assuming you have Wikimedia-style URLs: User-agent: * Disallow: /w/ Disallow: /wiki/Special:Search Disallow: /wiki/Special:Random
Your server will be able to handle a lot more if you set up as much
caching
as you can http://www.mediawiki.org/wiki/Manual:Cache. No sense
letting
all that spare RAM rot. :)
On Sat, Jan 31, 2009 at 10:02 PM, Philip Beach <beachboy4231@gmail.com
wrote:
I already have checked the access logs. It appears that Google and
Yahoo
are indeed generating a lot of traffic. Good idea Rob, I've been working on this for a while.
Just out of curiosity, what should my robots.txt look like for
Mediawiki.
Does anything need to be disallowed?
On Sat, Jan 31, 2009 at 8:30 PM, Platonides Platonides@gmail.com
wrote:
You should check the access logs for which is causing the error.
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Ok thanks, but how can I be sure it's working before all my pages drop off google (if it were wrong). Is there some way to validate it in the context of my site?
Thanks again
On 2/1/09, Benjamin Lees emufarmers@gmail.com wrote:
TryUser-agent: * Disallow: /index.php Disallow: /skins/ Disallow: /Special:Search Disallow: /Special:Random
Some other good rules to include are Disallow: /MediaWiki: Disallow: /Template:
and maybe Disallow: /Category:
On Sun, Feb 1, 2009 at 2:06 AM, Philip Beach beachboy4231@gmail.com wrote:
Haha true about rotting RAM, I'll look into that. I am not using wikimedia style URL's, sadly :( it just didn't happen when the site was first set up and I can't move it now, for various reasons. All of my files are in the web-root /. However, through an apache alias, my url is mywiki.com/Pagename.
How would robots.txt look for that? Would I simply drop the preceeding /wiki, like this?
User-agent: * Disallow: /Special:Search Disallow: /Special:Random
Thanks a ton!
On Sun, Feb 1, 2009 at 1:47 AM, Benjamin Lees emufarmers@gmail.com wrote:
Assuming you have Wikimedia-style URLs: User-agent: * Disallow: /w/ Disallow: /wiki/Special:Search Disallow: /wiki/Special:Random
Your server will be able to handle a lot more if you set up as much
caching
as you can http://www.mediawiki.org/wiki/Manual:Cache. No sense
letting
all that spare RAM rot. :)
On Sat, Jan 31, 2009 at 10:02 PM, Philip Beach <beachboy4231@gmail.com
wrote:
I already have checked the access logs. It appears that Google and
Yahoo
are indeed generating a lot of traffic. Good idea Rob, I've been working on this for a while.
Just out of curiosity, what should my robots.txt look like for
Mediawiki.
Does anything need to be disallowed?
On Sat, Jan 31, 2009 at 8:30 PM, Platonides Platonides@gmail.com
wrote:
You should check the access logs for which is causing the error.
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Google offers such a tool with their Webmaster Tools < https://www.google.com/webmasters/tools%3E; I'm sure there are plenty of similar tools out there.
On Sun, Feb 1, 2009 at 1:17 PM, Philip Beach beachboy4231@gmail.com wrote:
Ok thanks, but how can I be sure it's working before all my pages drop off google (if it were wrong). Is there some way to validate it in the context of my site?
Thanks again
On 2/1/09, Benjamin Lees emufarmers@gmail.com wrote:
TryUser-agent: * Disallow: /index.php Disallow: /skins/ Disallow: /Special:Search Disallow: /Special:Random
Some other good rules to include are Disallow: /MediaWiki: Disallow: /Template:
and maybe Disallow: /Category:
On Sun, Feb 1, 2009 at 2:06 AM, Philip Beach beachboy4231@gmail.com
wrote:
Haha true about rotting RAM, I'll look into that. I am not using
wikimedia
style URL's, sadly :( it just didn't happen when the site was first set
up
and I can't move it now, for various reasons. All of my files are in the web-root /. However, through an apache alias, my url is mywiki.com/Pagename.
How would robots.txt look for that? Would I simply drop the preceeding /wiki, like this?
User-agent: * Disallow: /Special:Search Disallow: /Special:Random
Thanks a ton!
On Sun, Feb 1, 2009 at 1:47 AM, Benjamin Lees emufarmers@gmail.com wrote:
Assuming you have Wikimedia-style URLs: User-agent: * Disallow: /w/ Disallow: /wiki/Special:Search Disallow: /wiki/Special:Random
Your server will be able to handle a lot more if you set up as much
caching
as you can http://www.mediawiki.org/wiki/Manual:Cache. No sense
letting
all that spare RAM rot. :)
On Sat, Jan 31, 2009 at 10:02 PM, Philip Beach <
beachboy4231@gmail.com
wrote:
I already have checked the access logs. It appears that Google and
Yahoo
are indeed generating a lot of traffic. Good idea Rob, I've been working on this for a while.
Just out of curiosity, what should my robots.txt look like for
Mediawiki.
Does anything need to be disallowed?
On Sat, Jan 31, 2009 at 8:30 PM, Platonides Platonides@gmail.com
wrote:
You should check the access logs for which is causing the error.
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Okay, I have ruled out bots. After disallowing all user-agents for a few days the load did not decrease, my site continued having errors. I have also, with the help of my host, ruled out DOS attacks. I have also optimized the database. I don't know what else to try, my users are still getting this error every couple of minutes.
Service Temporarily Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
It seems when analyzing the load factor compared to the time the errors appear that the load steadily increases to a factor of about 25, then the error appears, the load decreases, and it goes back online. My host said this was the way that the server behaves when under a heavy load of legit users and that I should focus on optimizing the database. However, this is a brand new quad-core server. My old server (dual core, half the ram) was running the same exact database with no problems.
Does anyone have any ideas?
On Mon, Feb 2, 2009 at 3:34 AM, Benjamin Lees emufarmers@gmail.com wrote:
Google offers such a tool with their Webmaster Tools < https://www.google.com/webmasters/tools%3E; I'm sure there are plenty of similar tools out there.
On Sun, Feb 1, 2009 at 1:17 PM, Philip Beach beachboy4231@gmail.com wrote:
Ok thanks, but how can I be sure it's working before all my pages drop off google (if it were wrong). Is there some way to validate it in the context of my site?
Thanks again
On 2/1/09, Benjamin Lees emufarmers@gmail.com wrote:
TryUser-agent: * Disallow: /index.php Disallow: /skins/ Disallow: /Special:Search Disallow: /Special:Random
Some other good rules to include are Disallow: /MediaWiki: Disallow: /Template:
and maybe Disallow: /Category:
On Sun, Feb 1, 2009 at 2:06 AM, Philip Beach beachboy4231@gmail.com
wrote:
Haha true about rotting RAM, I'll look into that. I am not using
wikimedia
style URL's, sadly :( it just didn't happen when the site was first
set
up
and I can't move it now, for various reasons. All of my files are in
the
web-root /. However, through an apache alias, my url is mywiki.com/Pagename.
How would robots.txt look for that? Would I simply drop the preceeding /wiki, like this?
User-agent: * Disallow: /Special:Search Disallow: /Special:Random
Thanks a ton!
On Sun, Feb 1, 2009 at 1:47 AM, Benjamin Lees emufarmers@gmail.com wrote:
Assuming you have Wikimedia-style URLs: User-agent: * Disallow: /w/ Disallow: /wiki/Special:Search Disallow: /wiki/Special:Random
Your server will be able to handle a lot more if you set up as much
caching
as you can http://www.mediawiki.org/wiki/Manual:Cache. No sense
letting
all that spare RAM rot. :)
On Sat, Jan 31, 2009 at 10:02 PM, Philip Beach <
beachboy4231@gmail.com
wrote:
I already have checked the access logs. It appears that Google and
Yahoo
are indeed generating a lot of traffic. Good idea Rob, I've been
working
on this for a while.
Just out of curiosity, what should my robots.txt look like for
Mediawiki.
Does anything need to be disallowed?
On Sat, Jan 31, 2009 at 8:30 PM, Platonides <Platonides@gmail.com
wrote:
> You should check the access logs for which is causing the error. > > > _______________________________________________ > MediaWiki-l mailing list > MediaWiki-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > _______________________________________________ MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Not all bots will necessarily pick up robots.txt changes within a couple days. But if these issues persist, then maybe they're indicating that your site is popular—probably a good thing, even if you now have some work to do. Unless you're in the mood to spend more money, set up the caching that I mentioned before; it's the most effective way to optimize MediaWiki's performance without adding more hardware.
On Mon, Feb 2, 2009 at 10:20 PM, Philip Beach beachboy4231@gmail.comwrote:
Okay, I have ruled out bots. After disallowing all user-agents for a few days the load did not decrease, my site continued having errors. I have also, with the help of my host, ruled out DOS attacks. I have also optimized the database. I don't know what else to try, my users are still getting this error every couple of minutes.
Service Temporarily Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
It seems when analyzing the load factor compared to the time the errors appear that the load steadily increases to a factor of about 25, then the error appears, the load decreases, and it goes back online. My host said this was the way that the server behaves when under a heavy load of legit users and that I should focus on optimizing the database. However, this is a brand new quad-core server. My old server (dual core, half the ram) was running the same exact database with no problems.
Does anyone have any ideas?
On Mon, Feb 2, 2009 at 3:34 AM, Benjamin Lees emufarmers@gmail.com wrote:
Google offers such a tool with their Webmaster Tools < https://www.google.com/webmasters/tools%3E; I'm sure there are plenty of similar tools out there.
On Sun, Feb 1, 2009 at 1:17 PM, Philip Beach beachboy4231@gmail.com wrote:
Ok thanks, but how can I be sure it's working before all my pages drop off google (if it were wrong). Is there some way to validate it in the context of my site?
Thanks again
On 2/1/09, Benjamin Lees emufarmers@gmail.com wrote:
TryUser-agent: * Disallow: /index.php Disallow: /skins/ Disallow: /Special:Search Disallow: /Special:Random
Some other good rules to include are Disallow: /MediaWiki: Disallow: /Template:
and maybe Disallow: /Category:
On Sun, Feb 1, 2009 at 2:06 AM, Philip Beach <beachboy4231@gmail.com
wrote:
Haha true about rotting RAM, I'll look into that. I am not using
wikimedia
style URL's, sadly :( it just didn't happen when the site was first
set
up
and I can't move it now, for various reasons. All of my files are in
the
web-root /. However, through an apache alias, my url is mywiki.com/Pagename.
How would robots.txt look for that? Would I simply drop the
preceeding
/wiki, like this?
User-agent: * Disallow: /Special:Search Disallow: /Special:Random
Thanks a ton!
On Sun, Feb 1, 2009 at 1:47 AM, Benjamin Lees <emufarmers@gmail.com
wrote:
Assuming you have Wikimedia-style URLs: User-agent: * Disallow: /w/ Disallow: /wiki/Special:Search Disallow: /wiki/Special:Random
Your server will be able to handle a lot more if you set up as
much
caching
as you can http://www.mediawiki.org/wiki/Manual:Cache. No
sense
letting
all that spare RAM rot. :)
On Sat, Jan 31, 2009 at 10:02 PM, Philip Beach <
beachboy4231@gmail.com
>wrote:
> I already have checked the access logs. It appears that Google
and
Yahoo
> are > indeed generating a lot of traffic. Good idea Rob, I've been
working
> on > this > for a while. > > Just out of curiosity, what should my robots.txt look like for
Mediawiki.
> Does anything need to be disallowed? > > On Sat, Jan 31, 2009 at 8:30 PM, Platonides <
Platonides@gmail.com
wrote: > > > You should check the access logs for which is causing the
error.
> > > > > > _______________________________________________ > > MediaWiki-l mailing list > > MediaWiki-l@lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > _______________________________________________ > MediaWiki-l mailing list > MediaWiki-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > _______________________________________________ MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Is Mediawiki the only thing you have running on your server or do you have other webscripts/services/pages running as well?
mediawiki-l@lists.wikimedia.org