Not all bots will necessarily pick up robots.txt changes within a couple
days. But if these issues persist, then maybe they're indicating that your
site is popular—probably a good thing, even if you now have some work to do.
Unless you're in the mood to spend more money, set up the caching that I
mentioned before; it's the most effective way to optimize MediaWiki's
performance without adding more hardware.
On Mon, Feb 2, 2009 at 10:20 PM, Philip Beach <beachboy4231(a)gmail.com>wrote;wrote:
Okay, I have ruled out bots. After disallowing all
user-agents for a few
days the load did not decrease, my site continued having errors. I have
also, with the help of my host, ruled out DOS attacks. I have also
optimized
the database. I don't know what else to try, my users are still getting
this
error every couple of minutes.
Service Temporarily Unavailable
The server is temporarily unable to service your request due to maintenance
downtime or capacity problems. Please try again later.
Additionally, a 404 Not Found error was encountered while trying to use an
ErrorDocument to handle the request.
It seems when analyzing the load factor compared to the time the errors
appear that the load steadily increases to a factor of about 25, then the
error appears, the load decreases, and it goes back online. My host said
this was the way that the server behaves when under a heavy load of legit
users and that I should focus on optimizing the database. However, this is
a
brand new quad-core server. My old server (dual core, half the ram) was
running the same exact database with no problems.
Does anyone have any ideas?
On Mon, Feb 2, 2009 at 3:34 AM, Benjamin Lees <emufarmers(a)gmail.com>
wrote:
Google offers such a tool with their Webmaster
Tools <
https://www.google.com/webmasters/tools>gt;; I'm sure there are plenty of
similar tools out there.
On Sun, Feb 1, 2009 at 1:17 PM, Philip Beach <beachboy4231(a)gmail.com>
wrote:
Ok thanks, but how can I be sure it's working
before all my pages drop
off google (if it were wrong). Is there some way to validate it in
the context of my site?
Thanks again
On 2/1/09, Benjamin Lees <emufarmers(a)gmail.com> wrote:
> TryUser-agent: *
> Disallow: /index.php
> Disallow: /skins/
> Disallow: /Special:Search
> Disallow: /Special:Random
>
> Some other good rules to include are
> Disallow: /MediaWiki:
> Disallow: /Template:
>
> and maybe
> Disallow: /Category:
>
> On Sun, Feb 1, 2009 at 2:06 AM, Philip Beach <beachboy4231(a)gmail.com
wrote:
> Haha true about rotting RAM, I'll look into that. I am not using
wikimedia
>> style URL's, sadly :( it just didn't happen when the site was first
set
up
>> and I can't move it now, for various reasons. All of my files are in
the
> >> web-root /. However, through an apache alias, my url is
> >>
mywiki.com/Pagename.
> >>
> >> How would robots.txt look for that? Would I simply drop the
preceeding
>>
/wiki, like this?
>>
>> User-agent: *
>> Disallow: /Special:Search
>> Disallow: /Special:Random
>>
>> Thanks a ton!
>>
>> On Sun, Feb 1, 2009 at 1:47 AM, Benjamin Lees <emufarmers(a)gmail.com
> >> wrote:
> >>
> >> > Assuming you have Wikimedia-style URLs:
> >> > User-agent: *
> >> > Disallow: /w/
> >> > Disallow: /wiki/Special:Search
> >> > Disallow: /wiki/Special:Random
> >> >
> >> > Your server will be able to handle a lot more if you set up as
much
> >> caching
> >> > as you can <http://www.mediawiki.org/wiki/Manual:Cache>. No
sense
> >> letting
> >> > all that spare RAM rot. :)
> >> >
> >> >
> >> > On Sat, Jan 31, 2009 at 10:02 PM, Philip Beach <
> beachboy4231(a)gmail.com
> >> > >wrote:
> >> >
> >> > > I already have checked the access logs. It appears that Google
and
>>
Yahoo
>> > > are
>> > > indeed generating a lot of traffic. Good idea Rob, I've been
working
> >> > > on
> >> > > this
> >> > > for a while.
> >> > >
> >> > > Just out of curiosity, what should my robots.txt look like for
> >> Mediawiki.
> >> > > Does anything need to be disallowed?
> >> > >
> >> > > On Sat, Jan 31, 2009 at 8:30 PM, Platonides <
Platonides(a)gmail.com
>
> >> > wrote:
> >> > >
> >> > > > You should check the access logs for which is causing the
error.
> > > >
> > > >
> > > > _______________________________________________
> > > > MediaWiki-l mailing list
> > > > MediaWiki-l(a)lists.wikimedia.org
> > > >
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
> > > >
> > > _______________________________________________
> > > MediaWiki-l mailing list
> > > MediaWiki-l(a)lists.wikimedia.org
> > >
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
> > >
> > _______________________________________________
> > MediaWiki-l mailing list
> > MediaWiki-l(a)lists.wikimedia.org
> >
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
> >
> _______________________________________________
> MediaWiki-l mailing list
> MediaWiki-l(a)lists.wikimedia.org
>
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
>
_______________________________________________
MediaWiki-l mailing list
MediaWiki-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
_______________________________________________
MediaWiki-l mailing list
MediaWiki-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
_______________________________________________
MediaWiki-l mailing list
MediaWiki-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
_______________________________________________
MediaWiki-l mailing list
MediaWiki-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l