Google indexes and caches around 20,000 pages a day from our website, which I consider a
small to medium to volume/traffic website. They do offer an option for slower indexing
speed if the load hurts the server, though. However, we all know Google indexing and
caching our pages is a necessary evil. Our site? Not so much :D
-----Original Message-----
From: wikitech-l-bounces(a)lists.wikimedia.org
[mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Mark Williamson
Sent: Tuesday, January 30, 2007 2:42 PM
To: Wikimedia developers
Subject: Re: [Wikitech-l] FW: Our IP was blocked by mistake
No -- I don't know the particulars, but I imagine pages are not cached by Google in a
single burst that puts such a huge load on the server, if this were the case most large
sites would have blocked Google (myspace, livejournal, ebay) and it would be much less
useful.
Mark
On 30/01/07, Webmaster <webmaster(a)tiosam.com> wrote:
Just something that came to my mind...
Google caches the wikipedia pages just like we were doing. Are you blocking Google as
well?
-----Original Message-----
From: Webmaster [mailto:webmaster@tiosam.com]
Sent: Tuesday, January 30, 2007 12:27 PM
To: 'Wikimedia developers'
Subject: RE: [Wikitech-l] Our IP was blocked by mistake
Thanks. That explains it. Also, the IP shown is not
enciclopedia.tiosam.com nor
ebaita.com: it is
www.tiosam.com where I just included the English version a couple of
weeks ago (
http://www.tiosam.com/Ingles/encyclopedia ). The load was probably google
indexing the pages in English, not what we already have cached for the Portuguese
version.
Anyway, I apologize for my ignorance. I'm downloading the dump and will start coding
as soon as I find out how it works. Thanks guys.
-----Original Message-----
From: wikitech-l-bounces(a)lists.wikimedia.org
[mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Ivan
Krstic
Sent: Tuesday, January 30, 2007 11:40 AM
To: Wikimedia developers
Subject: Re: [Wikitech-l] Our IP was blocked by mistake
Tim Starling wrote:
No faked user agent string? So I suppose you were
using "save as" in IE?
Mozilla/4.0%20(compatible;%20MSIE%207.0;%20Windows%20NT%205.2;%20.NE
T%
20CLR%201.1.4322;%20.NET%20CLR%202.0.50727)
No, he was using the Microsoft.XMLHTTP object from ASP, as he indicated in a previous
message. Said object identifies itself as MSIE and gives the .NET CLR version in the
User-Agent.
--
Ivan Krstić <krstic(a)solarsail.hcs.harvard.edu> | GPG: 0x147C722D
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
http://lists.wikimedia.org/mailman/listinfo/wikitech-l
--
Refije dirije lanmè yo paske nou posede pwòp bato.
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
http://lists.wikimedia.org/mailman/listinfo/wikitech-l