Google indexes and caches around 20,000 pages a day from our website, which I consider a small to medium to volume/traffic website. They do offer an option for slower indexing speed if the load hurts the server, though. However, we all know Google indexing and caching our pages is a necessary evil. Our site? Not so much :D
-----Original Message----- From: wikitech-l-bounces@lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Mark Williamson Sent: Tuesday, January 30, 2007 2:42 PM To: Wikimedia developers Subject: Re: [Wikitech-l] FW: Our IP was blocked by mistake
No -- I don't know the particulars, but I imagine pages are not cached by Google in a single burst that puts such a huge load on the server, if this were the case most large sites would have blocked Google (myspace, livejournal, ebay) and it would be much less useful.
Mark
On 30/01/07, Webmaster webmaster@tiosam.com wrote:
Just something that came to my mind... Google caches the wikipedia pages just like we were doing. Are you blocking Google as well?
-----Original Message----- From: Webmaster [mailto:webmaster@tiosam.com] Sent: Tuesday, January 30, 2007 12:27 PM To: 'Wikimedia developers' Subject: RE: [Wikitech-l] Our IP was blocked by mistake
Thanks. That explains it. Also, the IP shown is not enciclopedia.tiosam.com nor ebaita.com: it is www.tiosam.com where I just included the English version a couple of weeks ago (http://www.tiosam.com/Ingles/encyclopedia ). The load was probably google indexing the pages in English, not what we already have cached for the Portuguese version. Anyway, I apologize for my ignorance. I'm downloading the dump and will start coding as soon as I find out how it works. Thanks guys.
-----Original Message----- From: wikitech-l-bounces@lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Ivan Krstic Sent: Tuesday, January 30, 2007 11:40 AM To: Wikimedia developers Subject: Re: [Wikitech-l] Our IP was blocked by mistake
Tim Starling wrote:
No faked user agent string? So I suppose you were using "save as" in IE? Mozilla/4.0%20(compatible;%20MSIE%207.0;%20Windows%20NT%205.2;%20.NE T% 20CLR%201.1.4322;%20.NET%20CLR%202.0.50727)
No, he was using the Microsoft.XMLHTTP object from ASP, as he indicated in a previous message. Said object identifies itself as MSIE and gives the .NET CLR version in the User-Agent.
-- Ivan Krstić krstic@solarsail.hcs.harvard.edu | GPG: 0x147C722D
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- Refije dirije lanmè yo paske nou posede pwòp bato. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l