On 30/01/07, Mark Williamson node.ue@gmail.com wrote:
No -- I don't know the particulars, but I imagine pages are not cached by Google in a single burst that puts such a huge load on the server, if this were the case most large sites would have blocked Google (myspace, livejournal, ebay) and it would be much less useful.
Wikimedia allows spidering at a certain pace - it's going very fast that isn't allowable.
Wikimedia does sell live feeds as a service. (e.g.I think Answers.com use this.)
Most mirrors use the dumps. You can see from the download page how slow it is backing up en:wp :-)
- d.