On 30/01/07, Mark Williamson <node.ue(a)gmail.com> wrote:
No -- I don't know the particulars, but I imagine
pages are not cached
by Google in a single burst that puts such a huge load on the server,
if this were the case most large sites would have blocked Google
(myspace, livejournal, ebay) and it would be much less useful.
Wikimedia allows spidering at a certain pace - it's going very fast
that isn't allowable.
Wikimedia does sell live feeds as a service. (e.g.I think
Answers.com use this.)
Most mirrors use the dumps. You can see from the download page how
slow it is backing up en:wp :-)
- d.