RE: [Wikitech-l] Re: Yahoo! XML feed - Wikitech-l

2 Apr 2004

...

On Apr 2, 2004, at 00:53, Jimmy Wales wrote:
  It shouldn't run more than once per day at
first.  I'm not sure what
 their goals are with respect to how often they would *like*  to receive
  it, but daily is a fine start. 
It would take hours just to run a complete dump, which would be the 
equivalent of a sizeable fraction of our total daily page views. (Best 
case might be 100ms per page for 240,000 pages =~ 6 hours 40 minutes)

If we're going to run something like this daily, some sort of 
incremental updates are a must, though we can probably get away with 
stuffing the saved data per page in a database or such and slurping it 
back out fairly quickly.

-- brion vibber (brion @ pobox.com) 
Incremental need to check in the database each time if this is the last version, no ?

Why not using a slave db server on less loaded server ? (like zwinger for example (need to
add another ide drive))

Shaihulud