Tim Starling wrote:
My idea for faster, cheaper access to Wikipedia in poorer countries:
http://meta.wikimedia.org/wiki/Reducing_transit_requirements
It requires patching squid and extending the HTCP protocol. The latter should be no big deal, the protocol is only in the "experimental" phase after all.
On reflection, inventing new protocols for taking a real-time feed is probably uncessary for what will be a batch operation.
How about simply making plain-ASCII logfiles available, where each line consists of
<timestamp> <url>\n
which is grown in real-time by appending to it?
People wanting to freshen their caches can then download, trim off all entries previous to the last time they fetched, uniq the file to get only a single copy of each url, and then run a script to wget all the uls from inside their cache during off-peak hours, thus freshening it.
Very little software, a twenty-line perl script, no new protocols, and (I hope) achieves the same effect.
-- Neil