On Apr 2, 2004, at 01:39, Timwi wrote:
If the problem is that each article has navigational elements on the top, bottom, and left, which multiplies the time required to crawl things, then how about allowing something like &raw=1 which would output only the parsed page, without a skin or anything, with all the links in it also pointing to &raw=1, and then arranging with Yahoo to have them spider these pages, but still send people to the URL without &raw=1?
Then they'll still have to make 240,000+ HTTP connections to check every individual page for updates, which can take days or weeks depending on the crawl delay.
-- brion vibber (brion @ pobox.com)