2008/10/11 Anthony wikimail@inbox.org:
On Sat, Oct 11, 2008 at 12:32 PM, Thomas Dalton thomas.dalton@gmail.comwrote:
Exactly. The only way you can speed up the smaller dumps is the slow down the bigger ones (or throw more money at the problem), and no-one has given any reason why we should prioritise smaller dumps.
What about the original idea, to do the "current" dumps in a separate thread from the "history" dumps. With only one thread doing the "history" dumps they'd take a really long time, but at least the "current" dumps would finish in a reasonable amount of time.
There's something in that. Speed up the quick parts of all the dumps at the expense of the slow parts of all the dumps. It's fair to each project and probably correlates well with how people actually use the dumps (do we have download logs to see how often people actually download the whole thing?).