Nick Jenkins wrote:

Can the intermediate backup files please be placed in a separate directory (e.g. http://download.wikimedia.org/wikipedia/en/in-progress/ ), and only be moved to the real directory once they are complete and there were no errors (such as running out of disk space), so as to avoid confusion about what is and what is not a complete and valid dump?

I don't think there's any point in that; the only reason anyone would look in the directories is because the new front-end interface to download.wikimedia.org hasn't been written yet and we took down the old interface temporarily because it was pointing at the wrong files.

========================================================= 20050713_pages_full.xml.gz 2005-Jul-16 16:25:28 29.9G 20050924_pages_full.xml.bz2 2005-Oct-01 03:52:03 11.3G 20051002_pages_full.xml.bz2 2005-Oct-03 19:35:09 1.7G =========================================================

I have to presume that the "20051002_pages_full.xml.bz2" is an in-progress dump, because it is one tenth of the size of the file of the week before, despite using the same compression.

Indeed, in-progress.

Then, once you start to second-guessing whether something is a complete dump or not, it opens a can of worms: you then have to ask whether the 20050924_pages_full.xml.bz2 dump is complete, given that the previous dump is nearly 3 times the size. Or maybe it's because of the difference between gzip and bzip2 .... who can say for sure?

bzip2 does about 3-5 times better on the full-history dumps than gzip does; 7zip does another 3-5 times better than bzip2.

-- brion vibber (brion @ pobox.com)