On Sun, Nov 18, 2007 at 02:11:49PM -0500, David A. Desrosiers wrote:
On Sun, 2007-11-18 at 13:53 -0500, Anthony wrote:
For the immediate future a way to restart a broken dump is probably the most important. Find the last ~900K segment of the bz2 file, remove it, add the bzip2 end of file information, then concatenate the rest of the dump? Sound reasonable?
I raised something similar about 3-4 years ago, regarding the --rsyncable option of gzip, when the Wikipeda dump servers allowed us to rsync the dumps, instead of straight http.
Perhaps something similar should be revisited on the server-side, while preparing and compressing those dumps?
(waits for brion to counter with something valid to pull out the rug on my idea =)
Well, I gather the new version of rsync is *much* smarter that the old versions were about rilly, rilly big files, so perhaps this is worth revisiting.
Cheers, -- jra