-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Between other things I've been working on a distributed bzip2 compression tool which could help speed up generation of data dumps.
By trading LAN bandwidth for idle CPU elsewhere in the server cluster, an order-of-magnitude improvement in throughput seems reasonably practical; this could cut bzip2 compression time for the large English Wikipedia history dumps by a full day.
Status/documentation: http://www.mediawiki.org/wiki/dbzip2
Source: http://svn.wikimedia.org/viewvc/mediawiki/trunk/dbzip2
Updates on my (*blush*) development blog: http://leuksman.com/
I'm hoping something similar can be accomplished with 7zip as well...
- -- brion vibber (brion @ pobox.com)