-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Between other things I've been working on a distributed bzip2 compression tool
which could help speed up generation of data dumps.
By trading LAN bandwidth for idle CPU elsewhere in the server cluster, an
order-of-magnitude improvement in throughput seems reasonably practical; this
could cut bzip2 compression time for the large English Wikipedia history dumps
by a full day.
Status/documentation:
http://www.mediawiki.org/wiki/dbzip2
Source:
http://svn.wikimedia.org/viewvc/mediawiki/trunk/dbzip2
Updates on my (*blush*) development blog:
http://leuksman.com/
I'm hoping something similar can be accomplished with 7zip as well...
- -- brion vibber (brion @
pobox.com)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)
Comment: Using GnuPG with Mozilla -
http://enigmail.mozdev.org
iD8DBQFEfVkSwRnhpk1wk44RAv/wAJ9JJghnGX6wmPvVz8lX6WLa+sfeZwCg1wex
ZcadcVpBCsZP866C4gdG7/M=
=SpCq
-----END PGP SIGNATURE-----