On Wed, Jan 7, 2009 at 4:43 PM, Robert Rohde rarohde@gmail.com wrote:
reduction in size (11.1 GB). Because it is still a text based format, it stacks well with traditional file compressors (bz2: 89% reduction - 1.24 GB; 7z: 91% reduction - 1.07 GB).
Ruwiki dumps currently show: pages-meta-history.xml.7z 1.3 GB
Not really all that much of a win post 7z-ing considering the current performance numbers you mentioned. (No doubt your code could be made faster... but at the same time 7z is not the state of the art in raw compression ratio)
Not that your format wouldn't have many uses... but it doesn't appear to offer significant gains for bulk transport. (in the future it would be helpful if you cited the current compressed size when comparing new compressed sizes)