On 1/7/09 9:16 PM, Robert Rohde wrote:
Yes, you are right about that. For bulk transport and
storage it is
not a big improvement.
However, to work with ruwiki, for example, one generally needs to
decompress it to the full 170 GB. To work with enwiki's full revision
history, if such a dump is ever to exist again, would probably
decompress to ~2 TB. 7z and bz2 are not great formats if one wants to
extract only portions of the dump since there are few tools that would
allow one to do so without first reinflating the whole file. Hence,
one of the advantages I see in my format is being able to have a dump
that is still<10% the full inflated size while also being able to
parse out selected articles or selected revisions in a straightforward
manner.
*nod* this is an attractive option for that sort of case; you can pull
the metadata and grep for interesting changes in a much more manageable
file.
Note we'll also want to be considering different options for breaking up
the dump into smaller files, which makes parallel generation more
feasible as well as assisting some downloaders.
-- brion