Ariel T. Glenn wrote:
If one were clever (and I have some code that would enable one to be clever), one could seek to some point in the (bzip2-compressed) file and uncompress from there before processing. Running a bunch of jobs each decompressing only their small piece then becomes feasible. I don't have code that does this for gz or 7z; afaik these do not do compression in discrete blocks.
Ariel
The bzip2recover approach? I am not sure how much will be the gain after so much bit moving. Also, I was unable to continue from a flushed point, it may not be so easy. OTOH, if you already have an index and the blocks end at page boundaries (which is what I was doing), it becomes trivial. Remember that the previous block MUST continue up to the point where the next reader started processing inside the next block. And unlike what ttsiod said, you do encounter tags split between blocks in a normal compression.