Hello and happy new year to all fellow xml-crunchers out there. Anyone
who uses the bz2 file format of the page dumps, either for hadoop or for
some other reason might be interested in the following:
(For en wikipedia only for right now)
I've written a little program that will write a second copy of the
pages-articles bz2 file, as concatenated multiple bz2 streams with 100
pages per stream, with a separate index bz2 file which contains a list
of offsets/page ids/page titles where each offset is to the start of the
particular bz2 stream in the file.
The nice thing about multiple streams is that essentially these behave
like separate bz2 files, so you can just seek to that point in the file,
pass the data starting from that byte directly into the bz2 decompresser
of your choice, and work with it. No need to monkey around with
bit-aligned crap, nor with fudging together a bz2 header to fool the
library into thinking it's looking at a full file, nor with tossing away
the crc at the end.
The 100-pages-per stream makes the output somewhat bigger than the
regular pages-articles file but not excessively so.
I'm hoping this format will be useful to folks working with offline
readers, hadoop or other analysis tools. Let me know.
A first run of these, for the December en wp pages-articles file, is
available at
http://dumps.wikimedia.org/other/multistream/
and the second run (assuming this works, I just deployed the changes to
the python scripts) will be generated in the usual way as part of the
regular dumps and found with them at the normal location.
If the job runs ok I expect to enable it on the other projects soon
afterwards.
Ariel