<br>Hi,<br>
<br>
I think we should keep at least one version of a recent bz2 enwiki
pages-meta-history file because there are already some programs that use
the bz2 format directly, and I don't know of any program that uses the
7z format natively.<br>
<br>
heres some offline wiki readers that use the bz2 format:<br>
bzreader: http://code.google.com/p/bzreader/<br>
mzReader: http://homepage.ntlworld.com/bharat.vadera/MzReader/<br>
wikitaxi: http://www.wikitaxi.org/<br>
(note that none of these programs are currently setup for viewing the
pages-meta-history revision data or discussion pages)<br>
<br>
If there is no pages-meta-history in bz2 format available (currently
280GB for enwiki) then the 7z file will have to be converted to bz2
unless its possible to interface directly to the 7z file efficiently if
this is even possible. Since the 7z file will decompress to 5469GB as
Kevin showed, I think it would be hard for most people to decompress
this 7z file, but the 280GB bz2 file is still a reasonable size and can
be used without decompressing. So I think keeping at least a single
recent bz2 file would be the way to go. The dewiki keeps about 6 of
their pages-meta-history bz2 files (around 75GB each =450GB storage)
http://download.wikimedia.org/dewiki/ so I think enwiki should be able
to keep at least one, especially after all this time of not having any
of these files for enwiki.<br>
<br>
Also I wonder if it is possible to convert from 7z to bz2 without having
to make the 5469GB file first? If this can be done then having only 7z
files would be fine, as the bz2 file could be created with a "normal"
PC (ie one without a 6TB+ harddrive). This would be a good solution,
but not sure if it can be done. If it could though, might as well get
rid of all the large wiki's bz2 pages-meta-history files!<br>
<br>
cheers,<br>
Jamie<br>
<br><br>----- Original Message -----<br>From: Tomasz Finc <tfinc@wikimedia.org><br>Date: Tuesday, March 16, 2010 12:45 pm<br>Subject: Re: [Xmldatadumps-admin-l] 2010-03-11 01:10:08: enwiki Checksumming pages-meta-history.xml.bz2 :D<br>To: Kevin Webb <kpwebb@gmail.com><br>Cc: Wikimedia developers <wikitech-l@lists.wikimedia.org>, xmldatadumps-admin-l@lists.wikimedia.org, Xmldatadumps-l@lists.wikimedia.org<br><br>> Kevin Webb wrote:<br>> > I just managed to finish decompression. That took about 54 <br>> hours on an<br>> > EC2 2.5x unit CPU. The final data size is 5469GB.<br>> > <br>> > As the process just finished I haven't been able to check the<br>> > integrity of the XML, however, the bzip stream itself appears <br>> to be<br>> > good.<br>> > <br>> > As was mentioned previously, it would be great if you could compress<br>> > future archives using pbzib to allow for parallel <br>> decompression. As I<br>> > understand it, the pbzip files are reverse compatible with all<br>> > existing bzip2 utilities.<br>> <br>> Looks like the trade off is slightly larger files due to <br>> pbzip2's <br>> algorithm for individual chunking. We'd have to change the<br>> <br>> buildFilters function in http://tinyurl.com/yjun6n5 and install <br>> the new <br>> binary. Ubuntu already has it in 8.04 LTS making it easy.<br>> <br>> Any takers for the change?<br>> <br>> I'd also like to gauge everyones opinion on moving away from the <br>> large <br>> file sizes of bz2 and going exclusively 7z. We'd save a huge <br>> amount of <br>> space doing it at a slightly larger cost during compression. <br>> Decompression of 7z these days is wicked fast.<br>> <br>> let know<br>> <br>> --tomasz<br>> <br>> <br>> <br>> <br>> <br>> <br>> _______________________________________________<br>> Xmldatadumps-admin-l mailing list<br>> Xmldatadumps-admin-l@lists.wikimedia.org<br>> https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-admin-l<br>>