[Resending without the full list of articles, which caused the message to be bounced into moderation.]<br><br><div class="gmail_quote">Here are the results of a quick test I ran over the weekend, comparing a compressed excerpt from <a href="http://simple.wikipedia.org" target="_blank">simple.wikipedia.org</a> in mediawiki markup to the compressed parsoid representation of the same articles.  The list of articles is attached to this message. [Not any more.]<div>

<br></div><div>For the base case I used the processing pipeline for the OLPC&#39;s &quot;Wikipedia activity&quot;, source code at <a href="http://github.com/cscott/wikiserver" target="_blank">github.com/cscott/wikiserver</a></div>
<div>It begins with a hand-written &quot;portal page&quot;, then grabs all articles within two links of the portal page.  The original markup was taken from the simplewiki-20130112-pages-articles.xml dump.  Templates were then fully expanded, and just the selected articles were written.  Articles are separated by the character 0x01, a newline, the title of the article, a newline, the length of the article in bytes, a newline, and the character 0x02 and a newline.</div>

<div><br></div><div>For comparison, I took the list of articles included in the dump and wrote a small script to fetch them from parsoid, using the HEAD of the master branch from this weekend (2013-02-17, roughly).  I wrote the full parsoid HTML document (including top-level &lt;html&gt; tag, &lt;head&gt;, &lt;base href&gt;, and &lt;body&gt; but not including a &lt;!DOCTYPE&gt;) to a file, separating articles with the title of the article, a newline, the length of the article in bytes, and a newline.</div>

<div><br></div><div>Results, with and without compression:</div><div><br></div><div># of articles: 3640</div><div>Mediawiki markup, uncompressed: 18M</div><div>Parsoid markup, uncompressed: 199M</div><div><br></div><div>
Mediawiki markup, gzip -9 compressed: 6.4M</div>
<div>Parsoid markup, gzip -9 compressed: 26M</div><div><br></div><div>Mediawiki markup, bzip2 -9 compressed: 4.7M</div><div>Parsoid markup, bzip2 -9 compressed: 17M</div><div><br></div><div>Mediawiki markup, lzma -9 compressed: 4.4M</div>

<div>Parsoid markup, lzma -9 compressed: 15M</div><div><br></div><div>So there&#39;s currently a 10x expansion in the uncompressed size, but only 3-4x expansion with compression.</div><span class="HOEnZb"><font color="#888888"><div>
  --scott</div><div><br></div><div>
-- <br>                         ( <a href="http://cscott.net/" target="_blank">http://cscott.net/</a> )
</div>
</font></span></div><br>