Hi Rami,<div><br></div><div>If I recall correctly, we use the diff library from google (<a href="http://code.google.com/p/google-diff-match-patch/">http://code.google.com/p/google-diff-match-patch/</a>)</div><div>and the total size is about 420Gb (after decompression).</div>
<div><br></div><div>But you can also just download a couple of chunks and see if you can handle those.</div><div>Best,</div><div>Diederik</div><div><br><br><div class="gmail_quote">On Fri, Nov 4, 2011 at 5:56 PM, Rami Al-Rfou&#39; <span dir="ltr">&lt;<a href="mailto:rmyeid@gmail.com">rmyeid@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div dir="ltr">Hi Diederik,<div><br></div><div>I have two questions:</div><div><ol><li>Which algorithm you used to get the added/removed content between two revisions of wikipedia?</li>
<li>What is the size of the diffdb dump after extracting? I do not want to waste wikipedia bandwidth if I know that I can not deal with it ;).</li>


</ol></div><div>By the way, what you did  is exactly what I just started working on to implement for my project, so thanks a lot :)</div><div><br></div><div>Regards.<br><br><div class="gmail_quote"><div><div></div><div class="h5">
On Fri, Nov 4, 2011 at 13:19, Diederik van Liere <span dir="ltr">&lt;<a href="mailto:dvanliere@gmail.com" target="_blank">dvanliere@gmail.com</a>&gt;</span> wrote:<br>


</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div class="h5"><span style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">Dear Wiki Researchers,</span><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">




<br></div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px"><br></div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">
During the summer we have worked on Wikihadoop [0], a tool that allows us to create the diffs between two revisions of a Wiki article using Hadoop.</div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">




Now I am happy to announce that the entire diffdb is available for download at http://<a href="http://dumps.wikimedia.org/other/diffdb/" style="color:rgb(17, 85, 204)" target="_blank">http://dumps.wikimedia.org/other/diffdb/</a></div>




<div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px"><br></div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">
This dataset is based on the English Wikipedia April 2011 XML dump files. The advantage of this dataset is that:</div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">
a) You can search for specific content being added / removed</div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">b) Measure more accurately how much text an editor has added or removed</div>




<div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px"><br></div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">
We are currently working on a Lucene-based application [1] that will allow us to quickly search for specific strings being added or removed.</div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">




<br></div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">If you have any questions, then please let me know!</div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">




<br></div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">[0] <a href="https://github.com/whym/wikihadoop" style="color:rgb(17, 85, 204)" target="_blank">https://github.com/whym/wikihadoop</a></div>




<div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">[1] <a href="https://github.com/whym/diffindexer" style="color:rgb(17, 85, 204)" target="_blank">https://github.com/whym/diffindexer</a></div>




<div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px"><br></div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">
<br></div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">Best regards,</div><span><font color="#888888"><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">

<br></div><div style="color:rgb(34, 34, 34);font-family:arial, sans-serif;font-size:13px">Diederik van Liere</div>
</font></span><br></div></div><div class="im">_______________________________________________<br>
Wiki-research-l mailing list<br>
<a href="mailto:Wiki-research-l@lists.wikimedia.org" target="_blank">Wiki-research-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/wiki-research-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/wiki-research-l</a><br>
<br></div></blockquote></div><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Rami Al-Rfou&#39;<div><a href="tel:631-371-3165" value="+16313713165" target="_blank">631-371-3165</a></div></div>
<br>
</font></div></div>
<br>_______________________________________________<br>
Wiki-research-l mailing list<br>
<a href="mailto:Wiki-research-l@lists.wikimedia.org">Wiki-research-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/wiki-research-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/wiki-research-l</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>&lt;a href=&quot;<a href="http://about.me/diederik" target="_blank">http://about.me/diederik</a>&quot;&gt;Check out my <a href="http://about.me" target="_blank">about.me</a> profile!&lt;/a&gt;<br>

</div>