Detailed technical report on an undergraduate student project at Virginia Tech (work in progress) to import the entire English Wikipedia history dump into the university&#39;s Hadoop cluster and index it using Apache Solr, to &quot;allow researchers and developers at Virginia Tech to benchmark configurations and big data analytics software&quot;:<br>
<br>
Steven Stulga, &quot;English Wikipedia on Hadoop Cluster&quot;<br>
<a href="https://vtechworks.lib.vt.edu/handle/10919/70932" target="_blank">https://vtechworks.lib.vt.edu/handle/10919/70932</a> (CC BY 3.0)<br>
<br>
IIRC this has rarely or never been attempted due to the large size of the dataset - 10TB uncompressed. And it looks like the author here encountered an out of memory error that he wasn&#39;t able to solve before the end of term...<br><br>
--<br>
Tilman Bayer<br>
Senior Analyst<br>
Wikimedia Foundation<br>
IRC (Freenode): HaeB<br>
<br><br>-- <br>Sent from Gmail Mobile<br>