[Foundation-l] Public repositories for research dumps

Felipe Ortega glimmer_phoenix at yahoo.es
Tue Jun 23 16:26:40 UTC 2009


Since just a few hours ago, a new public repository has been created to host WikiXRay database dumps, containing info extracted from public Wikipedia dbdumps. The image is hosted by RedIRIS (in short, the Spanish equivalent of Kennisnet in Netherlands).



These new dumps are aimed to save time and effort to other researchers, since they won't need to parse the complete XML dumps to extract all relevant activity metadata. We used mysqldump to create the dumps from our databases.. 

As of today, only some of the biggest Wikipedias are available. However,  in the following days the full set of available languages will be ready for downloading. The files will be updated regularly.

The procedure is as follows:

1. Find the research dump of your interest. Download and decompress it in your local system.

2. Create a local DB to import the information.

3. Load the dump file, using a MySQL user with insert privileges:

$> mysql -u user -p passw myDB < dumpfile.sql

And you're done.

Final warning. 3 fields in the revision table are not reliable yet:


All remaining fields/values are trustable (in particular rev_len, rev_num_words, and so forth).




More information about the foundation-l mailing list