On Apr 6, 2012 4:43 AM, "fox" <fox91@anche.no> wrote:

On 06/04/2012 10:15, emijrp wrote:
> 2012/4/3 Alex Buie <abuie@archive.org <mailto:abuie@archive.org>>
>     I wonder how well python's lxml handles multigigabyte XML files...
>     Guess we'll see :)
>
>
> Pywikipediabot uses cElementTree for Python, which is fast as hell.

We've been using cElementTree for a lot of time in wiki-network
(https://github.com/volpino/wiki-network) a suite of scripts to analyize
dumps of wikipedia, in particular for social network analysis purposes.
It's really fast even on huge dumps, like enwiki-pages-meta-history

It's open source so you are welcome to use it and contribute to the project!

--
f.

 "Always code as if the guy who ends up maintaining your code will be a
  violent psychopath who knows where you live."
 (Martin Golding)

()  ascii ribbon campaign - against html e-mail
/\  www.asciiribbon.org   - against proprietary attachments

http://about.me/fox91


_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l