Excellent, thanks guys. I'm assuming that I shouldn't have to worry about
malformed xml (hopefully, haha), which makes it even easier/faster.
Alex
On Apr 6, 2012 4:43 AM, "fox" <fox91(a)anche.no> wrote:
On 06/04/2012 10:15, emijrp wrote:
2012/4/3 Alex Buie <abuie(a)archive.org
<mailto:abuie@archive.org>>
I wonder how well python's lxml handles multigigabyte XML files...
Guess we'll see :)
Pywikipediabot uses cElementTree for Python, which is fast as hell.
We've been using cElementTree for a lot of time in wiki-network
(
https://github.com/volpino/wiki-network) a suite of scripts to analyize
dumps of wikipedia, in particular for social network analysis purposes.
It's really fast even on huge dumps, like enwiki-pages-meta-history
It's open source so you are welcome to use it and contribute to the
project!
--
f.
"Always code as if the guy who ends up maintaining your code will be a
violent psychopath who knows where you live."
(Martin Golding)
() ascii ribbon campaign - against html e-mail
/\
www.asciiribbon.org - against proprietary attachments
http://about.me/fox91
_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l