Excellent, thanks guys. I'm assuming that I shouldn't have to worry about malformed xml (hopefully, haha), which makes it even easier/faster.
Alex On Apr 6, 2012 4:43 AM, "fox" fox91@anche.no wrote:
On 06/04/2012 10:15, emijrp wrote:
2012/4/3 Alex Buie <abuie@archive.org mailto:abuie@archive.org> I wonder how well python's lxml handles multigigabyte XML files... Guess we'll see :)
Pywikipediabot uses cElementTree for Python, which is fast as hell.
We've been using cElementTree for a lot of time in wiki-network (https://github.com/volpino/wiki-network) a suite of scripts to analyize dumps of wikipedia, in particular for social network analysis purposes. It's really fast even on huge dumps, like enwiki-pages-meta-history
It's open source so you are welcome to use it and contribute to the project!
-- f.
"Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live." (Martin Golding)
() ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments
Xmldatadumps-l mailing list Xmldatadumps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l