On 4/5/2011 9:38 AM, Samat wrote: I did a substantial project that worked from the XML dumps. I designed a recursive descent parser in C# that, with a few tricks, almost decodes wikipedia markup correctly. Getting it right is tricky, for a number of reasons, however, my approach preserved some semantics that would have been lost in the HTML dumps. If I had to do it all again, I'd probably download a copy of the mediawiki software, load Wikipedia into it, and then add some 'hooks' into the template handling code that would let me put traps on important templates and other parts of the markup handling so template nesting gets handled correctly. Old static dumps, going back to June 2008, are available: http://static.wikipedia.org/ In your case, I'd do the following: install a copy of the mediawiki software, http://lifehacker.com/#!163707/geek-to-live--set-up-your-personal-wikipedia <http://lifehacker.com/#%21163707/geek-to-live--set-up-your-personal-wikipedia> get a list of all the pages in the wiki by running a database query, and then write a script that does http requests for all the pages and saves them in files. This is programming of the simplest type, but getting good speed could be a challenge. I'd seriously consider using Amazon EC2 for this kind of thing, renting a big DB server and a big web server, then writing a script that does the download in parallel.

On 4/5/2011 4:00 PM, Platonides wrote Could be true, but what's fascinating for me about Wikipedia is all of the unscrambled eggs that can be found in the middle of otherwise unstructured text.