First of all, I have to admit I have not read all 50 emails, but here are my two cents.

Most importantly, I think we should stop storing wikitext. Storing wikitext makes it hard to make changes in the syntax, because it would break pretty much every existing page. Wikitext is an ambiguous way of storing 'the way it is meant'; XML is a clear way of doing this. As the text is compressed, using wikitext or XML does not make that big of a difference.

However, XML makes parsing much easier. Yes, it will need two steps, but when regenerating the page from the database, it's much easier (no ugly regexps, just a simple SAX parser). Besides, as a pywikipedia developer, I'd like to have XML output ;)

To change the format to XML (and updating the wikitext format at the same time) means we need four important things: an 'old wikitext'->XML converter, a XML->'good wikitext' converter, a 'good wikitext'->XML converter and a XML->HTML parser. (s/converter/parser, if you care about the exact words). The 'good wikitext' and html parsers should be fairly easy; the first is just plain hard.

I have tried to build a parser by using standard systems, but I have given up, and I have built a basic lexer + parser by hand. It is by no means complete, and I have not worked on it for some time, as Ping Yeh has a more complete implementation. He was busy refactoring it, but with little time, there was little progress. My code is available at http://svn.wikimedia.org/viewvc/pywikipedia/trunk/pywikiparser/ - feel free to take a look at it :)

To summarize: We should switch to storing a much more descriptive format so changes in the wikitext format do not break anything: the wikitext can just be generated from the XML, in whichever format you want. This means it should be able to use (cleaned up) mediawiki wikitext, wikicreole or many other systems - per user. (Although as far as I can see wikicreole isn't available as context free grammar either..)

--valhallasw

P.S. Some people, when confronted with a problem, think `I know, I'll use regular expressions.' Now they have two problems. --jwz