[Wikitext-l] Parser status update

[Gabriel Wicke]

Hello,

so far the HTML5 parser integration seems to have turned out quite well.
180 parser tests are now passing, and most of the remaining ones are
about missing functionality in later stages of the parser pipeline.

Since this week, the produced HTMl DOM can also be converted to WikiDom
(or close to it). A sample result of the [[en:Barack Obama]] article is
available at http://dev.wikidev.net/gabriel/tmp/obama.wikidom.txt. The
unoptimized parse to WikiDom without template expansions etc currently
takes about 35 seconds on my laptop.

The various moving parts of the setup (and how to try it out) are
described in https://www.mediawiki.org/wiki/Future/Parser_development.
In glorious ASCII, it might look roughly like this:

PEG wiki/HTML tokenizer         (could also be any SAX-style parser)
-> Token stream transformations
-> HTML5 tree builder
-> HTML DOM tree
-> DOM Postprocessors +-> (X)HTML
                      +-> DOMConverter -> WikiDom -> Visual Editor

The tokenizer is built from a completely static grammar, and leaves all
configuration-dependent behavior to later stages. Most interesting bits
happen in token stream transformations, which are dispatched using a
registration mechanism by token type. The order of handlers can be
specified, and early handlers can abort further processing for a token.
Syntax-specific transformations on a token can register for early
processing, so that later transformations on a token can operate on a
normalized version of the token. MediaWiki's special quote handling for
italic/bold for example is implemented in a core extension that
registers handlers for 'quote', 'newline' and the special 'eof' token.
Lists and a simple version of the Cite extension are similarly
implemented. A general emulation of parser hook behavior on top of the
token stream is quite straightforward. Both collected tokens between
tags and plain text based on source positions noted in tokens are available.

The token transform dispatcher class is prepared for asynchronous
processing of tokens, which is already used in a synchronous fashion for
the back-reference behavior of the italic/bold extension. This ability
to overlap operations on multiple tokens will be very important for
template expansions. Doing template expansions on the token level makes
it possible to render unbalanced templates like the table start / row /
end combinations for viewing, while encapsulating those if the output is
destined for the visual editor. Template expansion is currently WIP.

So far for now, looking forward to your thoughts!

Gabriel