Good evening,
this week I looked at different ways of cajoling overlapping, improperly
nested or otherwise horrible but real-life wiki content into the WikiDom
structure for consumption by the visual editor currently in development.
So far, MediaWiki delegates the sanitization of those horrors to html
tidy, which employs (mostly) good heuristics to make sense of its input.
The [HTML5] spec finally standardized parsing and error recovery for
HTML, which seems to overlap widely with what we need for the new parser
(how far?). Open-source reference implementations of the parser spec are
available in Java [VNU] that compiles to C++ and Javascript
(
http://livedom.validator.nu/) through GWT, and PHP and Python ports at
[HLib]. Modern browsers have similar implementations built in.
The reference parsers all use a relatively simple tokenizer in
combination with a mostly switch-based parser / tree builder that
constructs a cleaned-up DOM from the token stream. Tags are balanced and
matched using a random-access stack, with a separate list of open
formatting elements (very similar to the annotations in WikiDom). For
each parsing context and token combination an error recovery strategy
can be directly specified in a switch case.
The strength of this strategy is clearly the ease of implementing error
recovery. The big disadvantage is the absence of a nicely declarative
grammar, except perhaps a shallow one for the tokenizer. (Is there
actually an example of a parser with serious HTML-like error recovery
and an elegant grammar?)
In our specific visual editor application, performing a full error
recovery / clean-up while constructing the WikiDom is at odds with the
desire to round-trip wiki source. Performing full sanitation only in the
HTML serializer while doing none in the Wikitext serializer seems to be
a better fit. The WikiDom design with its support for overlapping
annotations allows the omission of most early sanitation for inline
elements. Block-level constructs however still need to be fully parsed
so that implicit scopes of inline elements can be determined (e.g.,
limiting the range of annotations to table cells) and a DOM tree can be
built. This tree then allows the visual editor to present some sensible,
editable outline of the document.
A possible implementation could use a simplified version of the current
PEG parser mostly as a combined Wiki and HTML tokenizer, that feeds a
token stream to a parser / tree builder modeled on the HTML5 parsers.
Separating the sanitation of inline and block-level elements to minimize
early sanitation seems to be quite doable.
What do you think about this general direction of building on HTML
parsers? Where should a wiki parser differ in its error recovery
strategy? How important is having a full grammar?
Gabriel
[HTML5] Parsing spec:
http://dev.w3.org/html5/spec/Overview.html#parsing
[VNU] Ref impl. (Java, C++, JS):
http://about.validator.nu/htmlparser/
Live JS parser demo:
http://livedom.validator.nu/
[HLib] PHP and Python parsers:
http://code.google.com/p/html5lib/