[Wikitext-l] On parsing tokenized wikitext

[Andreas Jonsson]

In my previous post I covered the lexer.  Here I will describe the
parser, the parser context and the listener interface.  After the
lexer's extensive job att providing a reasonably well formed token
stream, the parser's job becomes completely straightforward.

== The parser

For inlined elemements, the parser will just mindlessly report these
to the context object:

inline_element: 
word|space|special|br|html_entity|link_element|format|nowiki|table_of_contents|html_inline_tag 

     ;

space: token = SPACE_TAB  {IE(CX->onSpace(CX, $token->getText($token));)}
     ;

etc.

The lexer guarantees that a closing token will not appear before
a corresponding opening token, and the parser context takes care of
nesting formats and removing empty format tags.

For block elements, the only special thing the parser need to pay
attention to is the fact that end tokens may be missing.  Therefore,
end-of-file is always accepted instead of the closing token, for
instance:

html_div:
     token = HTML_DIV_OPEN
     {
         CX->beginHtmlDiv(CX, $token->custom);
     }
     block_element_contents
     (HTML_DIV_CLOSE|EOF)
     {
         CX->endHtmlDiv(CX);
     }
     ;

The rule 'block_element_contents' covers all parser productions.  The
lexer will restrict which tokens that may appear.  For instance
'HTML_DIV_CLOSE' will never appear before a corresponding
'HTML_DIV_OPEN'.  Also, list items and table cells will not appear
unless the current block context is correct.  I have also introduced
a max nesting level limit in the lexer, so stack space is also not an
issue.

== The parser context

The parser context relays the parser events to a listener, but it will
insert and remove events to produce a well formed output.  For instance:

text '' italic <b><strong /> bold-italic
bold </b> text

will result in an event stream to the listener that will look like this:

text <i> italic <b> bold-italic </b></i>
<b> bold </b> text

Two mechanisms are used to implement this:

* The call to the "begin" method is delayed until some actual inlined
   content is produced.  The call is never taken if an "end" event is
   recieved before such content.

* The order of the formats is maintained so that inner formats can be
   closed and reopened when a non-matching end token is recieved.

So, most of the parser context's methods look like this:

static void
beginHtmlStrong(MWPARSERCONTEXT *context, pANTLR3_VECTOR attr)
{
     MW_DELAYED_CALL(        context, beginHtmlStrong, endHtmlStrong, 
attr, NULL);
     MW_BEGIN_ORDERED_FORMAT(context, beginHtmlStrong, endHtmlStrong, 
attr, NULL, false);
     MWLISTENER *l = &context->listener;
     l->beginHtmlStrong(l, attr);
}

static void
endHtmlStrong(MWPARSERCONTEXT *context)
{
     MW_SKIP_IF_EMPTY(     context, beginHtmlStrong, endHtmlStrong, NULL);
     MW_END_ORDERED_FORMAT(context, beginHtmlStrong, endHtmlStrong, NULL);
     MWLISTENER *l = &context->listener;
     l->endHtmlStrong(l);
}

Block elements are already guaranteed by the lexer to be well nested,
so the context typically does not need to do anything special about
those.  Only the wikitext list elements needs to be resolved by the
context.

== The listener

The listening application needs to implement the MWLISTENER interface.
I haven't added support for all features yet, but at the moment, there
are 91 methods in this interface.  They are trivial to implement,
though.  The only thing to think about is that it is the listener's
responsibility to escape the contents of nowiki and special
characters, and also to filter the attribute lists.

/Andreas