It would be really nice to be able to include hooks after the lexer, but before actual parsing.
That could be done, but I would not recommend it. What application do you have in mind?
Well, the current implementation of my editor uses a bunch of regexes (like the current parser) to determine where to inject spans or divs into the wikitext. Having a more accurate representation (the tokenized wikitext that the lexer outputs) would allow for more accurate injection. Then again, it would be complicated to interface that with PHP, I guess?
How would you handle hooks, tag extensions, parser functions and magic words anyway? Will you leave this to some post-processing stage in PHP or have things interact during parsing?
Regards, Jan Paul