On 11/10/07, Steve Sanbeg ssanbeg@ask.com wrote:
But some constructs in MW require an FSM to tokenize
Our Noodly Master can tokenise anything. I think we manage to tokenise them ourselves in the current parser, though, so I don't think he's REQUIRED to tokenise them.