On Sat, 10 Nov 2007 10:28:32 +1100, Andrew Garrett wrote:
On 11/10/07, Steve Sanbeg ssanbeg@ask.com wrote:
But some constructs in MW require an FSM to tokenize
Our Noodly Master can tokenise anything. I think we manage to tokenise them ourselves in the current parser, though, so I don't think he's REQUIRED to tokenise them.
OK, but the point is that some constructs are too complicated for flex to parse them with simple regular expressions, mostly because it requires some state information to find the tokens. So adding an FSM seems like the most reasonable way to tokenize for a bison parser.
Yeah, maybe not required; technically, even flex isn't required.