Re: [Wikitech-l] WikiDump Parsing

You'll do much better with a purpose-built lexical analyzer. If nothing else, this'll work when titles and tags span across buffers - something which the previous program will choke on on.

Released into the public domain as a trivial example program. Save as 'titleparse.l' and 'make titleparse'.

%option noyywrap

%{ #include <stdio.h> %}

%x TITLE

%%

<INITIAL>"<title>" { BEGIN TITLE; } <TITLE>"</title>" { BEGIN INITIAL; putchar('\n'); } <TITLE>"&lt;" { putchar('<'); } <TITLE>"&gt;" { putchar('>'); } <TITLE>"&quote;" { putchar('"'); } <INITIAL>.|\n /* ignored */

%%

int main(int argc, char *argv[]) { yylex(); exit(0); }