[Mediawiki-l] Tokenizer problems

[Timwi]

Hi,

is someone currently working on the code in CVS? The way it is now, it 
doesn't seem to work.

Specifically, the following input text:

	''Test''

generates this output:

<p><font color="#FF0000"><b>'</b></font><font 
color="#FFFF00"><b></b></font><font color="#FF0000"><b>'</b></font><font 
color="#FFFF00"><b></b></font>'Test''

It seems that the Tokenizer thinks each apostrophe is an individual 
token, but it should consider '' as a single token.

Could someone fix this please?
Thanks,
Timwi