Trevor,
Can we deconstruct the current parser's processing
steps and build a set
of rules that must be followed?
I think the commonly-used structures are quite clearly defined, but the
behaviour of these strange permutations is quite unspecified. The parser
output for the case reported in the bug already changed in the meantime..
I think we need to get a dump of English Wikipedia and
start using a
simple PEG parser to scan through it looking for patterns and figuring
out how often certain things are used - if ever.
I just ran an en-wiki article dump through a zcat/tee/grep pipeline:
pattern count example
------------------------------------------------------------------
^ 548498738 (total number of lines)
^; 681495
^;[^:]+: 153997 ; bla : blub
^[;:*#]+;[^:]+: 3817 *; bla : blub
^;; 2332
^[:;*#]*;[^:]*:: 41 most probably ;::
^[;:*#]*;[^:]+:: 17 ;; bla :: blub
Nested definition lists are not exactly common. Lines starting with ';;'
often appear as comments in code listings. The most common other
application appears to be indentation and emphasis. Any change in the
produced structure that keeps indentation and bolding should thus avoid
breaking pages.
Ward Cunninham had a setup that could do this sort of
thing on a
complete en-wiki dump in like 10-15 minutes, and a fraction of the dump
(still tens of thousands of article in size) in under a minute. We
supposedly have access to him and his mad science laboratory - now would
be a good time to get that going.
Will keep him in mind- we'll need to perform quite a few checks like
these while tweaking the parser. A pipeline with two grep patterns and
wc -l at the end ran just under 6 minutes on my notebook, so it is
actually quite doable. The javascript parser would take quite a bit
longer though ;)
Cheers,
Gabriel