On 9 August 2010 17:04, Mark Clements (HappyDog) <gmane(a)kennel17.co.uk> wrote: How prevalent is it in actual wikitext? Is it an edge case people actually use much, or are all instances of it basically errors? That'll be the question. - d.

2010-08-10 12:45, Thomas Dalton skrev: If you consider the large body of information tied to MediaWiki syntax, it is likely that for any border case, there is a revision of some page that will trigger that border case. Regarding strategy on how to replace the MediaWiki parser, I can see two extremes: 1. Search out all wierd edge cases and reproduce them in parser rules. Walk through the revisions of Wikipedia and for each edge case, note all revisions for which the parser rule for the edge case is executed. Based on the data determine which edge cases can be safely removed. Or define a conversion for the content. 2. Don't support any edge cases. Just consider the content broken and let the wiki users themself fix it. Historic revisions of pages will be permanently broken. I am trying to support as many edge cases as far as reasonable in my attemt to write a new parser. I seems, however, as if the parser is actively developed, and backwards compliancy with edge cases maybe isn't much of a concern. For instance, in 1.16.0beta3 we have: $text = $this->doAllQuotes( $text ); $text = $this->replaceInternalLinks( $text ); $text = $this->replaceExternalLinks( $text ); which in trunk is: $text = $this->replaceInternalLinks( $text ); $text = $this->doAllQuotes( $text ); $text = $this->replaceExternalLinks( $text ); So, it is now possible to have apostrophes in internal links, but still not in external. From the parser's point of view, the edge cases can be divided into "harmless", where a rule to support it does not increase the complexity of the parser significantly, and "harmful", where adding a rule to support them would either dramatically increase the size of the parser or make it possible to craft contents that will take more than linear time or memory to process. The edge cases surrounding links definitely fall into the harmful category. I will be writing a separate post about links later. Maybe it would be a good idea to provide som feedback to the user regarding bad syntax. In my parser implementation, I am considering generating special events for syntax that should be avoided. For instance: begin_table: begin = BEGIN_TABLE NEWLINE* ( { X->beginGarbageBlock(X, "Unsupported syntax: content between the {| and the first column in a table."); } ((inline_element)=> garbage_inline_text NEWLINE* )* block_elements? { X->endGarbageBlock(X); } )* { X->beginTable(X, $begin->custom); } ; Could for instance be rendered in html as: <div class="garbage" title="Unsupported syntax: content between ..."> </div>. /Andreas

On 8/11/10, Daniel Kinzler &lt;daniel(a)brightbyte.de&gt; wrote: This made me think of Quirks mode in browsers. We can make the new parser (and standardized wikitext syntax) more strict by treating many edge cases as errors, and if the new parser detected any error when parsing, use the old parser instead.

On Wed, Aug 11, 2010 at 6:02 PM, David Gerard &lt;dgerard(a)gmail.com&gt; wrote: Do we have a short list of "worst case scenario" pages, which use lots of special cases for some reason, and that we could use as a test set? Not something specially constructed, but real, live wikipedia pages. Magnus

On 12 August 2010 08:35, Daniel Kinzler &lt;daniel(a)brightbyte.de&gt; wrote: Yes. This is important: humans do not naturally write correctly-formed XML - they splatter a bunch of tag soup on a page, see if it rendered properly in "preview" and save when it looks good enough. That is, they learn the subtle nuances of wikitext the way they learn natural language. Trying to turn wikitext into a proper language, with invalid formations, will not I predict work well. Unless you aim for humans mostly not to use wikitext and instead to use only WYSIWYG or other structured editors. This would be a major change in the way WMF wikis work, however, and may not fly. There are some examples that clearly break. If you fail to close a <ref> tag, watch the sea of red in the rendering. So you may be able to push wikitext towards being something where syntax erriors exist. If that is considered a desirable goal. (I don't consider it one, but then again I want everything and a pony, like any unreasonable human does.) - d.