Jay Ashworth wrote: ("ding, dong. ding, dong.") There's <http://www.mediawiki.org/wiki/Markup_spec> which is a prose description, with several more formal BNF descriptions hanging off of it as subpages (see <http://www.mediawiki.org/ wiki/Special:Search/Markup_spec/BNF/> for a list).

On 8/16/06, Ivan Krstić &lt;krstic(a)solarsail.hcs.harvard.edu&gt; wrote: Out of curiosity, why not? What bits of markup screw over the project?

*sighs* Well, that's my cue. My sincere apologies for when this starts to ramble and lose coherency - I'm a bit tired today, and I've been focusing mostly on other things since the Wikimania Hacking Days. Let's see here. Please consider this an incomplete, unreliable list, meant solely as an indication of the basic problems encountered when attempting to formalize MediaWiki's wikitext... And I'm no expert on parsing, except in that I've spent a large part of the summer constructing parsers for essentially unparseable languages. Basic point, though, is that MediaWiki wikitext is INCREDIBLY context-sensitive. Single case that shows something interesting: '''hi''hello'''hi'''hello''hi''' Try running it through MediaWiki, and what do you get? <b>hi<i>hello</i></b><i>hi<b>hello</b></i><b>hi</b> In other words, you've discovered that the current syntax supports improper nesting of markup, in a rather unique fashion. I don't know of any way to duplicate this in any significantly formal system, although I believe a multiple-pass parser *might* be capable of handling it. In fact, some sort of multiple-pass parser (the MediaWiki parser) obviously can. Also, templates need to be transcluded before most of the parsing can take place, since in the current system, the text may leave some syntactically-significant constructs incomplete, finishing them in the transclusion stage... Beyond that... *pulls up his mostly-aborted attempt at a parser* Indefinite lookahead is required in some places, particularly in headings and magic variables - and for almost any other multi-part markup, if we want to do what the current parser does (ignore incomplete markup, treating it as if it had been properly escaped). This even holds true for bold and italics, since you need indefinite lookahead to be able to tell whether the first three quotes in '''this'' should be parsed as ''', <i>', or <b>. The situation gets even worse when you try to allow for improper nesting. Other places require fixed, but large, amounts of lookahead... freelinks require at least 9 characters, for example. Technically, I'll admit that a GLR parser (or a backtracking framework) could manage even the indefinite lookahead that I mentioned... but it's still problematic, since the grammar is left ambiguous in certain cases. Oh, right - and we'd need to special-case every tag-style piece of markup, including every allowed HTML tag, since formal grammars generally can't reference previously-matched text. This also applies to the heading levels - we'd need separate ad-hoc constructs for each level of heading we wanted to support, duplicating a lot of the grammar between each one. There are other complications as well - again, this list should be considered both incomplete and, possibly, inconsistent with reality. In summary, for most definitions of formal, it is impossible to write formal grammars for most significant subsets of current MediaWiki syntax. I had significant success with a regex-based grammar specification (using Martel), backed by a VERY general backend capable of back-tracking and other clever tricks (mxTextTools) - but the recursive structure is virtually impossible to handle in a regex-based framework. - Eric Astor P.S. As indicated above, I honestly feel that the difficulties aren't insurmountable - if you're willing to build an appropriate parsing framework, which will be semi-formal at best. P.P.S. When possible, in my *copious* free time (</sarcasm>), I'm hoping to take another frontend to mxTextTools (SimpleParse, to be specific), modify it sufficiently to support all the necessary features, and then build something capable of parsing the current MediaWiki syntax (although I might have to drop support for improper nesting). I've no idea if or when this might happen, but I'm considering it a long-term goal if the current situation doesn't improve. -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.10.10/419 - Release Date: 8/15/2006

On Thu, Aug 17, 2006 at 12:23:57AM -0400, Simetrical wrote: It wouldn't help. The problems are semantic, not syntactical. I think. :-) Cheers, -- jr 'IE: no, no, no, no, no!' a -- Jay R. Ashworth jra(a)baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274 The Internet: We paved paradise, and put up a snarking lot.

On Thu, Aug 17, 2006 at 01:56:29AM -0400, Simetrical wrote: Well that's fine... but *getting from Eric's example to yours* is a problem of semantics: what did the user *mean*. Without disambiguating rules, there may be no way to tell. Please make the following sentence gramatically correct, solely by adding punctuation: Semantics. Fine, but a) those are not the only problems he's having, and b) No [[Flag Day]]s. Cheers, -- jra -- Jay R. Ashworth jra(a)baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274 The Internet: We paved paradise, and put up a snarking lot.

On 8/17/06, Steve Bennett &lt;stevage(a)gmail.com&gt; wrote: It's at the bottom of the English [[Flag Day]] article: "Flag day is also a term used in discussing computer systems to denote a change which will require a complete restart or conversion of a sizable body of software or data. This usage of the term originates from an obscure such change in the [[Multics]] operating system's definition, which was scheduled for the US's Flag Day, June 14th, 1966. [http://www.catb.org/jargon/html/F/flag-day.html]"

On Thu, Aug 17, 2006 at 05:21:00PM +0200, Steve Bennett wrote: I added it; you'll note my edit comment, querying whether it was time for a forman disambig page. I didn't put it up top because it seemed awfully cluttered up there already. More details are in the linked Jargon file entry. Cheers, -- jra -- Jay R. Ashworth jra(a)baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274 The Internet: We paved paradise, and put up a snarking lot.

On 8/17/06, Jay R. Ashworth &lt;jra(a)baylink.com&gt; wrote: So what if we had a "lossless" wikisyntax to XML converter? It seems like that wouldn't be an impossibility (given we're already parsing wikisyntax to _HTML_). What are the reactions to e.g. converting the backend to use that XML storage, then enforcing it on the editor side, as well? Obviously we'd have to be clever on the conversion (like making VERY sure it's a "lossless" switch, and finding a computationally feasible way to get it done - maybe update every article as it's touched?). To my way of thinking, if we had an XML backend store and a reliable conversion path, then we could: a) Provide wikisyntax editing to those who want it (by filtering through the converter) b) Develop meaningful wysiwyg editing tools without having to first reimplement the wikisyntax parser in javascript and every other language we want to touch. c) Allow direct access to the XML, making all kinds of researchers happy. d) Incrementally roll out changes to bring things more in line with Semantic Web, again with conversion paths. Engineering wise, a "lossless" path to me could be developed by developing these components: 1. Wikisyntax <-> WikiXML converters. 2. WikiXML -> HTML renderer. Determing that it is working properly can be done by testing against the Wikipedia corpus. If we can go from WikiXML to Wikisyntax and back, byte-exact, we've acheived our goal. Maybe it's ok to relax that restriction (especially if we can determine in some other way the page is corrupt or invalid - or maybe we have a list of exceptions), but I think it's one that's both acheivable and reasonable. We may also want to do validation on the HTML render path; if we want to be really strict we can require that the conversion path gives identical output (perhaps sans whitespace?) to the current parser & renderer. Once we have everything in XML, there are a number of good tools and standards to enable us to be Unicode compliant, to do various kinds of conversions and updates on the XML, and otherwise process our data, so we can evolve it forward to meet our needs. In any case - if we find that having a lossless path would satisfy the constraints, then those who are interested can focus on writing a validation framework... and then they can go implement it. :) -- Ben Garney Torque Technologies Director GarageGames.Com, Inc.

Join the dark side, Jay. You know you want to. ;) On 8/17/06, Jay R. Ashworth &lt;jra(a)baylink.com&gt; wrote: Well, I wouldn't expect WT -> XML to be any slower than our current parser (we could just modify it to output the XML instead of the wikitext); actually, it would be quite a lot faster, no doubt, because all sorts of things like template substitutions wouldn't have to occur. So it would probably be slow but manageable. XML -> WT should be virtually instantaneous, as should XML -> HTML, because parse time would be one-pass at C speed (so probably a few milliseconds or less, as opposed to 800 ms for the current parser). There would of course be all sorts of weirdness in XML -> WT, so it might not be as fast as XML -> HTML, but the WMF gets a *hell* of a lot more cache misses on page views than on page edits, I would bet a substantial sum of money.

On 8/18/06, Jay R. Ashworth &lt;jra(a)baylink.com&gt; wrote: Actually, I wasn't. Interesting idea, but I don't think there's any point, since a wikiXML -> HTML parser probably wouldn't be a noticeable bottleneck and so we may as well maintain support for non-XSL-supporting clients (does your Blackberry support XSL? :) ). I hadn't actually thought of using XSL definitions at all, but in fact, that may be the obvious choice. I'll have to look at that some more . . . I never paid much attention to it before now. Those too. Those get tons of edits, but even more views.

On 8/18/06, Jay R. Ashworth &lt;jra(a)baylink.com&gt; wrote: Touché.

On Thu, Aug 17, 2006 at 09:52:39AM -0400, Simetrical wrote: Quite so. :-) Well, if you could guarantee a perfect roundtrip from XML to WT, sure, I guess. But I don't know that WT is structurally clean enough to make that workable. Hee. Yeah; probably. Cheers, -- jra -- Jay R. Ashworth jra(a)baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274 The Internet: We paved paradise, and put up a snarking lot.

On 8/17/06, Steve Bennett &lt;stevage(a)gmail.com&gt; wrote: Maybe, maybe not, but people expect that if there's a ''' open, then ''' will close it. They won't expect an intervening '' (or [[ or ] or anything else) to affect matters. Besides, what would you suggest they type, '''hi''''''''hello'''hi'''hello''''''''hi'''? God only knows what that would do. Well, I don't think it's so much a downside as something that's impossible to work into a formal grammar. I'm guessing the issue is that templates mix syntax with semantics: the semantics of the template influence the output of the parse tree. So a pass to replace all the templates has to be done before you can even start talking about a formal grammar. But it's a given, yes, as you say. A URL-like thing that was typed without any particular surrounding syntax (it gets autolinked). Similar lookahead would presumably be necessary for RFCs, ISBNs, and PMIDs (okay, that's enough to convince me to agree that they should be ditched :) ). In general, a lookahead of no more than one character is considered desirable. He *seems* to be saying that you'd have to make special rules for each allowed HTML tag, and presumably each allowed attribute and property thereof, and maybe even every combination of them (!). Would there be any advantage in leaving those out of the grammar and keeping Parser and Sanitizer separate as they are now?

On Thu, Aug 17, 2006 at 05:12:22PM +0200, Steve Bennett wrote: Parsers don't have to be single pass... and ours isn't now. Is it? It is; I believe it renders as <strong>, not <bold>. Cheers, -- jra -- Jay R. Ashworth jra(a)baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274 The Internet: We paved paradise, and put up a snarking lot.

On Thu, Aug 17, 2006 at 01:27:16PM -0400, Simetrical wrote: Aha! Got it. Excellent point. Cheers, -- jra -- Jay R. Ashworth jra(a)baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274 The Internet: We paved paradise, and put up a snarking lot.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Steve Bennett wrote: The use of bold to indicate emphasis in typographic circles is strongly discouraged. In terms of italics, they should be used for emphasis, though the <em> tags are more suited for the task. Although most major style guides (such as MLA) recommend the use of italics for book titles, it is technically improper to mark them with <i> since this precludes a simple CSS change to change the style from one format to another. Ideally, it would probably be <span class="book">. Now, if any editor actually wants to type that all out...

On 8/17/06, Edward Z. Yang &lt;edwardzyang(a)thewritingpot.com&gt; wrote: I guess the smallest we could do it would be {{t|Gone with the Wind}} - it's not inconceivable. Steve

Steve Bennett wrote: But think of the power that you can potentially bring {{u|Gone With the Wind}} - underline it {{uc|Gone With the Wind}} - Uppercase it {{r|Gone With the Wind}} - Reverse it {{sup|Gone With the Wind}} - Super script it {{r,sup|Gone With the Wind}} - Reverse and Superscript it {{s|Gone WIth teh Wind}} - Sub-Title and so on and so on. I jest of course.....maybe. r

Edward Z. Yang wrote: But look at the power of the richness of the options. Literally could reduce everything to a single syntax with a bazillion options. Of course in might start looking like lisp (bad lisp for sure), but think of the shortened learning curve. Yes I'm still jesting. I lurk in the darkness just waiting for odd things like this to indeed make a pseudo-point about how with only a slight exaggeration someone might get the idea that I "foolishly" float here. Yes it is ugly and even bad. I'm sure a better solution will be found. r

Really?!? I'm gonna hafta' dig deepa' wid this stuff :) Same way I learned just about every other tag/directive/class, etc. I read a book (or two or seven). I'm a great believer in reading then doing, though there is no experience quite like the one afforded by raw empiricalism (warts and all). There is no better teacher than making a mistake and no better student that the one that admits the error and learns from it. As Piet Hein said (and I paraphrase) "The secret to success is to err and err and err, but less and less and less." But that's just me. r

On 17/08/06, Steve Bennett &lt;stevage(a)gmail.com&gt; wrote: I think he's referring to the parser's behaviour of replacing anything that looks like a valid UR[IL] when the protocol is in a specific configuration array. Rob Church

On 18/08/06, Steve Bennett &lt;stevage(a)gmail.com&gt; wrote: It's completely configurable by the site administrator. Incidentally, adding "file://" to the array doesn't guarantee success, since many modern browsers will point-blank refuse to render such links in their default configuration. Rob Church

On 8/18/06, Steve Bennett &lt;stevage(a)gmail.com&gt; wrote: Oh, it gets better. Ok, here's a test for everyone. Without actually testing these, predict how each of the following will be parsed: *http://[[foo]].com *http://www.[[foo]].com *[[http://foo.com]] *[[the site http://foo.com is cool]] *[[http://foo.com is cool]] *[[foo http://foo.com]] *[[ http://foo.com]] *[[http://foo.com ]] The difference in the tooltips generated by the last two is particularly interesting! How did you score? Steve

On 18/08/06, Jay R. Ashworth &lt;jra(a)baylink.com&gt; wrote: Being intimate with the parser leads to bad, bad things. Post-commit, it's like waking up the morning after to find your one-nighter's led to her stealing your cash and anything else you had on you. Rob Church

On Wed, Aug 16, 2006 at 11:15:34PM -0400, Ivan Krsti?? wrote: I suspect this will be circular: How complete a spec you can produce will depend on how much you're willing to change when you find out you can't specify it completely. How much such a spec will be permitted to prescribe change will depend on how much change it needs to prescribe. I don't know how useful it will be to have wikitext specified strictly, and I don't think we'll be able to tell until we see how far off we are, and what might need to be tweaked. It wouldn't be *completely* necessary to have a flag day for all possible changes to wikitext... Cheers, -- jr 'see also [[mode bit]]' a -- Jay R. Ashworth jra(a)baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274 The Internet: We paved paradise, and put up a snarking lot.

On 8/16/06, Jay R. Ashworth &lt;jra(a)baylink.com&gt; wrote: Well, obviously the idea is to feed it to yacc or something, so as much as possible would be good. But we might be stuck with a couple of extra passes in the PHP, or manually coded into the C parser, if it turns out to really not be usefully specifiable. (But hey, C++ and Perl can get away with grammars that aren't LALR(1), why can't we?)

On Thu, Aug 17, 2006 at 08:53:34AM +0100, Neil Harris wrote: And I'm glad you did, Neil, because I don't reacall having seen it before. Sounds worth looking into. Do you have any experience working with them? Who's the Hot Guy on this topic? Cheers, -- jra -- Jay R. Ashworth jra(a)baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274 The Internet: We paved paradise, and put up a snarking lot.