Brion Vibber wrote: I think it would make a lot of sense to standardize the wiki syntax and maybe also to introduce an XML representation of Wiki (whereas this might be more difficult than it seems). It seems to me that a person who is being considered to be an authority within the Wiki World has to take the lead on this. It's silly, but it seems like people tick this way ... Cheers Michael -- Michael Wechner Wyona - Open Source Content Management - Apache Lenya http://www.wyona.com http://lenya.apache.org michael.wechner(a)wyona.com michi(a)apache.org +41 44 272 91 61

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Evan Prodromou wrote: Last I heard they gave it up and are sponsoring another, apparently identical project. I don't expect it to fare any better. - -- brion vibber (brion @ pobox.com / brion @ wikimedia.org)

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Rob Church wrote: Well I've been advocating that since the second I heard of such projects. If they start doing it, let me know. ;) - -- brion vibber (brion @ pobox.com / brion @ wikimedia.org)

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Simetrical wrote: Ok, if they *stop* doing it let me know! ;) Seriously, though, we should all get together and make a concerted effort. I would recommend making a start by looking at the XML structure used by Magnus' conversion tools; if we need to adapt that further to integrate properly or if we can just poke around with that? Having a clear data model is a definite must; making the parser for it clean and fast can follow. :) - -- brion vibber (brion @ pobox.com / brion @ wikimedia.org)

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Magnus Manske wrote: I'd recommend modeling those off of XSLT. <xsl:if> and <xsl:choose>

Can I maybe suggest please giving some examples that you encountered of the 10-15% hard category, and the 5% very hard category? I ask so that if anyone feels tempted to start on defining the behaviour, we can gently suggest doing the harder stuff *first* (with examples), thus hopefully preventing the situation where we have multiple unfinished 80%-done definitions, and no 100%-complete formal definitions. All the best, Nick.

Eric Astor wrote: Clarification: The cases I mentioned can't be handled by a context-free parser - but both of them are parseable if ''' is parsed as a TOGGLE-BOLD item, and '' is handled similarly as a TOGGLE-ITALICS. That lets you escape the context-sensitivity, and handle the rendering properly on the output end while scanning left-to-right. - Eric

Simetrical wrote: Not necessary. Much easier if you just define ''''' as a separate token. ''''' a '' b ''' => <b><i>a</i>b</b> ''''' a ''' b '' => <i><b>a</b>b</i>

On 2/11/07, Steve Bennett &lt;stevagewp(a)gmail.com&gt; wrote: Well, the ideal is to get as much of the grammar as possible formalized, not merely informally summarized. Formalization should be possible with some inspection and testing, but of course you have to have a good understanding of formal grammars (I don't, for one, at least not yet). There are a number of wiki pages scattered around Meta and MediaWiki.org containing attempts at formalizations, which of course anyone can contribute to. On 2/11/07, Steve Bennett &lt;stevagewp(a)gmail.com&gt; wrote: Major changes are probably out of the question for the foreseeable future, just because of the annoyance of transition, with months and months of people using the wrong syntax and screwing up articles. Tweaks to the behavior in corner cases are, of course, fine.

Lars Aronsson wrote: Actually, // and ** are at least as clear, and are most definitely parsable by a fixed-lookahead context-free grammar - even an unaugmented LL(k) grammar could probably handle it. <i> and <b> are unambiguous, but ugly and language-dependent. MediaWiki's current behavior "fixes" many of the issues with its ambiguous bold/italics representation with little ad-hoc DWIM-type behavior. It works, but cannot be represented by a CFG and is difficult to extend. Oh - and ***** would most likely be disambiguated to <b>*</b>. Easy to handle in a CFG with lookahead, and almost certainly what the user meant. Slight disagreement on terms: The current syntax is convertible to HTML - it is not parseable. At the least, it is not currently parsed... No internal representation is generated, and the system just makes something on the close order of 50 regex passes per page to convert it into HTML. Possible - though difficult. I'd actually welcome someone creating a program to convert MediaWiki's syntax from apostrophes to <i> and <b>, as that could technically provide a more formal specification of how the MediaWiki parser handles apostrophes. However, at the moment, the only such program we have is MediaWiki itself. - Eric

On Mon, 2007-12-02 at 11:17 -0500, Simetrical wrote: So, it's probably worth noting that the Wikiwyg library (http://www.wikiwyg.net/ ) doesn't actually parse Wikitext. What happens is that at the user's request (clicking "edit this page", for example), the HTML of the page (output by MW itself) is put into "editable mode" on the client side. This uses some fancy features built into the two main browsers to provide WYSIWYG editing of HTML content. If the user chooses to switch over to "wikitext mode", or when they choose to save, then a process walks the DOM tree of the HTML content and outputs wikitext. This is either shown in the "wikitext mode" window or submit back to the server. (NB: this is a very tricky process that works fine for simple markup, sections, tables, etc. and is extremely difficult for things like magic words, transclusion, category listings, etc. It can be done, but we'll probably need to make some changes to our parser output to e.g. "tag" a <span> of text as coming from a particular magic word.) If the user switches from wikitext mode to WYSIWYG edit mode (or preview mode, or even "raw HTML editing" mode), the library uses Ajax to submit the wikitext it has back to a special page on the server to be rendered, and then shows the results of that rendering. If you think about it, it's really a clever system. MediaWiki is excellent at converting wikitext to HTML, and you really shouldn't have any other software do that job. And the client nowadays is really good at parsing HTML into a DOM tree, so it should take on the job of the HTML -> Wikitext conversion. The actual conversions only happen at times that the user has asked for a change of mode or submit for a save, so the delay isn't unexpected or frustrating for the user. ~Evan ________________________________________________________________________ Evan Prodromou &lt;evan(a)prodromou.name&gt; http://evan.prodromou.name/

I can't answer that question for a current dump, but I can answer it for a dump of EN that's about 15 months old (this was done as part of http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Wiki_Syntax ). The formats and figures are shown below, and I've added examples to show a single line that would cause it to be logged (assuming the rest of the wikitext in that article is well-formed). Basically they're all about _balance_ - if you open a bit of paired syntax, you should close it. Some syntaxes must be closed on the same line (e.g. ''' ), and some must be closed in the same article (e.g. {| ). Note however that these figures are from only a few months after a previous run (I think - it *was* a while ago), so the figures for now I'm guestimating would be between 2 and 4 times higher - because it's been so long since it was last done, and because there are probably more contributions now, and "wikitext format errors introduced" is probably directly proportional to the number of contributions. ---------------------------------------------- mysql> select format, count(*) as count from malformed_page group by format order by count desc; +-------------------------+-------+ | format | count | +-------------------------+-------+ | '' | 7161 | example: this is a ''test | ''' | 1248 | example: this is a '''test | '' and ''' | 1155 | example: this is a '''test'' | ''' and '' | 1091 | example: this is a ''test''' | ] | 587 | example: this is a] test | [ and ]] | 507 | example: this is a [test]] | ]] | 417 | example: this is a test]] | [[ | 413 | example: this is a [[test | [ | 372 | example: this is [a test | [[ and ] | 347 | example: this is a [[test] | {| | 261 | example: {| (and never close it) | |} | 238 | example: |} (and never open it) | --> | 67 | example: <!-- blah --> --> | <div> | 60 | example: <div> <div> blah </div> | {{ | 46 | example: {{ {{delete}} | <!-- | 43 | example: <!-- <!-- blah --> | </div> | 39 | example: <div> blah </div> </div> | }} | 34 | example: {{delete}} }} | ]] and [[ | 24 | example: this is a ]]test[[ | == and === | 20 | example: ==heading=== | ] and [ | 14 | example: this is a ]test[ | [[image: | 11 | example: [[image: [[image:test.gif]] | === and == | 8 | example: ===heading== | '' and [[ | 5 | example: this ''is a [[test | [ and '' | 5 | example: this [is a ''test | '' and ]] | 5 | example: this ''is a]] test | <code> | 5 | example: <code> <code> for i=1 </code> | </pre> | 4 | example: <pre> for i=1 </pre> </pre> | </nowiki> | 4 | example: <nowiki> for i=1 </nowiki> </nowiki> | '' and ] | 3 | etc .... | ]] and ''' | 3 | | ]] and '' | 3 | | ] and '' | 2 | | </math> | 2 | | [[ and '' | 2 | | '' and [[ and ] | 2 | | </code> | 2 | | === and ==== | 2 | | '' and ''' and ] | 2 | | [ and ''' | 1 | | ]] and '' and ''' | 1 | | [[ and ] and [ | 1 | | [ and ]] and '' | 1 | | ''' and [[ | 1 | | <math> | 1 | | ==== and === | 1 | | ]] and [[ and '' | 1 | | ] and [ and '' | 1 | | ''' and '' and ]] | 1 | | ''' and '' and [[ and [ | 1 | | ''' and ]] | 1 | | ]] and ] and ''' and '' | 1 | | [ and '' and ''' | 1 | +-------------------------+-------+ 53 rows in set (0.57 sec) mysql> ---------------------------------------------- Note: ''''' was treated as ''' + '' (rather than as a separate category), so it will be mixed in with the above figures for ''' and ''. Not that I'm aware of - humans were better at it anyway, because some of the above were false positives (e.g. some math formulas), and the ''' and '' & ''' and '' tests had lots of false positives. If a robot went around blindly automatically fixing these, it'd be banned for vandalism. However, some automated approach could be good, as it's an ongoing problem with no real closure (like sorting the mail), so people eventually have enough of doing it (as I did), and move onto other stuff. All the best, Nick.

To really give you the creeps, 1<b>2''3</b>4''5 is rendered as 1<b>2<i>3</i>4</b>5 which is just wrong, and IMHO /should/ be rendered as 1<b>2<i>3</i></b><i>4</i>5 But don't worry, my wiki2xml tool doesn't do it correctly either, though it does generate valid XML without tidyhtml... Magnus

Gerard Meijssen wrote: I was wondering... would it be possible for nap: users to enter "smart-quotes", namely &rsquo;&rsquo; or ’’? The advantage of this character over &apos; is that you can represent it in plain text (no HTML code), even in Category tags and page titles. It also looks nicer if you have a good font. You could, of course, add ’ to a <charinsert> tag in [[MediaWiki:Edittools]] to make this character easier to insert. -- Minh Nguyen &lt;mxn(a)zoomtown.com&gt; [[en:User:Mxn]] [[vi:User:Mxn]] [[m:User:Mxn]] AIM: trycom2000; Jabber: mxn(a)myjabber.net; Blog: http://mxn.f2o.org/

Gerard Meijssen wrote: What about rendering ' key on nap ’ ? Would this conflict with anything? Bold and italic would be done via <b> <i>, and being a different character, wouldn't break other wikis references (e.g. interwikis).

Jared Williams wrote: Keeping an abstract tree as an intermediate representation helps, but does not fix, this problem. Dealing with things like '''italics'' is non-trivial in any case, as if we're going to retain this behavior, no context-free grammar (at least with fixed lookahead) can possibly suffice. Whatever happens to handle this, it will have to be at a separate stage from the original parsing. What remains is a question of how many extra stages we will need. - Eric

On Sun, Feb 11, 2007 at 10:02:33PM -0500, Eric Astor wrote: I know no one ever likes this question, but I'm going to ask it again anyway: Is that problem easier or harder to deal with than whatever problems you would have if you just redefined bold to *this* and italics to _that_? Everyone keeps saying that causes horrible collisions, but *I* don't think that most of them are that difficult to disambig. Cheers -- jra -- Jay R. Ashworth jra(a)baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274

On 12/02/07, Simetrical &lt;Simetrical+wikilist(a)gmail.com&gt; wrote: It's bloody criminal, isn't it? Rob Church

Why not add *this* and _this_ to the "Extended MediaWiki" markup, then just make vanilla MW syntax one of the many light-markups supported? So when you author an article, you can pick from MediaWiki, Extended MediaWiki, Markdown, Textile, APT, Usenet, Plain Text, XHTML, etc? I'm only partly kidding. It makes sense for Wikipedia to have exactly one supported syntax, but for other wikis, having the flexibility to pick a markup would alleviate these kinds of disputes. Perhaps putting in enough hooks to allow extension devs to make their own parser grammars? (Not sure if there are enough hooks for this now btw). More seriously though, I think adding any new magic to the MediaWiki parser is just fueling the fire surrounding the parsability of the syntax - which I am against. Also, regarding parsability, has anyone attempted to write Doxia[1] plugins for MW syntax? I'm considering giving it a whirl if there isn't any prior art. [1] http://maven.apache.org/doxia/ On 2/12/07, Jay R. Ashworth &lt;jra(a)baylink.com&gt; wrote:

Jay R. Ashworth wrote: Not neccesary: We have a parser class. Make it independent of the rest of the code. Then you can make 'alternative parsers'. I volunteer for making the 'plain text' one ;)

On 2/12/07, Rob Church &lt;robchur(a)gmail.com&gt; wrote: How much of this task is possible without looking at the code base? How much can be done merely by inspection? Would it be possible for someone (Brion, yourself?) to structure this apparently very large task in such a way that non-developers like me would be able to contribute bite-size pieces of work? Such a task might look something like "Define the grammar for the [[Category:...]] statement". Do we have anything like that? Steve