2008/7/14 Tomaž Šolc <tomaz.solc(a)zemanta.com>om>: I suspect it's highly unlikely that we'll ever have a situation where any wikitext will come up with "SYNTAX ERROR" or equivalent. (Some templates on en:wp do something like this for bad parameters, but they try to make the problem reasonably obvious to fix.) Basically, the stuff's gotta work for someone who can't work a computer or think in terms of this actually being a computer language rather than text with markup. I would *guess* that an acceptable failure mode would be just to render the text unprocessed. The thing to do with particularly problematic "bad" constructs would be to go through the wikitext corpus and see how often they're actually used and how fixable they are. Remember also third-party users of MediaWiki, who may expect a given bug effect to work as a feature. - d. - d.

Don't worry Mingli, It is almost certainly impossible to parse wikitext "one time" - it's too beautifully complex for that. In another 8 to 10 months, someone will try again, there will be a big flareup of activity regarding a standardized, formalized, perfectly context-free mediawiki grammar and subsequent language-agnostic parser. At the end of that struggle and strife, we'll be back here where we started. I'm not being cynical here (nor am I trying to prematurely instigate another flamewar) - it's just the nature of the problem. A lot of really bright minds have attempted to fit wikitext into a traditional grammar mold. The problem is that it's not a traditional grammar. My recommendation is to address the actual reason why someone might want a context-free grammar in the first place. Considering how much time and creative energy has been spent on trying to create the one-true-parser, I wonder whether it would be easier to simply port the existing Parser to other languages directly (regular expressions and all). I bet it would be. -- Jim R. Wilson (jimbojw) On Mon, Jul 14, 2008 at 9:10 AM, mingli yuan <mingli.yuan(a)gmail.com> wrote: > Thansk, Tomaž and David. > > > Wikimedia projects have been accumulated so huge a repository of knowledge. > And these knowledge should be used in a wider circumstances. Could you > imagine that wikipedia articles was always bounded with a php regexp parser? > Then any formal description of the wikitext is welcomed. We should free the > knowledge from its format. > > Thanks again. > > Regards, > Mingli > > On Mon, Jul 14, 2008 at 9:01 PM, David Gerard <dgerard(a)gmail.com> wrote: >> >> 2008/7/14 Tomaž Šolc <tomaz.solc(a)zemanta.com>om>: >> >> > - From my observations I believe that the only possible way that any >> > formal grammar will replace the current PHP parser is if the MediaWiki >> > team is prepared to change the current philosophy of desperately trying >> > to make sense of any kind of broken string of characters the user >> > provides i.e. if MediaWiki could throw up a syntax error on invalid >> > input and/or they significantly reduce the number of valid constructs >> > (all horrible combinations of bold/italics markup come to mind) >> > Given my understanding of the project I find this extremely unlikely. >> > But then I'm not a MediaWiki developer, so I might be completely wrong >> > here. >> >> >> I suspect it's highly unlikely that we'll ever have a situation where >> any wikitext will come up with "SYNTAX ERROR" or equivalent. (Some >> templates on en:wp do something like this for bad parameters, but they >> try to make the problem reasonably obvious to fix.) Basically, the >> stuff's gotta work for someone who can't work a computer or think in >> terms of this actually being a computer language rather than text with >> markup. I would *guess* that an acceptable failure mode would be just >> to render the text unprocessed. >> >> The thing to do with particularly problematic "bad" constructs would >> be to go through the wikitext corpus and see how often they're >> actually used and how fixable they are. >> >> Remember also third-party users of MediaWiki, who may expect a given >> bug effect to work as a feature. >> >> >> - d. >> >> >> >> - d. >> _______________________________________________ >> Wikitext-l mailing list >> Wikitext-l(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikitext-l > > > _______________________________________________ > Wikitext-l mailing list > Wikitext-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitext-l > >

2008/7/14 Jim R. Wilson <wilson.jim.r(a)gmail.com>om>: The question is how to implement such highly desirable features as a good WYSIWYG editor without knowing what each of those bits of syntax actually mean. A parseable grammar would make it just that bit closer. A grammar with a small amount of context might be workable (to minimise the recalculation needed at each keystroke). Etc. The other use case is a 100% (once the really stupidly unnecessary emergent effects are ignored) provably correct reimplementation in another language, e.g. an optional C-based parser, a Java parser, etc., etc. - d.

Hi, Jim and Jay. Thanks for your reply. I had ported part of the existing parser into Java years ago. I still remembered the hard time, it is hard to understand the existing code, and the result regexp-based parser is buggy. I think at least two things are needed: * an abstract description of what the parser do. the rules should be clear. and it should be independent with the language implemented. maybe regexp based, or context-free grammar plus something more, or something totally different. * a test suit. I know we already have one in svn. and this test suit is independent with the programming language. that is a good start. On the other hand, I see other endeavor to standardize wiki markup outside the WikiMedia community. I mean wikiCreole in WikiSym2006 and WikiModel from sourceforge. After taking a quick look, I just want to know what make wikiCreole different? Is it an endeavor to standardize wiki markup, or it is simply just another wiki markup? On Tue, Jul 15, 2008 at 2:40 AM, Jay R. Ashworth <jra(a)baylink.com> wrote:

Hi My experience is that is was easier to learn how the parser behaves from studying example outputs than to deduce that from its source code. In the ideal case, a clean parser implementation (in any language) would be almost as good as a formal definition of the syntax. That's basically the reason I see for all of us trying to come up with a context-free grammat - it gives you a parser that is easy to understand and easy to port to other platforms. Now the current MediaWiki parser is all but clean. It's not a simple class with well defined interfaces that you can stick into another PHP program. It also doesn't generate a clean parse tree - it mangles strings until it arrives at something HTML-like and then cleans it up. Since the syntax includes all sorts of ways a page can interact with data outside of the current the interface of such a pluggable class would probably be pretty complex. Maybe one way of making some progress is to decide on this interface and push the existing parser towards it? By the way, are any of you attending Wikimania? I would love to participate in any discussion on this topic. Best regards Tomaž