Jens Frank wrote: Please send me your *entire* test string, as otherwise I can't reproduce the problem! If I type just "{{stub}}", with or without a trailing newline, I do not get an error. I get: <article><paragraph><template>stub</template></paragraph></article> Whoa! Thanks loads for this one, Segmentation faults are of course very important to catch. The error was that in parsetree.c:convertPipeSeriesToText(), I used a freeRecursively() where it should have been just free(). While I was at it, I've also performance-optimised it. :) Thanks, Timwi

Stephan Walter wrote: The page has its encoding declared as ISO-8859-1. The behaviour you described is correct in this case. Timwi

Rowan Collins wrote: Not entirely, but mostly :) Here are my thoughts: * Redirects are not passed through the lex/yacc parser at all. They can be recognised with a regular expression that takes the magic words into account. * Things like __NOTOC__ and stuff can be handled like this: * Regard *everything* of the form __CAPITALLETTERS__ as a special token * Have the post-processing step remove and process them * What to do with unrecognised ones is open to debate. Options: 1. turn them back into text 2. ignore them. They're rare enough to require <nowiki> if you want to actually write them. * The "Media", "Image" and other namespaces are handled in post-processing. The parser sees them as text within the link target. * The template pseudo-variables (e.g. CURRENTMONTH) are similarly handled in post-processing. * HTML tags and extension names are either not internationalised, or all translations of them are made to work on all Wikipedias. Timwi

On Mon, 20 Sep 2004 13:56:36 +0100, Rowan Collins &lt;rowan.collins(a)gmail.com&gt; wrote: How about just changing the first redirect to point to the page the second one is pointing to? That is change: A -> B -> C to A -> C so when people show up at a they just see "this is a redirect" rather than "this is a broken redirect" due to the software having solved that automatically.

On Mon, 20 Sep 2004 16:40:30 +0100, Timwi &lt;timwi(a)gmx.net&gt; wrote: Hm, figures! I really must start finishing what I start, rather than leaving it on a "to-do" list for six months and then discovering its redundant... Still, doesn't this mean the parser needs to recognise "#REDIRECT <linkpattern>" as a special token? And doesn't that, in turn, present a problem if we want to retain MagicWord i18n? True. So long as the ones that function now will still function, we should be fine. <snip> It's a step backwards because instead of being able to say "yes, we have an i18n system for you to customise these 'magic words' according to your language and preference", we will be saying "the parser knows a few variants for each of these 'magic words'; if you want to add any more, tell us, and we'll recompile the parser for you". In my opinion this is quite a big deal - essentially, we are dropping a feature of the software. Not that I've come up with a workable alternative yet, it just seems a shame. And no, I don't think anyone will need the Chinese translation of <math> on the English Wikipedia, and for that precise reason see no reason the English wikipedia should be parsing for it. Sorry, I'm not with you on this one - are you suggesting that the Chinese parser be specifically compiled to only cope with the Chinese magic words? If so, perhaps you misunderstood my problem with having all variants coded in: its not that they will all work everywhere, but that adding new ones would (if I understand the whole yacc concept correctly) require recompiling a new parser, rather than just tweaking the appropriate language or settings file. I agree, this would not be at all elegant. Yeah, as soon as I wrote it, I realised that it would end up rather expensive efficiency-wise. Indeed, I hadn't thought of the necessity to parse whole bunches of HTML tags relative to each other. I suppose you could have HTML tags explicitly defined, and have a not-extension that spat back things which looked like extensions but weren't; it would still mean that any disallowed HTML or HTML-like tag would add an extra call to the parser, and I'm not sure if that would be an acceptable price or not. So maybe you're right, and the only workable solution is to have all variants hard-coded in the parser. I guess this is where we come to regret adopting an "extension" syntax that matches/conflicts with the syntax used by "allowed bits of HTML". If we want to keep true customisation of magic words (where "editing the source" != "customisation") the best idea I've come up with is: 1) hard-code all allowed HTML into the parser. This means maximum efficiency for those bits, and the ability to handle relationships between them, etc. 2) treat everything else matching "<"+some_letters+">" as an "extension" and spew out its contents as one element of the parse tree. If the receiving PHP script then says "there's no such extension", it escapes the "<" and ">", and passes the contents back to be parsed normally. This would be acceptable IFF a) cases of erroneous <tags> were pretty rare, so the load created by parsing them separately was not too high; AND b) it was acceptable that erroneous tags inside complex HTML structures would break that structure - e.g. "<table><tr><foo><td>Some text</td></foo></tr></table>" would not parse correctly, because the second call to the parser would contain just "<td>Some text</td>", which would probably be impossible to parse properly in isolation. Of course, even this could be avoided if the entire text were re-parsed, with the erroneous tags escaped, but that's adding yet more overhead... -- Rowan Collins BSc [IMSoP]

On Thu, 23 Sep 2004 20:09:21 +0100, Timwi &lt;timwi(a)gmx.net&gt; wrote: But why make that a special case? Why say "before using the nice eficient real parser, use a not-a-parser to check for the #REDIRECT directive, and have it do some voodoo" Far better to just have the parser recognise "#REDIRECT" (and any variants anyone wants) and output a parse tree with a special redirect node. Well, technically, no, but Language*.php and LocalSettings.php are more like configuration files that happen to be executable for convenience. Editing the declaration of $wgMagicWordsEn in Language.php is no more difficult or involved than, say, editing a .ini file. Actually, I have to admit I had no idea how difficult it would be, but I assumed it would mean having at least a compiler, if not a compiler-compiler and a whole load of other tools. Editing PHP doesn't need that kind of thing, and the way its designed now, you needn't notice your editing code. If it were possible to only require a c compiler, it would certainly be a favour to other admins running MediaWiki. It's going to be annoying enough for some of them to have to deal with a binary part as well as PHP. It's oh so tempting to say "let's change it" but a) I'd be mobbed by everyone who voted for the current syntax (which includes myself) and b) we'd have to go through changing exisitng uses of <math>, or make it a special case, or something. Yep, I think you're probably right on that one. And as you say, the more things that can e done inside the parser, the better, since outside means PHP, and is likely to be less efficient. -- Rowan Collins BSc [IMSoP]

On Wed, 29 Sep 2004 22:36:36 +0200, Timwi &lt;timwi(a)gmx.net&gt; wrote: No probs: it's about 3 months quicker than my replies to messages from friends seem to end up being... :-/ Well, why is the new parser in general better? Because it separates out parsing into a proper parser rather than mingling at as special-case checks everywhere, and thus makes the software easier to maintain. This is parsing too, so lets put it in the parser, not in special-case checks everywhere. We wouldn't invoke the parser *just* to do that. We'll still need a parse-on-save for things like updating link tables; if it comes back saying "this is a redirect", we use that information to set the is_redirect flag. OTOH, the code *outside* the parser will have to, and more: if you want to avoid putting redirects through the parser at all, that code has to do the following: 1] spot that the page is a redirect 2] determine what it is a redirect to 3] find all the article's entries in the links tables (links, brokenlinks, categorylinks) and delete all those except the target of the redirect. 1 has to be done one way or another anyway; 2 and 3 would be natural side-effects of parsing *any* page before save (unless I've completely misunderstood the current code structure, or you have some magic way of avoiding this), so why duplicate them in special case code just for redirects? No, all output code will need a way of rendering a <redirect> element - that's no more a special case than any other element. If you had an output system that was guaranteed *never* to deal with an "&redirect=no" request, you could simply leave this output undefined; otherwise, the parser and output will have to do *something* with the result, and I've always hated the 'misinterpret it as a numbered list' approach. I imagine an outputter to something static like LaTeX would want to either: * always follow redirects to their destination (so it won't see any actual redirect pages anyway; except for double-redirects, but they're broken anyway); note that this has nothing to do with the parser whatsoever, since it is a navigation issue, and shouldn't require access to the page's content at all (currently, the page's text *is* accessed, somewhere in Title.php I believe, but it needn't be) or: * render redirects as cross-references [e.g. "Such and such: See So and so"] (in which case having the parser output some explanation that this is a redirect would be very helpful indeed). No, it's not very hard; it's just harder than not needing one. Remember that most web-hosting is accessed by FTP, not SSH; any compilation has to happen on a different system, usually a home PC; with any luck, things will end up binary compatible with the server. So: plain-text options: great; needs-a-c-compiler options: slightly awkward, but perhaps a necessary evil; needs-a-compiler-compiler options: you're no longer an administrator, you're a developer. For how long? As new features come along, what are the chances that they will be back-ported to the not-a-parser? How many versions will be released before the not-a-parser is completely incompatible with large parts of the codebase? I'm not saying this whole thing is a bad idea - I think it's extremely sensible - just that we do need to minimise the hassle for other MediaWiki admins who want to use the latest version, but don't want to play the role of developer. BTW, did anyone ever find a compiler-compiler that could output PHP? -- Rowan Collins BSc [IMSoP]

Timwi wrote: From the point of view of semantics and abstract syntax, it would make some sense for these to be represented in the AST. If it turns out to be difficult to do in lex/yacc that's fine, but perhaps a first-pass postprocessor could stick them into the tree? It'd be particularly useful when retargeting the parser to non-HTML outputs. Depending on implementation, it might even be as efficient or more efficient, since it'd avoid an "if this is really an image, render it this way" check on every link node in the AST. -Mark