Thank you for bringing those issues to the public discussion, Tim, they are really worth it. On Tue, May 1, 2012 at 11:15 AM, Tim Starling &lt;tstarling(a)wikimedia.org&gt; wrote: I like this part. Also, I really enjoy the idea of making a separate parser frame for script instead of running it in the parent template's frame. I am a bit leery though about the part where you suggest that name-value arguments ({{#invoke:module|func|param=value}}) should be parsed by engine, not the script. Don't you have to expand those arguments in order to parse them, hence making any form of lazy-expanding impossible? This is the part which I strongly oppose. Providing direct preprocessor access to Lua scripts is a bad idea. There are two key reasons for this: 1. Preprocessor is slow. 2. You would have to work out many very subtle issues with time out and nested Lua scripts. This includes timeout subtleties caused by the preprocessor slowness (load a slow template, and given the small Lua time limit, it will cause PHP to show a fatal error due to emergency timeout; even if you fix it, the standalone version uses ulimit, and it may be more difficult to fix). Now, let me go through your suggested use cases and propose some alternatives: 1. As an alternative to a string literal, to include snippets of wikitext which are intended to be editable by people who don't know Lua. I think it would be in fact better if you provided an interface for getting unprocessed wikitext. Or a preprocessor DOM. Preprocessed text makes it is difficult to combine human-readable and machine-readable versions. 2. During migration, to call complex metatemplates which have not yet been ported to Lua, or to test migrated components independently instead of migrating all at once. That would eventually lead them to becoming permanent. Bugzilla quips, an authoritative reference on Wikimedia practices, says that "temporary solutions have a terrible habit of becoming permanent, around here". Hence I would suggest that we avoid the temptation in first place. 3. To provide access to miscellaneous parser functions and variables. Now, this is a really bad idea. It is like making a scary hack an official way to do things. It actually defies the first design principle you state. preprocess( "{{FULLPAGENAME}}" ) is not only much more uglier than using appropriate API like mw.page.name(), it is also a one of the slowest ways to do this. I have benchmarked it, and it is actually ~450 times slower than accessing the title object directly. Lua was (and is) meant to improve the readability of templates, not to clutter them with stuff like articlesNum = tonumber( preprocess( "{{NUMBEROFARTICLES:R}}" ) ). Solution: proper API would do the job (actually I am currently working on it). 4. To allow Lua to construct tag invocations, such as <ref> and <gallery>. We could make a #tag-like function to do this, just as we do with parser functions. I feel myself much more comfortable with the original return {expand = true} idea, which causes the wikitext to be expanded in the new Scribunto call frame. Thank you for writing such a detail description. I am a bit puzzled about the "always use named arguments scheme" part, because it is not how the standard Lua library works. I guess that's all my concerns for now. Thanks, Victor.

On 02/05/12 03:24, Victor Vasiliev wrote: No, you don't have to expand the arguments in order to extract equals signs for name/value pairs. The equals signs are already identified by the preprocessor's parser, for the purposes of lazy expansion of template arguments. See PPFrame::newChild() and the implementation of the #switch parser function. [...] We can limit the input size, or temporarily reduce the general parser limits like post-expand include size and node count. We can also hook into PPFrame::expand() to periodically check for a Lua timeout, if that is necessary. The preprocessor is slow now, it won't become slower by allowing Lua to call it. The scenario you give in brackets will not happen. If a Lua timeout occurs when the parser is executing, the Lua script will terminate when the parser returns control to it. The timeout is not missed. It doesn't matter if there are several levels of parser/Lua recursion when a timeout occurs. LuaSandbox is able to unwind the stack efficiently. The emergency timeout mechanism is functionally equivalent to PHP's request timeout, so the emergency timeout can probably just be infinite, and we can rely on the request timeout to terminate long-running parse requests, as we do now. We could have a Lua script time limit of a few seconds, and a request timeout of 3 minutes. Maybe you are thinking of some sort of virtual wikidata system involving extracting little snippets of text from infobox invocations or something. I am not. I would rather use the real wikidata for that. I am talking about including large, wikitext-formatted chunks of content language. I don't think it's morally wrong to provide a migration tool. Migration will be a huge task, and will continue for years. People who migrate metatemplates to Lua will need lots of tools. We can provide an API for such things at some point in the future. I am not very keen on just merging whatever interface you are privately working on, without any public review. I am publishing my proposed interface before I write the code for it, so that I can respond to the comments on it without appearing to be too invested in any given solution. I wish that you would occasionally do the same. Rewriting code that you've spent many hours on can be emotionally difficult. Perhaps that's why you've made no more changes to ustring.c despite the problems with its interface. That would lead to double-expansion in cases where text derived from input arguments need to be concatenated with wikitext to be expanded. Consider: return { expand = true, text = formatHeader( frame.args.gallery_header ) .. '\n' .. '<gallery>' .. images .. '</gallery>' } It gives flexibility for future development. That was not a core principle driving the design of the standard Lua library. -- Tim Starling

On 02/05/12 04:51, Gabriel Wicke wrote: I can see how that would make sense when you're writing a parser, but given the target audience for the Lua API, I think I would prefer to provide an abbreviated interface. How about frame.args.name as an abbreviation for frame:getArgument('name'):expandTo( 'text/x-mediawiki' ) ? And how about frame.plainArgs.name as an abbreviation for frame:getArgument('name'):expandTo( 'text/plain' ) ? I know you're not really asking for a review of Parsoid and its interfaces, but I worry as to whether your use of text/plain to indicate wikitext with comments stripped is appropriate. In MediaWiki, PPFrame::expand() takes any combination of 5 boolean flags, and modifies its behaviour based on which of the Parser's 4 output types is selected, so if you were to match it for flexibility, you would need 128 MIME types. If I were to provide Lua with a richer interface to PPFrame::expand(), I would be inclined to support at least some of those flags via named options, rather than rolling them up into a single string parameter. So instead of expandTo( 'text/plain' ) we might have: frame:getArgument('name'):expand{ expand_args = false, expand_templates = false, respect_noinclude = false, strip_comments = true } Or, if forwards-compatibility requires that we don't support so many orthogonal options, some of the options could be rolled in together. The preceding could perhaps be written as: frame:getArgument('name'):expand{ plain = true } That doesn't preclude the use of overrides: frame:getArgument('name'):expand{ plain = true, strip_comments = false } But it does seem like a can of worms. How about providing getArgument(), which will return an opaque ParserValue object with a single method called expand(). This method would theoretically take named parameters, but currently, none are defined. With no parameters, it provides some kind of reasonable template-expanding behaviour. Then frame.args would provide an abbreviated syntax for expand() with no parameters. If there is a compelling use case for "plain" expansion, then we would have to decide what options to PPFrame::expand() are needed to support that use case, and then we would need to decide how to map them to parameters to ParserValue.expand(). Yes, this is an interesting idea. But I think I would prefer the factory to be a frame method rather than a global function. Also, again, I am skeptical about the value of using a MIME type. How about an interface allowing either: frame:newParserValue( 'value text' ) or named arguments: frame:newParserValue{ text = 'value text', fruitiness = 'high', } Yes, this is attractive, and could be done in the same way as ParserValue objects above. But I think there is still a need for an abbreviated interface: frame:newTemplateParserValue{title = 'tpl', args = args}:expand() abbreviated to: frame:expandTemplate{title = 'tpl', args = args} It doesn't just make the text shorter, it also reduces the number of concepts that the user has to understand before they are able to use the interface. I know that adding such concepts gives greater flexibility, but an increase in the number of concepts will steepen the learning curve, and the terminology required to explain them risks being daunting. For example, if someone has never programmed before, you can't expect them to understand terms like "opaque object". For me, the main motivation behind providing a parallel "advanced interface" along the lines you suggest would be to establish a direction for future interface development. Interfaces evolve mostly by analogy, so providing a well thought-out "advanced interface" will influence future development even if nobody ever uses it. -- Tim Starling

Is it possible to hook Lua function calls? If so, I'd make a template expansion a "call" to a function with that name. That was the interface I envisioned when thinking how I'd do it if making the language from scratch to suit wikitext (I drafted some code, but didn't reach to a barely mature level).

On 05/02/2012 04:01 AM, Tim Starling wrote: Yep, that looks good to me. Maybe the specialized wikitext argument variant could be called 'wikitextArgs' so that the general variant can be used for ParserValues instead of the getArgument method? Adding more xxxArgs methods does not seem to scale that well, and would introduce a lot of extra method names to remember. It would also encourage users to pick one representation when they would not need to care about it, especially if they just pass through some content. I am not that happy with the text/plain bit too. text/x-mediawiki with a separate progress or 'processing stage' component might be better, as it would not conflate processing stage with the format. I am using a numerical 'rank' value to track progress internally in Parsoid, but a string would likely be user-friendlier for an external API like this. There is an example for this further down in this post. Some grepping through core and extension code left me with the impression that there are relatively few common sets of flags used. As an example, NO_ARGS and NO_TEMPLATES always seem to be used as a pair in situations where just comment and noinclude (and company) handling is needed. If there remain use cases for fully orthogonal flags, then those could still be supported with optional (named) argument of course, as you note. Yes, this looks very good to me. ParseValue with a heavily defaulted expand() method should be a good compromise between convenience for the currently common case without compromising the ability to work with other content types, or specify the processing stage through a name or flags. So the 'plain' example could look somewhat like this: arg:expand( format = 'tokens/x-mediawiki', phase = '0.1_noComments' ) --- named processing phase I'd be happy with that too. Custom child frames could still be created using a frame:newChild method. +1, but I think some way to indicate the type and processing state of the passed-in value will be needed if non-wikitext values are to be supported. Further processing of a ParserValue depends on this knowledge. Optional (named) arguments could be employed for this too: frame:newParserValue{ type = 'tokens/x-mediawiki', value = { { type = 'tag', name = 'a', attribs = { href = 'http://foo' } }, "Some link text", { type = 'endtag', name = 'a' } } } Defaulting to wikitext and fully-preprocessed text for the processing phase would be fine with me. Any kind of type identifiers (MIME or not) could of course be used. MIME has the advantage of being somewhat known already, but other type identifiers might have other advantages. +1 I pretty much agree. The abbreviated interface adds some complication by providing a second way to do the same thing. This still seems to be worth it if it helps people to get started. Gabriel