On Wed, Dec 14, 2011 at 9:00 PM, Tim Starling tstarling@wikimedia.org wrote:
We still want to do something about parser performance in the first half of 2012, so we're going to bring forward our other performance project, i.e. server-side scripting embedded in wikitext. That's a project which is still at an early stage of planning. We will need to define its scope, and to bite the bullet and make some tough design choices (such as Lua versus JavaScript), if it's going to progress from pipe dream to reality.
Let's resolve to have all of the big pieces nailed down by January 31. In particular, it should be possible to make the WikiScript vs Lua vs Javascript decision well before that time.
There already seems to be a credible starting point for Lua and WikiScript, but not yet one for Javascript or any other language. Is there anyone willing to put together a Javascript proof-of-concept?
Rob
Rob Lanphier <robla <at> wikimedia.org> writes:
Tim Starling <tstarling <at> wikimedia.org> wrote:
We still want to do something about parser performance in the first half of 2012, so we're going to bring forward our other performance project, i.e. server-side scripting embedded in wikitext. That's a project which is still at an early stage of planning. We will need to define its scope, and to bite the bullet and make some tough design choices (such as Lua versus JavaScript), if it's going to progress from pipe dream to reality.
Let's resolve to have all of the big pieces nailed down by January 31. In particular, it should be possible to make the WikiScript vs Lua vs Javascript decision well before that time.
There already seems to be a credible starting point for Lua and WikiScript, but not yet one for Javascript or any other language. Is there anyone willing to put together a Javascript proof-of-concept?
Rob
Is someone picking up this gauntlet? :)
I am interested in contributing some time to this because I'm a relatively new mediawiki developer and at first glance, templates are just... repellant. I already spent an afternoon setting up the Lua extension on my dev wiki and writing an Infobox template. If you look at some of the templates that our users at Wikia build ( a good example is this 217K monster Infobox at Fallout wiki)
http://fallout.wikia.com/index.php?title=Template:Infobox&action=raw
It's clear that this is a LOT of work to just output a <TABLE>. However, writing raw Lua code to output HTML is not much fun without a good template library. Here is a simple example:
http://owen.wikia-dev.com/wiki/InfoboxLua?action=raw
It is littered with embedded HTML and string.format statements. Ugh. I'm going to look at something simple like Moustache (which has both JS and Lua implementations already) as a proof of concept. Does anybody have a better suggestion? I think Lua (or JS) by itself is not enough, there has to be a nice library of utility functions. Has anyone thought about what that will really look like?
I do have some spare time over the holiday and I can take a shot at hacking together a node.js equivalent of the Lua extension. Is that really the highest priority though? I'm willing to do it, but it seems like there are also a lot of other questions to be answered besides just the languagechoice.
Owen@wikia
On 12/21/11 11:18 AM, Owen Davis wrote:
I'm going to look at something simple like Moustache (which has both JS and Lua implementations already) as a proof of concept.
I really like the idea of using JavaScript+Mustache+Node.js. Plus there's a Mustache compiler for Node: https://github.com/raycmorgan/Mu, which should give us a little performance boost.
I don't have anything particular against Lua and no particular love for JavaScript, but JS seems more web-native; plus code and skills developed can accrue to both client- and server-side of the web.
Pete (in crunch mode at work, so no time to code up proofs-of-concept; mea culpa)
----- Original Message -----
From: "Peter Kaminski" kaminski@istori.com
I don't have anything particular against Lua and no particular love for JavaScript, but JS seems more web-native; plus code and skills developed can accrue to both client- and server-side of the web.
Just so I'm clear: this is a server-side question?
Cause if it's client side, "JS is already in the browser" seems pretty compelling to me...
Cheers, -- jra
Yep, this is for server side code, giving users a better template programming language.
On Dec 22, 2011, at 7:17 AM, Jay Ashworth wrote:
----- Original Message -----
From: "Peter Kaminski" kaminski@istori.com
I don't have anything particular against Lua and no particular love for JavaScript, but JS seems more web-native; plus code and skills developed can accrue to both client- and server-side of the web.
Just so I'm clear: this is a server-side question?
Cause if it's client side, "JS is already in the browser" seems pretty compelling to me...
Cheers,
-- jra
Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
* Owen Davis owen@wikia-inc.com [Wed, 21 Dec 2011 19:18:46 +0000 (UTC)]:
It is littered with embedded HTML and string.format statements. Ugh. I'm going to look at something simple like Moustache (which has both JS and Lua implementations already) as a proof of concept. Does anybody have a better suggestion? I think Lua (or JS) by itself is
not
enough, there has to be a nice library of utility functions. Has
anyone
thought about what that will really look like?
My guess that scripts interpreter should have MediaWiki API bindings: http://www.mediawiki.org/wiki/Api and Wiki DOM bindings, either something like already existing extension: http://www.mediawiki.org/wiki/Extension:Wiki_Object_Model or, even better WikiDOM manipulation: http://www.mediawiki.org/wiki/Visual_editor/WikiDom_Specification when the PEG Parser will be stable enough.
Building an API wrapper around Lua http client probably is not enough, because local API should be performed through FauxRequest (but still no POST emulation?). I don't know how PHP and Lua can bind to each other. I know a local pro in Lua working in local uni CS lab, however he does not work with MediaWiki. Lua is used for macros and automation in C / C++ projects. Dmitriy
* Owen Davis owen@wikia-inc.com [Wed, 21 Dec 2011 19:18:46 +0000 (UTC)]:
It is littered with embedded HTML and string.format statements. Ugh. I'm going to look at something simple like Moustache (which has both JS and Lua implementations already) as a proof of concept. Does anybody have a better suggestion? I think Lua (or JS) by itself is
not
enough, there has to be a nice library of utility functions. Has
anyone
thought about what that will really look like?
Also it would be great if Lua binding could perform parser functions. I believe that Extension:WikiScripts uses parser frame and can use current template parameters, when available: http://www.mediawiki.org/wiki/Extension:WikiScripts Template library (tpl_) allows the script to access the parser and the parameters of the template that invokes the script.
tpl_arg( argname[, default] ) returns the argument which name is specified as an argument, or the default value if it is not set (if no default is specified, it returns false). tpl_named_args() returns all the named arguments to the template. tpl_numbered_args() returns all the numbered arguments. tpl_is_transcluded() returns whether the code is invoked from a template.
Dmitriy
I think that would be a nice feature to have. Amazingly enough, WikiScripts is not of the extensions that we have in our codebase at wikia (I think we're over 800+), but I will take a look at the implementation. The <lua> parser hook allows for parameters, but that's not the same. Having access to normal template parameters would allow a lua template to act as a drop in replacement for an existing template, which is essential.
We are in a code freeze period here @ wikia, but when we return from the holidays in January I can get the Lua extension enabled on a test wiki (lua.wikia.com) so that other people can experiment with this.
On Dec 21, 2011, at 9:29 PM, Dmitriy Sintsov wrote:
- Owen Davis owen@wikia-inc.com [Wed, 21 Dec 2011 19:18:46 +0000
(UTC)]:
It is littered with embedded HTML and string.format statements. Ugh. I'm going to look at something simple like Moustache (which has both JS and Lua implementations already) as a proof of concept. Does anybody have a better suggestion? I think Lua (or JS) by itself is
not
enough, there has to be a nice library of utility functions. Has
anyone
thought about what that will really look like?
Also it would be great if Lua binding could perform parser functions. I believe that Extension:WikiScripts uses parser frame and can use current template parameters, when available: http://www.mediawiki.org/wiki/Extension:WikiScripts Template library (tpl_) allows the script to access the parser and the parameters of the template that invokes the script.
tpl_arg( argname[, default] ) returns the argument which name is specified as an argument, or the default value if it is not set (if no default is specified, it returns false). tpl_named_args() returns all the named arguments to the template. tpl_numbered_args() returns all the numbered arguments. tpl_is_transcluded() returns whether the code is invoked from a template.
Dmitriy
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
* Owen Davis owen@wikia-inc.com [Thu, 22 Dec 2011 11:50:48 -0800]:
I think that would be a nice feature to have. Amazingly enough, WikiScripts is not of the extensions that we have in our codebase at wikia (I think we're over 800+), but I will take a look at the implementation. The <lua> parser hook allows for parameters, but
that's
not the same. Having access to normal template parameters would allow
a
lua template to act as a drop in replacement for an existing template, which is essential.
WikiScripts parses and executes it's scripts via PHP, which has both advantages and disadvantages. Lua extension should run faster, however it is not suitable for every hosting out there, because PHP Lua module is uncommon. I wonder whether it's possible to run around without PHP Lua extension, perhaps to daemonize Lua then exchange data between apache/mod-php and lua processes via named pipes? It is really sad that PHP itself does not have stable "VM" allowing to execute safe subset of it's own code in server-side scripts (it had only some beta or alpha status module). I wonder whether it is possible to cross-translate the subset of WikiScripts or Lua into PHP source, then PHP eval it (however would that work with HipHop?).
Also it would be great if WikiScripts or Lua extension allowed to easily bind functions / class wrappers for another MediaWiki extensions. The extension I develop uses PHP eval for interpretation of the scripts, I am looking for adaption of WikiScripts to use it in conjunction with my extension. Dmitriy
On Fri, Dec 23, 2011 at 9:26 AM, Dmitriy Sintsov questpc@rambler.ru wrote:
Also it would be great if WikiScripts or Lua extension allowed to easily bind functions / class wrappers for another MediaWiki extensions. The extension I develop uses PHP eval for interpretation of the scripts, I am looking for adaption of WikiScripts to use it in conjunction with my extension. Dmitriy
Well, WikiScripts, if we decide to develop it seriously, will certainly have extension interface (it is on my large TODO list for that). Something tells me it would be possible to do in Lua as well. All we have to do now is to settle down on what way we choose.
--vvv
* Victor Vasiliev vasilvv@gmail.com [Fri, 23 Dec 2011 09:46:46 +0400]:
On Fri, Dec 23, 2011 at 9:26 AM, Dmitriy Sintsov questpc@rambler.ru wrote:
Also it would be great if WikiScripts or Lua extension allowed to
easily
bind functions / class wrappers for another MediaWiki extensions.
The
extension I develop uses PHP eval for interpretation of the scripts,
I
am looking for adaption of WikiScripts to use it in conjunction with
my
extension. Dmitriy
Well, WikiScripts, if we decide to develop it seriously, will certainly have extension interface (it is on my large TODO list for that). Something tells me it would be possible to do in Lua as well. All we have to do now is to settle down on what way we choose.
Victor, what do you think about making WikiScripts syntax more similar to the subset of Lua or JavaScript syntax? That probably should not be too hard? What's about cross-translating to PHP then eval()? Lua is great, however, it's a bit strange to use two interpreters (PHP+Lua) together. That limits hosting possibilities and it's something like using two similar screwdrivers for the same screw. It is a shame PHP itself does not have stable good VM extension for execution of scripts. Dmitriy
On Fri, Dec 23, 2011 at 9:54 AM, Dmitriy Sintsov questpc@rambler.ru wrote:
Victor, what do you think about making WikiScripts syntax more similar to the subset of Lua or JavaScript syntax? That probably should not be too hard?
It is already almost a JavaScript subset
What's about cross-translating to PHP then eval()?
Bad idea. If you really care about performance, you will be able to install native parser (already implemented, but not integrated) and that shall solve most performance problems.
Lua is great, however, it's a bit strange to use two interpreters (PHP+Lua) together. That limits hosting possibilities and it's something like using two similar screwdrivers for the same screw.
Not really. Lua was designed as a sandboxable language, and PHP was not.
--vvv
----- Original Message -----
From: "Victor Vasiliev" vasilvv@gmail.com
Lua is great, however, it's a bit strange to use two interpreters (PHP+Lua) together. That limits hosting possibilities and it's something like using two similar screwdrivers for the same screw.
Not really. Lua was designed as a sandboxable language, and PHP was not.
This is a really critical point: if you're going to provide an interpreted language to end-users from within a program that is, itself, written in an interpreted language, *you cannot use the underlying interpreter* to run the end-users' programs, unless that interpreter has sandboxing built-in.
If you try, you will almost certainly be exposing yourself to critical security vulnerabilities. You're almost *better* off picking a different language, so that you're not tempted to try.
Cheers, -- jra
On 23.12.2011 18:30, Jay Ashworth wrote:
This is a really critical point: if you're going to provide an interpreted language to end-users from within a program that is, itself, written in an interpreted language, *you cannot use the underlying interpreter* to run the end-users' programs, unless that interpreter has sandboxing built-in. If you try, you will almost certainly be exposing yourself to critical security vulnerabilities. You're almost *better* off picking a different language, so that you're not tempted to try. Cheers, -- jra
I remember that PHP had some outdated and unmaintained sandboxing PECL module, however it's unmaintained for a long time. http://php.net/manual/en/runkit.sandbox.php Dmitriy
On 23.12.2011 18:21, Dmitriy Sintsov wrote:
I remember that PHP had some outdated and unmaintained sandboxing PECL module, however it's unmaintained for a long time. http://php.net/manual/en/runkit.sandbox.php Dmitriy
Dmitry Zenovich applied for maintaining it last year, although little seems to have been done in the official repo. He seems to have been working on https://github.com/zenovich/runkit/ I worked with runkit some years ago and it wasn't hard to make it run. The downside is that you need a threaded php.
* Keisial keisial@gmail.com [Sun, 25 Dec 2011 23:16:46 +0100]:
On 23.12.2011 18:21, Dmitriy Sintsov wrote:
I remember that PHP had some outdated and unmaintained sandboxing
PECL
module, however it's unmaintained for a long time. http://php.net/manual/en/runkit.sandbox.php Dmitriy
Dmitry Zenovich applied for maintaining it last year, although little seems to have been done in the official repo. He seems to have been working on https://github.com/zenovich/runkit/ I worked with runkit some years ago and it wasn't hard to make it run. The downside is that you need a threaded php.
Wikimedia has experienced PHP/C developers who probably can maintain and update such module. Maybe even cross-translation from subset of JS into PHP (with caching) then executing via runkit could be possible. But threading is not something that is desirable in current environment. So maybe it was a bad idea. Dmitriy
I was working with Victor a bit to see how close to JS we could get WikiScripts syntax to be, and it seemed like it was possible but some things (like objects) would either be a hack to "fake" in some ways or a larger job to get working. Not sure if he's got more progress in that way to show, but subsets are good because you can always add more later. As long as we don't add support for something that's not standard javascript (super-set) we have a lot of options in the future. Web standards are important - we should strive to use them as much as possible.
- Trevor
On Sun, Dec 25, 2011 at 10:08 PM, Dmitriy Sintsov questpc@rambler.ruwrote:
- Keisial keisial@gmail.com [Sun, 25 Dec 2011 23:16:46 +0100]:
On 23.12.2011 18:21, Dmitriy Sintsov wrote:
I remember that PHP had some outdated and unmaintained sandboxing
PECL
module, however it's unmaintained for a long time. http://php.net/manual/en/runkit.sandbox.php Dmitriy
Dmitry Zenovich applied for maintaining it last year, although little seems to have been done in the official repo. He seems to have been working on https://github.com/zenovich/runkit/ I worked with runkit some years ago and it wasn't hard to make it run. The downside is that you need a threaded php.
Wikimedia has experienced PHP/C developers who probably can maintain and update such module. Maybe even cross-translation from subset of JS into PHP (with caching) then executing via runkit could be possible. But threading is not something that is desirable in current environment. So maybe it was a bad idea. Dmitriy
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
I do have some spare time over the holiday and I can take a shot at hacking together a node.js equivalent of the Lua extension. Is that really the highest priority though? I'm willing to do it, but it seems like there are also a lot of other questions to be answered besides just the languagechoice.
A few thougths on one of these questions, namely sandboxing and time limits:
Google Caja (http://code.google.com/p/google-caja/) promises to provide control about accessible objects and parts of the DOM for JavaScript. This still does not solve the issue of run time limits. WebWorkers could be employed for something like this, but have a very high overhead for short script snippets.
In general, I believe that run time limits are harder to enforce for heavily jitted runtimes. In these, context switching is often coupled to memory allocation, so a tight jitted loop without memory allocations would need to be aborted using heavier process control mechanisms. Simple interpreters (as CPython) can use bytecode op counters for this purpose. While lower in process control overhead, this solution then suffers from bytecodes with different runtime characteristics. Memory-allocating bytecodes are especially problematic in this regard.
Gabriel
wikitech-l@lists.wikimedia.org