Making the Lua/Javascript decision (Re: Performance roadmap update)

List overview All Threads
Download

newer

older

New committers

Files with unknown copyright...

Rob Lanphier

21 Dec 2011 21 Dec '11

8:33 a.m.

On Wed, Dec 14, 2011 at 9:00 PM, Tim Starling <tstarling(a)wikimedia.org> wrote:

...

We still want to do something about parser performance in the first half of 2012, so we're going to bring forward our other performance project, i.e. server-side scripting embedded in wikitext. That's a project which is still at an early stage of planning. We will need to define its scope, and to bite the bullet and make some tough design choices (such as Lua versus JavaScript), if it's going to progress from pipe dream to reality.

Let's resolve to have all of the big pieces nailed down by January 31. In particular, it should be possible to make the WikiScript vs Lua vs Javascript decision well before that time. There already seems to be a credible starting point for Lua and WikiScript, but not yet one for Javascript or any other language. Is there anyone willing to put together a Javascript proof-of-concept? Rob

Show replies by date

Owen Davis

21 Dec 21 Dec

8:18 p.m.

New subject: [Wikitech-l] Making the Lua/Javascript decision (Re: Performance roadmap update)

Rob Lanphier <robla <at> wikimedia.org> writes:

...

Tim Starling <tstarling <at> wikimedia.org> wrote:

Is someone picking up this gauntlet? :) I am interested in contributing some time to this because I'm a relatively new mediawiki developer and at first glance, templates are just... repellant. I already spent an afternoon setting up the Lua extension on my dev wiki and writing an Infobox template. If you look at some of the templates that our users at Wikia build ( a good example is this 217K monster Infobox at Fallout wiki) http://fallout.wikia.com/index.php?title=Template:Infobox&action=raw It's clear that this is a LOT of work to just output a <TABLE>. However, writing raw Lua code to output HTML is not much fun without a good template library. Here is a simple example: http://owen.wikia-dev.com/wiki/InfoboxLua?action=raw It is littered with embedded HTML and string.format statements. Ugh. I'm going to look at something simple like Moustache (which has both JS and Lua implementations already) as a proof of concept. Does anybody have a better suggestion? I think Lua (or JS) by itself is not enough, there has to be a nice library of utility functions. Has anyone thought about what that will really look like? I do have some spare time over the holiday and I can take a shot at hacking together a node.js equivalent of the Lua extension. Is that really the highest priority though? I'm willing to do it, but it seems like there are also a lot of other questions to be answered besides just the languagechoice. Owen@wikia

Peter Kaminski

9 p.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

On 12/21/11 11:18 AM, Owen Davis wrote:

...

I'm going to look at something simple like Moustache (which has both JS and Lua implementations already) as a proof of concept.

I really like the idea of using JavaScript+Mustache+Node.js. Plus there's a Mustache compiler for Node: <https://github.com/raycmorgan/Mu>, which should give us a little performance boost. I don't have anything particular against Lua and no particular love for JavaScript, but JS seems more web-native; plus code and skills developed can accrue to both client- and server-side of the web. Pete (in crunch mode at work, so no time to code up proofs-of-concept; mea culpa)

Jay Ashworth

22 Dec 22 Dec

4:17 p.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

----- Original Message -----

...

From: "Peter Kaminski" <kaminski(a)istori.com>

...

I don't have anything particular against Lua and no particular love for JavaScript, but JS seems more web-native; plus code and skills developed can accrue to both client- and server-side of the web.

Owen Davis

8:29 p.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

Yep, this is for server side code, giving users a better template programming language. On Dec 22, 2011, at 7:17 AM, Jay Ashworth wrote:

...

----- Original Message -----

From: "Peter Kaminski" <kaminski(a)istori.com>

I don't have anything particular against Lua and no particular love for JavaScript, but JS seems more web-native; plus code and skills developed can accrue to both client- and server-side of the web.

Just so I'm clear: this is a server-side question? Cause if it's client side, "JS is already in the browser" seems pretty compelling to me... Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Dmitriy Sintsov

6:24 a.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

* Owen Davis <owen(a)wikia-inc.com> [Wed, 21 Dec 2011 19:18:46 +0000 (UTC)]:

...

It is littered with embedded HTML and string.format statements. Ugh. I'm going to look at something simple like Moustache (which has both JS and Lua implementations already) as a proof of concept. Does anybody have a better suggestion? I think Lua (or JS) by itself is

not

...

enough, there has to be a nice library of utility functions. Has

anyone

...

thought about what that will really look like?

My guess that scripts interpreter should have MediaWiki API bindings: http://www.mediawiki.org/wiki/Api and Wiki DOM bindings, either something like already existing extension: http://www.mediawiki.org/wiki/Extension:Wiki_Object_Model or, even better WikiDOM manipulation: http://www.mediawiki.org/wiki/Visual_editor/WikiDom_Specification when the PEG Parser will be stable enough. Building an API wrapper around Lua http client probably is not enough, because local API should be performed through FauxRequest (but still no POST emulation?). I don't know how PHP and Lua can bind to each other. I know a local pro in Lua working in local uni CS lab, however he does not work with MediaWiki. Lua is used for macros and automation in C / C++ projects. Dmitriy

Dmitriy Sintsov

6:29 a.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

* Owen Davis <owen(a)wikia-inc.com> [Wed, 21 Dec 2011 19:18:46 +0000 (UTC)]:

...

not

...

enough, there has to be a nice library of utility functions. Has

anyone

...

thought about what that will really look like?

Also it would be great if Lua binding could perform parser functions. I believe that Extension:WikiScripts uses parser frame and can use current template parameters, when available: http://www.mediawiki.org/wiki/Extension:WikiScripts Template library (tpl_) allows the script to access the parser and the parameters of the template that invokes the script. tpl_arg( argname[, default] ) returns the argument which name is specified as an argument, or the default value if it is not set (if no default is specified, it returns false). tpl_named_args() returns all the named arguments to the template. tpl_numbered_args() returns all the numbered arguments. tpl_is_transcluded() returns whether the code is invoked from a template. Dmitriy

Owen Davis

8:50 p.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

I think that would be a nice feature to have. Amazingly enough, WikiScripts is not of the extensions that we have in our codebase at wikia (I think we're over 800+), but I will take a look at the implementation. The <lua> parser hook allows for parameters, but that's not the same. Having access to normal template parameters would allow a lua template to act as a drop in replacement for an existing template, which is essential. We are in a code freeze period here @ wikia, but when we return from the holidays in January I can get the Lua extension enabled on a test wiki (lua.wikia.com) so that other people can experiment with this. On Dec 21, 2011, at 9:29 PM, Dmitriy Sintsov wrote:

...

* Owen Davis <owen(a)wikia-inc.com> [Wed, 21 Dec 2011 19:18:46 +0000 (UTC)]:

not

enough, there has to be a nice library of utility functions. Has

anyone

thought about what that will really look like?

Dmitriy Sintsov

23 Dec 23 Dec

6:26 a.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

* Owen Davis <owen(a)wikia-inc.com> [Thu, 22 Dec 2011 11:50:48 -0800]:

...

that's

...

not the same. Having access to normal template parameters would allow

...

lua template to act as a drop in replacement for an existing template, which is essential.

WikiScripts parses and executes it's scripts via PHP, which has both advantages and disadvantages. Lua extension should run faster, however it is not suitable for every hosting out there, because PHP Lua module is uncommon. I wonder whether it's possible to run around without PHP Lua extension, perhaps to daemonize Lua then exchange data between apache/mod-php and lua processes via named pipes? It is really sad that PHP itself does not have stable "VM" allowing to execute safe subset of it's own code in server-side scripts (it had only some beta or alpha status module). I wonder whether it is possible to cross-translate the subset of WikiScripts or Lua into PHP source, then PHP eval it (however would that work with HipHop?). Also it would be great if WikiScripts or Lua extension allowed to easily bind functions / class wrappers for another MediaWiki extensions. The extension I develop uses PHP eval for interpretation of the scripts, I am looking for adaption of WikiScripts to use it in conjunction with my extension. Dmitriy

Victor Vasiliev

6:46 a.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

On Fri, Dec 23, 2011 at 9:26 AM, Dmitriy Sintsov <questpc(a)rambler.ru> wrote:

...

Also it would be great if WikiScripts or Lua extension allowed to easily bind functions / class wrappers for another MediaWiki extensions. The extension I develop uses PHP eval for interpretation of the scripts, I am looking for adaption of WikiScripts to use it in conjunction with my extension. Dmitriy

Well, WikiScripts, if we decide to develop it seriously, will certainly have extension interface (it is on my large TODO list for that). Something tells me it would be possible to do in Lua as well. All we have to do now is to settle down on what way we choose. --vvv

Dmitriy Sintsov

6:54 a.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

* Victor Vasiliev <vasilvv(a)gmail.com> [Fri, 23 Dec 2011 09:46:46 +0400]:

...

On Fri, Dec 23, 2011 at 9:26 AM, Dmitriy Sintsov <questpc(a)rambler.ru> wrote:

Also it would be great if WikiScripts or Lua extension allowed to

easily > bind functions / class wrappers for another MediaWiki extensions.

The

...

> extension I develop uses PHP eval for interpretation of the scripts,

...

am looking for adaption of WikiScripts to use it in conjunction with

extension. Dmitriy

Victor, what do you think about making WikiScripts syntax more similar to the subset of Lua or JavaScript syntax? That probably should not be too hard? What's about cross-translating to PHP then eval()? Lua is great, however, it's a bit strange to use two interpreters (PHP+Lua) together. That limits hosting possibilities and it's something like using two similar screwdrivers for the same screw. It is a shame PHP itself does not have stable good VM extension for execution of scripts. Dmitriy

Victor Vasiliev

8:05 a.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

On Fri, Dec 23, 2011 at 9:54 AM, Dmitriy Sintsov <questpc(a)rambler.ru> wrote:

...

Victor, what do you think about making WikiScripts syntax more similar to the subset of Lua or JavaScript syntax? That probably should not be too hard?

It is already almost a JavaScript subset

...

What's about cross-translating to PHP then eval()?

Bad idea. If you really care about performance, you will be able to install native parser (already implemented, but not integrated) and that shall solve most performance problems.

...

Lua is great, however, it's a bit strange to use two interpreters (PHP+Lua) together. That limits hosting possibilities and it's something like using two similar screwdrivers for the same screw.

Not really. Lua was designed as a sandboxable language, and PHP was not. --vvv

Jay Ashworth

3:30 p.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

----- Original Message -----

...

From: "Victor Vasiliev" <vasilvv(a)gmail.com>

...

> Lua is great, however, it's a bit strange to use two interpreters > (PHP+Lua) together. That limits hosting possibilities and it's > something > like using two similar screwdrivers for the same screw.

...

Not really. Lua was designed as a sandboxable language, and PHP was not.

This is a really critical point: if you're going to provide an interpreted language to end-users from within a program that is, itself, written in an interpreted language, *you cannot use the underlying interpreter* to run the end-users' programs, unless that interpreter has sandboxing built-in. If you try, you will almost certainly be exposing yourself to critical security vulnerabilities. You're almost *better* off picking a different language, so that you're not tempted to try. Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274

Dmitriy Sintsov

6:21 p.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

On 23.12.2011 18:30, Jay Ashworth wrote:

...

I remember that PHP had some outdated and unmaintained sandboxing PECL module, however it's unmaintained for a long time. http://php.net/manual/en/runkit.sandbox.php Dmitriy

Keisial

25 Dec 25 Dec

11:16 p.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

On 23.12.2011 18:21, Dmitriy Sintsov wrote:

...

I remember that PHP had some outdated and unmaintained sandboxing PECL module, however it's unmaintained for a long time. http://php.net/manual/en/runkit.sandbox.php Dmitriy

Dmitry Zenovich applied for maintaining it last year, although little seems to have been done in the official repo. He seems to have been working on https://github.com/zenovich/runkit/ I worked with runkit some years ago and it wasn't hard to make it run. The downside is that you need a threaded php.

Dmitriy Sintsov

26 Dec 26 Dec

7:08 a.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

* Keisial <keisial(a)gmail.com> [Sun, 25 Dec 2011 23:16:46 +0100]:

...

On 23.12.2011 18:21, Dmitriy Sintsov wrote: > I remember that PHP had some outdated and unmaintained sandboxing

PECL

...

module, however it's unmaintained for a long time. http://php.net/manual/en/runkit.sandbox.php Dmitriy

Trevor Parscal

9 Jan 9 Jan

7:51 p.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

I was working with Victor a bit to see how close to JS we could get WikiScripts syntax to be, and it seemed like it was possible but some things (like objects) would either be a hack to "fake" in some ways or a larger job to get working. Not sure if he's got more progress in that way to show, but subsets are good because you can always add more later. As long as we don't add support for something that's not standard javascript (super-set) we have a lot of options in the future. Web standards are important - we should strive to use them as much as possible. - Trevor On Sun, Dec 25, 2011 at 10:08 PM, Dmitriy Sintsov <questpc(a)rambler.ru>wrote;wrote:

...

* Keisial <keisial(a)gmail.com> [Sun, 25 Dec 2011 23:16:46 +0100]:

On 23.12.2011 18:21, Dmitriy Sintsov wrote: > I remember that PHP had some outdated and unmaintained sandboxing

PECL

module, however it's unmaintained for a long time. http://php.net/manual/en/runkit.sandbox.php Dmitriy

Wikimedia has experienced PHP/C developers who probably can maintain and update such module. Maybe even cross-translation from subset of JS into PHP (with caching) then executing via runkit could be possible. But threading is not something that is desirable in current environment. So maybe it was a bad idea. Dmitriy _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Gabriel Wicke

23 Dec 23 Dec

4:01 p.m.

New subject: Making the Lua/Javascript decision (Re: Performance roadmap update)

...

I do have some spare time over the holiday and I can take a shot at hacking together a node.js equivalent of the Lua extension. Is that really the highest priority though? I'm willing to do it, but it seems like there are also a lot of other questions to be answered besides just the languagechoice.

A few thougths on one of these questions, namely sandboxing and time limits: Google Caja (http://code.google.com/p/google-caja/) promises to provide control about accessible objects and parts of the DOM for JavaScript. This still does not solve the issue of run time limits. WebWorkers could be employed for something like this, but have a very high overhead for short script snippets. In general, I believe that run time limits are harder to enforce for heavily jitted runtimes. In these, context switching is often coupled to memory allocation, so a tight jitted loop without memory allocations would need to be aborted using heavier process control mechanisms. Simple interpreters (as CPython) can use bytecode op counters for this purpose. While lower in process control overhead, this solution then suffers from bytecodes with different runtime characteristics. Memory-allocating bytecodes are especially problematic in this regard. Gabriel

4490

days inactive

4509

days old

wikitech-l@lists.wikimedia.org

Manage subscription

17 comments

9 participants

tags (0)

participants (9)

Dmitriy Sintsov
Gabriel Wicke
Jay Ashworth
Keisial
Owen Davis
Peter Kaminski
Rob Lanphier
Trevor Parscal
Victor Vasiliev