Hi folks,
As you may know, WMF's Platform Engineering group plans to embark on a major performance initiative this year, and had chosen inline scripting as having the biggest potential impact given what's practical now. Tim Starling build a Lua prototype last year which showed a lot of promise for making things much faster. One major decision before embarking on this effort was a decision on whether we'd stick with Lua or try another language such as Javascript or Victor's WikiScript implmentation. I wanted to make a decision by the end of the month[1], and I think we've done it.
We've decided to build a deployable version of Lua as a new alternative to wiki markup for templates, barring some scandalous revelation about Lua's lurid past or other unforeseen barrier. Tim will be leading this effort, and will start on the implementation some time after the dust settles on the 1.19 deployment and the Git migration. The project page for this is located here: http://www.mediawiki.org/wiki/Lua_scripting
Rough notes from our meeting yesterday are also available [2]
Rob
[1] http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/57769/... [2] http://www.mediawiki.org/wiki/Lua_scripting/Meeting_2012-01-25
Doesn't seem to have arrived into the list. Resending.
On 27/01/12 16:17, Platonides wrote:
- Is there a simple Lua syntax guide?
http://www.lua.org/manual/5.2/ looks more like a grammar. Certainly not suitable for end users.
- The decision seems to have been Lua vs JS. What discarded a custom
language? (such as WikiScripts)
- How hard is it to translate Lua to PHP?
(for templates which then get converted into extensions)
Where would code live?
I think it _would_ be possible to make an extension defined with
<wikiscript> to take over the page. (avoiding the need of template wrappers)
- What's the expected way of implementing 'libraries' in the wiki?
On Sun, Jan 29, 2012 at 2:51 AM, Platonides Platonides@gmail.com wrote:
- Is there a simple Lua syntax guide?
http://www.lua.org/manual/5.2/ looks more like a grammar. Certainly not suitable for end users.
Wikipedia article has some nice examples.
- The decision seems to have been Lua vs JS. What discarded a custom
language? (such as WikiScripts)
As far as I am aware, those are mostly performance issues. Although there may be maintaining issues as well.
- How hard is it to translate Lua to PHP?
(for templates which then get converted into extensions)
About as hard as JS -> PHP (since they are very similar in many questions).
Where would code live?
I think it _would_ be possible to make an extension defined with
<wikiscript> to take over the page. (avoiding the need of template wrappers)
- What's the expected way of implementing 'libraries' in the wiki?
That's sort of unsolved question. Check out the meeting notes for some thoughts on that.
Answer to #4 is almost certainly separate pages as far as I understand. This approach allows us to have syntax highlighting, code editor and a better debugger. According to meeting notes, this may also be problematic for visual editor.
The way WikiScripts extensions did it is that there are special code pages which are called modules, and you can export stuff (functions and constants and probably classes in Lua) from modules like you do in CommonJS and call one module's methods from another's. I am now working on the Scripting extension which is a language-independent scripting module based on WikiScripts code:
http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/Scripting/
--vvv
On 29/01/12 09:51, Platonides wrote:
Doesn't seem to have arrived into the list. Resending.
On 27/01/12 16:17, Platonides wrote:
- Is there a simple Lua syntax guide?
http://www.lua.org/manual/5.2/ looks more like a grammar. Certainly not suitable for end users.
Yes, the first edition of Programming in Lua is available online:
The reference manual is under a free license, so I imagine we would fork it on mediawiki.org, incorporating changes for standard library functions that are changed or removed, and documenting any MediaWiki-specific libraries.
- The decision seems to have been Lua vs JS. What discarded a
custom language? (such as WikiScripts)
Mostly performance.
- How hard is it to translate Lua to PHP?
(for templates which then get converted into extensions)
I don't know why you would want to do that, but I imagine it would be no harder than converting between any other pair of weakly typed procedural languages.
Server side Lua scripts could potentially execute much faster than Zend PHP, so we might start wondering how hard it would be to convert the other way.
- Where would code live?
In a "scripts" namespace.
- I think it _would_ be possible to make an extension defined with
<wikiscript> to take over the page. (avoiding the need of template wrappers)
I'm not sure what you mean by that.
- What's the expected way of implementing 'libraries' in the wiki?
I think it would be best to follow the Lua conventions here:
http://www.lua.org/pil/15.html http://www.lua.org/manual/5.2/manual.html#6.3
Basically, a Lua package is a script which returns a table of functions. There are a number ways to do this, such as:
local P;
function P.foo() ... end
return P;
A require() function would return the table. Initially require() would only fetch packages from the local wiki, but when interwiki transclusion is introduced, it would be expanded to allow libraries on remote wikis to be fetched. We would isolate the global namespace of packages to prevent global variable exports. So package invocation would be along the lines of:
package = require("The package title"); package.foo()
Assuming I understood him correctly, Trevor expressed a preference for having a function that returns a function value which returns a table when called, instead of just returning the table directly. For example:
packageFunction = getScript("The package title") package = packageFunction() package.foo()
We could have that interface as well, if there's an application for it. For consistency with the Lua reference manual, I don't think this interface should be called require().
-- Tim Starling
Tim Starling wrote:
On 29/01/12 09:51, Platonides wrote:
Doesn't seem to have arrived into the list. Resending.
On 27/01/12 16:17, Platonides wrote:
- Is there a simple Lua syntax guide?
http://www.lua.org/manual/5.2/ looks more like a grammar. Certainly not suitable for end users.
Yes, the first edition of Programming in Lua is available online:
Thanks for this. I've added this link to (the brand new) https://www.mediawiki.org/wiki/Lua/Resources.
I also un-redirected https://www.mediawiki.org/wiki/Lua and fleshed it out a bit.
MZMcBride
Keeping this lua thread alive with some news. :)
For anyone interested in playing around with Lua on a wiki, I've enabled the Lua extension on http://lua.wikia.com It's just a test wiki right now with no real content. It will probably remain a test wiki, although if users want to flesh it out into a more authoritative site, that would be cool too. I am using http://www.mediawiki.org/wiki/Extension:Lua
Right now I've experimented a little bit and run across some questions.
Question 1) is that the right extension to be using? Is there another extension being developed?
Question 2) How to load external Lua code and make it available?
I created an example of an info box generated from lua here: http://lua.wikia.com/wiki/InfoboxLua However, it's clear that the first thing you need is a template library. I can't figure out how to load a third party lua library and make it available on a page. This probably needs some kind of support in the extension itself. For example:
http://lua.wikia.com/wiki/TestExpand?action=raw
Here I am using the lua loadstring() function to load/eval a simple template library and attempting to execute it in the same page. That doesn't work, although the same code works just fine from the command line.
I originally tried with a template like loadstring('{{Expand.Lua}}' and the template actually expands properly but runs into the same problem. The function created by load string is not actually executable. I'm not sure if it's a sandboxing issue or something else yet, I haven't had time to investigate. There's also a loadfile() function but that reads off the local file system. I suppose I could add things to the local lua include path and load them that way, but it seems better to have a self contained 100% wiki based solution there.
Has anybody gotten that far, or farther with Lua? I'm happy to hack on the extension a little bit if that's the proper solution here.
Owen
On Feb 1, 2012, at 3:51 PM, MZMcBride wrote:
Tim Starling wrote:
On 29/01/12 09:51, Platonides wrote:
Doesn't seem to have arrived into the list. Resending.
On 27/01/12 16:17, Platonides wrote:
- Is there a simple Lua syntax guide?
http://www.lua.org/manual/5.2/ looks more like a grammar. Certainly not suitable for end users.
Yes, the first edition of Programming in Lua is available online:
Thanks for this. I've added this link to (the brand new) https://www.mediawiki.org/wiki/Lua/Resources.
I also un-redirected https://www.mediawiki.org/wiki/Lua and fleshed it out a bit.
MZMcBride
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 29/02/12 09:01, Owen Davis wrote:
Keeping this lua thread alive with some news. :)
For anyone interested in playing around with Lua on a wiki, I've enabled the Lua extension on http://lua.wikia.com It's just a test wiki right now with no real content. It will probably remain a test wiki, although if users want to flesh it out into a more authoritative site, that would be cool too. I am using http://www.mediawiki.org/wiki/Extension:Lua
Right now I've experimented a little bit and run across some questions.
Question 1) is that the right extension to be using? Is there another extension being developed?
Victor Vasiliev has been working on an extension called Scripting which has a feature set closer to the plans we've been discussing at WMF. I'm not sure what its level of maturity is. It uses my luasandbox extension (/trunk/php/luasandbox) rather than shelling out.
Question 2) How to load external Lua code and make it available?
Victor's code has a function called mw.importModule which may be of some use.
I created an example of an info box generated from lua here: http://lua.wikia.com/wiki/InfoboxLua However, it's clear that the first thing you need is a template library. I can't figure out how to load a third party lua library and make it available on a page. This probably needs some kind of support in the extension itself. For example:
http://lua.wikia.com/wiki/TestExpand?action=raw
Here I am using the lua loadstring() function to load/eval a simple template library and attempting to execute it in the same page. That doesn't work, although the same code works just fine from the command line.
I don't think we will support loadstring(), instead all code will have to be loaded via the module system.
-- Tim Starling
very cool.
how can we see the lua errors? how can we load wiki pages as data in lua?
thanks, mike
On Tue, Feb 28, 2012 at 11:01 PM, Owen Davis owen@wikia-inc.com wrote:
For anyone interested in playing around with Lua on a wiki, I've enabled the Lua extension on http://lua.wikia.com It's just a test wiki right now with no real content. It will probably remain a test wiki, although if users want to flesh it out into a more authoritative site, that would be cool too. I am using http://www.mediawiki.org/wiki/Extension:Lua
On 01/26/2012 08:44 PM, Rob Lanphier wrote:
Coverage: https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2012-01-30/Techno...
http://developers.slashdot.org/story/12/02/01/1429229/wikipedia-chooses-lua-...
http://www.reddit.com/r/programming/comments/p4sve/wikipedia_chooses_lua_as_...
wikitech-l@lists.wikimedia.org