On Tue, Aug 14, 2012 at 7:49 PM, Tim Starling <tstarling(a)wikimedia.org> wrote: Huzzah! Next step is to deploy this on mediawiki.org, which we plan to do soon (as early as next week). From there, we'll need some time figuring out the performance characteristics of this (making sure we're actually coming out ahead) as well as converting some key templates over. Of course, I'm using "we" pretty loosely; Tim's largely been managing this process himself. :) Just poking around this evening, I imported some work that TheDJ did over from scribunto.wmflabs.org over to test2, and it seems to work about as well there: http://test2.wikipedia.org/wiki/Template:Coord/testcases It looks like there still some work to do to finish off that template, but he made a lot of progress. One obvious target for converting to Lua would be the Cite template. It would be really great to take an article with a long parse time (e.g. the "Barack Obama" or "The Beatles"), import it to test2, and try to get the parse time down to something reasonable simply by converting the Cite+other key templates to Lua. Rob

On Tuesday, August 14, 2012 at 11:31 PM, Rob Lanphier wrote: Template:Cite is terrifyingly complex. What are some other key templates? There must be others like me who want the fame and fortune of being an early adopter, without all that the hard work :) -- Ori Livneh ori(a)wikimedia.org

I know if I happen to have time to learn Lua, I'd definitely tackle the ArticleHistory template. One of the more complex ones out there. *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerromeo(a)gmail.com On Mon, Aug 20, 2012 at 3:44 AM, Ori Livneh <ori(a)wikimedia.org> wrote:

Lua is now enabled on www.mediawiki.org. Note that this is not a temporary deployment. You can rewrite existing templates to use Lua, we're not going to break them by turning it off again. -- Tim Starling

Here it is: http://lists.wikimedia.org/pipermail/wikitech-l/2012-August/062329.html It was on August 15 and it says "next week". -- Amir 2012/8/22 Yury Katkov <katkov.juriy(a)gmail.com>om>:

I'm very sorry for my bureaucratic tone: I don't want to harm anyone and I'm as excited about new feature as everybody here. That's just seems like pretty important change on a pretty big website with a big community: it's rude to just modify it in all sort of ways without prior discussions, recognizable announcements and publicly available plans. ----- Yury Katkov On Wed, Aug 22, 2012 at 12:06 PM, Amir E. Aharoni <amir.aharoni(a)mail.huji.ac.il> wrote:

Like Amir said, it was announced a week ago. There were no objections. There were very few comments on the test2 deployment, probably because the Labs deployment served the same purpose. What we want from the mediawiki.org deployment is to start building a community of people who will convert templates to Lua, not just because they want to help us test the software, but also because they want to get useful things done. Their comments will help to drive future development. Our goal is to deploy this extension to all Wikimedia wikis. If you don't like that idea, now would be a good time to say something. The schedule for deployment has not yet been decided. It will depend on what bug reports and feature requests are submitted by the early adopters. Performance issues may be found, once we have real-world test cases. We will need to decide how much extra development work is needed before a full-scale rollout. It's been over a year since I started work on Lua support. From the outset, I wanted it to be a project with a constrained scope, a project that can be brought to completion, instead of trailing off into vapour. I wanted to make something happen. So my inclination is to push for deployment with a minimum of additional development work. But I'm not the target audience; my inclinations have to be weighed against the needs of the users. -- Tim Starling On 22/08/12 18:01, Yury Katkov wrote:

On 08/22/2012 12:18 PM, Tim Starling wrote: in the name of countless Wikipedians, who are struggeling with that horrible Template/Magic word/ParserFunctions syntax, I say: thank you :) This page is dedicated to its victims: http://meta.wikimedia.org/wiki/User:Church_of_emacs/Template_love Cheers, Tobias

Anyone else reminded of the recent topic:"Wikimedians are rightfully wary" I think the sorts of things that MZMcBride was talking about in his op-ed. Sorry for the slightly off topic post, but this seemed like as good a time as any to point out a real life example. Thank you, Derric Atzrott

----- Yury Katkov On Wed, Aug 22, 2012 at 7:08 PM, Tyler Romeo <tylerromeo(a)gmail.com> wrote: as far as I can see this extensions have been deployed : http://www.mediawiki.org/wiki/Extension:Scribunto , not this one: http://www.mediawiki.org/wiki/Extension:Lua

On Wed, Aug 22, 2012 at 2:39 PM, Ryan Kaldari <rkaldari(a)wikimedia.org> wrote: Furthermore, mediawiki.org is a community essentially owned by the developers, not to mention this feature is non-user facing. If people don't want to use it, they don't have to. I do believe consensus should be sought when enabling extensions like moodbar and what not on enwikipedia, but this is nothing like that situation. --bawolff

Which community should be consulted on this technical decision? What if enwiki wants prolog and dewiki wants lua and enwikitionary wants javascript? Occasionally technical decisions have to be made by the developers. The templating language is a technical decision. It's really not something that is up for editor community debate. This decision was hashed out over months (really a couple of years if we consider the original iteration of this idea) with the developer community. If the editor community wishes to take part in these kinds of decisions, they should join wikitech-l. - Ryan

This is the exact kind of attitude the op-ed in the Signpost is addressing. When making major feature decision, such as redoing the entire templating system, we cannot just say to editors "oh, if you want some input, go and join our mailing list". That's just a passive-aggressive way of pushing editors out of the conversation. How many purely editors, i.e., not developers, are on this list actively participating in discussion? And this isn't a technical decision, it's a requirements decision. We're not deciding what algorithm to use, or what object design to implement, we're deciding what features would be best for the users of Wikipedia. The reason this extension was implemented (hopefully) was so that users could have a better templating experience, but how can you possibly assume to know what is best for the user without asking the users themselves? And no, we cannot be expected to consult every language wiki, but on the other hand we cannot completely ignore the community and suddenly launch this new extension on them as if they'd known about it for years. *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerromeo(a)gmail.com On Wed, Aug 22, 2012 at 3:38 PM, Ryan Lane <rlane32(a)gmail.com> wrote:

Lay users don't write templates either. People who write templates are wizards. Templates make my eyes bleed and my mind hurt, and I've been developing for quite a long time. I don't think the editor community has much reason to participate. The template creator community does. They are more than technical to understand things on wikitech-l. The "community" is not a single thing. The community is made up of hundreds of sub-communities. If people are interested in technical decisions, they need to participate where technical decisions are made. - Ryan

On 22 August 2012 20:58, Ryan Lane <rlane32(a)gmail.com> wrote: AIUI the Lua idea was explicitly run past the few people who write the insanely horrible brainfuck-like ParserFunctions templates. (Is this correct?) They would be the relevant part of the editor community - most of the editor community just want the template itself to work, they neither know nor care about the details of the plumbing. So, has anyone translated {{cite}} to Lua? Is it comprehensible? Does it run faster? Well, one would think so. - d.

On Wed, Aug 22, 2012 at 12:51 PM, Tyler Romeo <tylerromeo(a)gmail.com> wrote: I'm not sure I entirely agree with this assessment. The community has complained about the current templating system -- the requirements decision is arguably that something should be done to improve how templates are created and their overall performance. The technical decision, i.e. what technology to use to solve this issue, is what was decided by selecting LUA. That said, I do agree that we should plan the roll-out rather than just tossing it over the wall -- preparing some initial documentation and tutorials to show HOW and WHY it's easier/faster than the old system would go a long way (imho) to assuage complaints of disconnect from what the community wants. Nabil

Fair enough. That satisfies my threshold for consulting the community. If the people who would be most affected had a chance to give their input I would say that this project is good to go then. This also seems perfectly fair. My points have been refuted and I humbly accept defeat. +1 million. This is a must. No one likes change, but anything we can do to make that change easier on people will make things better. The old templating system will still be functional correct? Thank you, Derric Atzrott

As long as people in the templating community were at least consulted with, then that's fine. I'm just saying we cannot randomly throw features onto users without discussing it with them. *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerromeo(a)gmail.com On Wed, Aug 22, 2012 at 4:13 PM, Chad <innocentkiller(a)gmail.com> wrote:

Hi everyone, At the risk of repeating what others have said, a quick note in response to Yury: On Wed, Aug 22, 2012 at 1:01 AM, Yury Katkov <katkov.juriy(a)gmail.com> wrote: Tim and I agreed that a further announcement probably wouldn't be necessary. We make rather large changes to mediawiki.org with less fanfare than what this has already received. The roadmap generally is published here: http://www.mediawiki.org/wiki/Roadmap ...and is updated frequently. A longer view is available here: http://www.mediawiki.org/wiki/Wikimedia_Engineering/2012-13_Goals Additionally, we provide (at least) monthly updates here: https://www.mediawiki.org/wiki/Lua_scripting Our transition to Lua is one development project without a product manager, which shows in the long-term planning for this. We know there's a lot of planning work needed for a general rollout to the other wikis, and would appreciate volunteers to help out with this. If you're interested, please either send me mail or drop a note on the talk page here: https://www.mediawiki.org/wiki/Talk:Lua_scripting Thanks! Rob

On Wed, Aug 22, 2012 at 1:24 PM, Tyler Romeo <tylerromeo(a)gmail.com> wrote: A good default assumption would be that we've done what we're supposed to be doing as software developers. User testing is a normal part of software development. It's kind of bad faith to assume we're not engaging end-users at all. - Ryan

On Wed, 22 Aug 2012 14:01:00 -0700, Risker <risker.wp(a)gmail.com> wrote: How? I don't see how the user who puts {{cite|...}} and never takes one look at the source for Template:Cite is the end user. Template inclusion syntax is the same. No matter if cite uses ugly ParserFunctions, Lua, or makes a call out to a custom written extension the syntax used to include the template stays the exact same. I don't see how these users who never even look at template source should care what the template they use is powered by. -- ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

On 8/22/12 2:01 PM, Risker wrote: I imagine most templates won't actually change at all. The main purpose of Lua/Scribunto is to replace the nightmarish under-the-hood uber-templates templates like Citation/core with something that isn't so nightmarish. Most templates just do simple parameter substitutions and if/else decisions and don't require any real programming language functionality. Templates that have very complex behavior, however, are extremely difficult to implement in WikiText (and very expensive to parse), and there are only a small handful of people who work on these templates. We should certainly recruit those editors to help us test Scribunto now that it's on mediawiki.org. Getting buy-in from them will be critical to the tool's success. As one of the editors who works on templates like Citation/core, I'm personally very excited about the project so far, and can't wait to see it in production. Ryan Kaldari

On Wed, Aug 22, 2012 at 5:36 PM, Risker <risker.wp(a)gmail.com> wrote: I believe it will be much better after the switch to Lua. Unlike wikitext, which isn't a programming language (but can be made to act sort of like one), Lua is an actual language. Wikitext programming only came about to begin with due to some quirks in transclusion that allowed {{qif}}. ParserFunctions came to fill that need, but were never really designed to do some of the complex logic that modern templates require. Lua, as a programming language (but the idea applies if we'd chosen JS, Cobol, or anything else) will handle this sort of thing far more elegantly than endless iterations of adding new ParserFunctions to handle more edge-cases. Granted, I've never written a complex template (only looked and shuddered). -Chad

Speaking of complex templates, has there been any work to move existing templates to Lua? Because I'd love to start on the ArticleHistory template if nobody else is doing it. *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerromeo(a)gmail.com On Wed, Aug 22, 2012 at 5:43 PM, Chad <innocentkiller(a)gmail.com> wrote:

On 08/22/2012 05:50 PM, David Gerard wrote: Seconding this. Yes, please please go ahead! -- Sumana Harihareswara Engineering Community Manager Wikimedia Foundation

Domas Mituzas wrote: Silly Domas. You forgot your trademark "Hi!" and "BR,". MZMcBride

Tim Starling wrote: Hi. I believe you and I discussed the _need_ to have working interwiki transclusion before this extension sees widespread deployment. I consider this a blocker to widespread deployment. There are _a lot_ of lessons to be learned from Gadgets and I think this one is key. (For anyone who's unaware, the Gadgets extension allows for essentially JavaScript modules to be installed on a wiki, with a user preference pane that allows users to enable and disable the modules on an individual basis. More information here: <https://www.mediawiki.org/wiki/Extension:Gadgets>.) Lua is going to make complex templates even more complex. It may make sense to centralize certain templates. For certain modules, it's definitely going to make sense to centralize. For other modules, it'll make less sense. Perhaps mediawiki.org will be that central repository; perhaps not. (By the way, you can see that I'm mixing up terminology: templates vs. modules. It's still unclear to me what relationship the Module namespace has to the Template namespace. Is it intended as a supplement? A replacement?) I don't think it was a bad idea to deploy Scribunto to mediawiki.org. It seemed kind of silly to make a guarantee that Lua won't be disabled at any time in the future, but I think you're okay with possibly looking silly if problems arise and Lua needs to be disabled temporarily. :-) That said, Tyler is certainly right that before there is a Wikimedia-wide deployment, there should be more thought about how to best utilize and manage this new template scripting language. I'm a huge fan of organic growth (and I think a good deal of organic growth will be needed here to ensure that template monsters don't suddenly show up next week before anyone is armed and ready!), but we also need to make sure we learn from past mistakes and experiences. I think the Gadgets evolution is the best (i.e., closest) model to study, but there may be others to study as well. Auto categorization, manual categorization, and a few other administrative features are missing (yes, I'll file bugs at some point). We also need better/smarter documentation (including a glossary of terms). But most of these issues aren't necessarily blockers to widespread deployment. The only other possible blocker that I think has come up is the third party user issue. English Wikipedia templates in particular are horribly common on third-party wikis. I believe you did some work to ensure that Lua/Scribunto would be supported in other hosting environments, but I'm not sure of the status of this. This issue could also be mitigated by centralized modules, of course. understood its purpose given the existence of test[1]. MZMcBride [1] https://test.wikipedia.org

Coordinates and location maps, to start with. I think that most wikis copy these two verbatim. 2012/8/23, Ryan Kaldari &lt;rkaldari(a)wikimedia.org&gt;rg>: -- Wysłane z mojego urządzenia przenośnego -- Matma Rex

Ryan Kaldari wrote: It's not really about expensive templates, it's about code fragmentation. Gadgets currently have this problem because there are no global gadgets: you get forks of the same code with very minor tweaks spread out across wikis. Or even forks with identical code, but a change to a particular wiki's gadget has to be manually synced to other wikis. It quickly becomes a nightmare. This is my concern with Lua modules, in a nutshell. MZMcBride

On Thu, Aug 23, 2012 at 4:00 AM, Tim Starling &lt;tstarling(a)wikimedia.org&gt; wrote: Famous words, kept in the archives forever :). Siebrand

On Wed, Aug 22, 2012 at 5:16 PM, Derric Atzrott &lt;datzrott(a)alizeepathology.com&gt; wrote: I don't think that the process of introducing Scribuntu, which we have been talking about for years now, compares in anyway to something like, for example, WikiLove or AFT which essentially were deployed out of nowhere (to make an hyperbole) Bryan

On 23 August 2012 10:38, Tei &lt;oscar.vives(a)gmail.com&gt; wrote: Speed is secondary. The main thing is that Lua is actually designed to be a usable programming language, whereas ParserFunctions is a Turing tarpit, only accidentally Turing-complete, in which "everything is possible but nothing of interest is easy." [1] Complicated ParserFunctions templates look like several days' Daily WTF [2] because of this. Even if there were no speed improvement, it would be a vast improvement just in programmability. (Though that would mean much more use of programmatic templates, so in practice we need speed too.) - d. [1] https://en.wikipedia.org/wiki/Turing_tarpit [2] http://thedailywtf.com/

On 23/08/12 07:18, Siebrand Mazeland (WMF) wrote: Indeed. I wonder how long it will be before someone uses Lua templates to do something like this: http://bellard.org/jslinux/ -- N.

On 23/08/12 15:10, Neil Harris wrote: Just to clarify, in case anyone gets the wrong impression from my last two posts -- I'm not snarking against Tim here: Lua on MediaWiki is fantastic, will be one of the best things ever for the platform (along with the Visual Editor) and I look forward to using it on Wikipedia! -- N.

way that Commons does for images. that have built up in the old template system. Could probably throw gadgets in the mix too. Thank you, Derric Atzrott

On Thu, Aug 23, 2012 at 11:39 AM, Derric Atzrott &lt;datzrott(a)alizeepathology.com&gt; wrote: Some previous discussions on this topic: https://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2011/05#Com… Support crosswiki template inclusion (transclusion => interwiki templates, etc.) https://bugzilla.wikimedia.org/show_bug.cgi?id=4547 Create "Data Commons" or "Wikidata" project (tracking) https://bugzilla.wikimedia.org/show_bug.cgi?id=30345 Best regards, Helder

On 24/08/12 06:37, Yury Katkov wrote: Perhaps ParserFunctions etc. could eventually be supported in Lua via a syntax extension of the new Lua mechanism, eliminating the need for separate extensions? -- N.

On 08/23/2012 12:30 AM, Mike Dupont wrote: Gadgets (v2) makes an attempt to fix the "no central repository" problem, so maybe that would be a place to start looking. But MZ is right. The duplication and inadvertent code forks that result from not having a way to easily re-use templates and Gadgets is a real problem that we should start to address. -- http://hexmode.com/ Human evil is not a problem. It is a mystery. It cannot be solved. -- When Atheism Becomes a Religion, Chris Hedges

Maybe it's just me opinion, but I believe that even with Lua, the existing templating system is not entirely obsolete. I'm sure ParserFunctions still has a legitimate purpose. For example, what about a template that's just a simple if statement (if this, else that). There's not really a need to make a Lua module for something that basic. *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerromeo(a)gmail.com On Fri, Aug 24, 2012 at 8:56 AM, Helder . &lt;helder.wiki(a)gmail.com&gt; wrote:

On 24/08/12 14:07, Derric Atzrott wrote: That's right. I see the hierarchy as being something like this: * simple templates written in the existing template markup, exactly as done at present * complex templates written in Lua, maintained on-wiki, perhaps in a central repositry, but again using normal wiki processes for editing, protection, etc. * common library routines written in Lua for addressing common utility functions used in many complex templates, maintained somewhere like git, with much stricter check-in rules, and viewed as being part of the core software Possibly ParserFunctions will end up being semantic sugar for invoking primitives written in Lua, as part of that set of library routines. -- N.