Forwarding from wikitech-l; discussion will be there.
-------- Original Message --------
Subject: [Wikitech-l] Extending Scribunto with new Lua functions
Date: Mon, 4 Feb 2013 14:46:44 +0100
From: Jens Ohlig <jens.ohlig(a)wikimedia.de>
Reply-To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>
To: wikitech-l(a)lists.wikimedia.org
Hello,
I guess this can be answered by Tim or Victor, but I'm grateful for any
pointers that can help me with a rather specific problem with Scribunto.
I'm currently working on the Wikidata project to include Lua functions
for templates that access Wikidata entities.
I've toyed around a bit and extended LuaCommon.php with a getEntities
function and a wikibase table to hold that function. Now I wonder if
there are any plans for Lua extensions outside the mw.* namespace.
I've added a wikibase.lua file and a wikibase.* namespace in Lua.
However, the way PHP and Lua play together and how Scribunto can be
extended looks a bit like black magic (which is to be expected, given
that Scribunto is far from finished).
Here are my questions:
1. Is there an easy way to add your own Lua functions (that call PHP Api
functions) to Scribunto other than writing them into LuaCommon.php?
2. Is using your own namespace the way to go?
3. Are there some kind of examples how to wrap PHP functions into the
Lua environment (using the frame etc)?
4. Is there any way to introspect or debug such wrapped functions?
Thanks for any suggestions!
Cheers,
Jens
--
Jens Ohlig
Software developer Wikidata project
Wikimedia Deutschland e.V.
Obentrautstr. 72
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das
Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hi Inez, hi Subbu
That's it!
I didn't know about prefix as I could not find anything about that variable at http://www.mediawiki.org/wiki/Parsoid. Maybe you should add that to the page.
But that alone didn't help! The thing that made it work was "$wgGroupPermissions['*']['read'] = false;" I had it on FALSE and when I changed it to TRUE, everything worked! Splendid, thank you.
Will it be possible to use Parsoid in the future with the variable being FALSE?
Stadtpirat!
>
> Message: 2
> Date: Fri, 25 Jan 2013 11:39:49 -0800
> From: Subramanya Sastry <ssastry(a)wikimedia.org>
> To: wikitext-l(a)lists.wikimedia.org
> Subject: Re: [Wikitext-l] No response to VisualEditor from Parsoid
> Message-ID: <5102DF85.7030002(a)wikimedia.org>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> In addition to what Inez suggested, another place to check based on
> #mediawiki-parsoid discussion about this report. -Subbu.
>
> ...
> <RoanKattouw> Does he have $wgGroupPermissions['*']['read'] === false
> perhaps?
> <marktraceur> Hm, that might be it
> <RoanKattouw> Does he have a current version of MW?
> <marktraceur> RoanKattouw: The permissions thing is exactly right
> <marktraceur> RoanKattouw: I didn't ask about the version
> <RoanKattouw> The perms thing will do it
> ...
> <RoanKattouw> The API refuses to do pretty much everything if you're
> logged out and have denied anons read rights
> <RoanKattouw> Actually, I'm pretty sure the only modules available in
> that case are help and login :)
> ...
>
>> Hi,
>>
>> In your localsettings.js (for Parsoid) you should have
>> "parsoidConfig.setInterwiki( 'skwiki', 'http://skwiki'; );" instead of
>> what you have now.
>>
>> Also try setting $wgVisualEditorParsoidURL = 'http://skwiki:8000'; and
>> $wgVisualEditorParsoidPrefix = 'skwiki' (make sure that nothing is
>> overwritten in extensions/VisualEditor/VisualEditor.php).
>>
>> Inez
>>
>> On Fri, Jan 25, 2013 at 3:25 AM, - - <stadtpirat11(a)ymail.com> wrote:
>>>
>>> Hi,
>>>
>>> I just asked this in IRC, but in case I need to go before I get a reply, I will ask you here:
>>>
>>> This really starts to bug me! I have no idea why Parsoid (and the VisualEditor) is not working! My wiki is working and the address is http://skwiki/w/Hauptseite ("Main_Page" is "Hauptseite" in German). I see the api page when I go to http://skwiki/api.php and I see the Parsoid page when I go to http://skwiki:8000. But I get errors when I click the VisualEditor link on http://skwiki/w/VisualEditor:Sandbox (no response).
>>>
>>> - The wiki is located at /usr/local/var/www/mediawiki/.
>>> - The webserver is configured to have the docroot for http://skwiki at /usr/local/var/www/mediawiki/
>>> - It also internally rewrites "/w/(.+)" to "index.php?title=$1"
>>> - /usr/local/var/www/mediawiki/extensions/Parsoid/js/api/localsettings.php is http://pastebin.com/BhB07BKY
>>> - /usr/local/var/www/mediawiki/LocalSettings.php is http://pastebin.com/Y1wLnwiL
>>>
>>> When I click the WYSIWYG-button at http://skwiki/w/VisualEditor:Sandbox, I get the following response in the web browser
>>>
>>>> {"error":{"code":"parsoidserver","info":"Error contacting the Parsoid server"}}
>>> and node js throws an error:
>>>
>>>> /usr/local/var/www/mediawiki/extensions/Parsoid/js/lib/mediawiki.WikiConfig.js:25
>>>> var nkeys = Object.keys( names );
>>>> ^
>>>> TypeError: Object.keys called on non-object
>>>> at Function.keys (native)
>>>> at Object.WikiConfig (/usr/local/var/www/mediawiki/extensions/Parsoid/js/lib/mediawiki.WikiConfig.js:25:21)
>>>> at MWParserEnvironment.switchToConfig (/usr/local/var/www/mediawiki/extensions/Parsoid/js/lib/mediawiki.parser.environment.js:215:21)
>>>> at ApiRequest.processListeners.processSome (/usr/local/var/www/mediawiki/extensions/Parsoid/js/lib/mediawiki.ApiRequest.js:53:4)
>>>> at process.startup.processNextTick.process._tickCallback (node.js:244:9)
>>> I really would like to get this working!
>>>
>>> Thanks,
>>>
>>> Stadtpirat
>>>
>>> ---------- Forwarded message ----------
>>> From: - - <stadtpirat11(a)ymail.com>
>>> To: "wikitext-l(a)lists.wikimedia.org" <wikitext-l(a)lists.wikimedia.org>
>>> Cc:
>>> Date: Fri, 25 Jan 2013 03:06:21 -0800 (PST)
>>> Subject: WG: No response to VisualEditor from Parsoid
>>> Hi,
>>>
>>>
>>> I just asked this in IRC, but in case I need to go before I get a reply, I will ask you here:
>>>
>>> This really starts to bug me! I have no idea why Parsoid (and the VisualEditor) is not working! My wiki is working and the address is http://skwiki/w/Hauptseite ("Main_Page" is "Hauptseite" in German). I see the api page when I go to http://skwiki/api.php and I see the Parsoid page when I go to http://skwiki:8000. But I get errors when I click the VisualEditor link on http://skwiki/w/VisualEditor:Sandbox (no response).
>>>
>>> - The wiki is located at /usr/local/var/www/mediawiki/.
>>> - The webserver is configured to have the docroot for http://skwiki at /usr/local/var/www/mediawiki/
>>> - It also internally rewrites "/w/(.+)" to "index.php?title=$1"
>>> - /usr/local/var/www/mediawiki/extensions/Parsoid/js/api/localsettings.php is http://pastebin.com/BhB07BKY
>>> - /usr/local/var/www/mediawiki/LocalSettings.php is http://pastebin.com/Y1wLnwiL
>>>
>>> When I click the WYSIWYG-button at http://skwiki/w/VisualEditor:Sandbox, I get the following response in the web browser
>>>
>>>> {"error":{"code":"parsoidserver","info":"Error contacting the Parsoid server"}}
>>> and node js throws an error:
>>>
>>>> /usr/local/var/www/mediawiki/extensions/Parsoid/js/lib/mediawiki.WikiConfig.js:25
>>>> var nkeys = Object.keys( names );
>>>> ^
>>>> TypeError: Object.keys called on non-object
>>>> at Function.keys (native)
>>>> at Object.WikiConfig (/usr/local/var/www/mediawiki/extensions/Parsoid/js/lib/mediawiki.WikiConfig.js:25:21)
>>>> at MWParserEnvironment.switchToConfig (/usr/local/var/www/mediawiki/extensions/Parsoid/js/lib/mediawiki.parser.environment.js:215:21)
>>>> at ApiRequest.processListeners.processSome (/usr/local/var/www/mediawiki/extensions/Parsoid/js/lib/mediawiki.ApiRequest.js:53:4)
>>>> at process.startup.processNextTick.process._tickCallback (node.js:244:9)
>>> I really would like to get this working!
>>>
>>> Thanks,
>>>
>>>
>>> Stadtpirat
Hi,
I just asked this in IRC, but in case I need to go before I get a reply, I will ask you here:
This really starts to bug me! I have no idea why Parsoid (and the VisualEditor) is not working! My wiki is working and the address is http://skwiki/w/Hauptseite ("Main_Page" is "Hauptseite" in German). I see the api page when I go to http://skwiki/api.php and I see the Parsoid page when I go to http://skwiki:8000. But I get errors when I click the VisualEditor link on http://skwiki/w/VisualEditor:Sandbox (no response).
- The wiki is located at /usr/local/var/www/mediawiki/.
- The webserver is configured to have the docroot for http://skwiki at /usr/local/var/www/mediawiki/
- It also internally rewrites "/w/(.+)" to "index.php?title=$1"
- /usr/local/var/www/mediawiki/extensions/Parsoid/js/api/localsettings.php is http://pastebin.com/BhB07BKY
- /usr/local/var/www/mediawiki/LocalSettings.php is http://pastebin.com/Y1wLnwiL
When I click the WYSIWYG-button at http://skwiki/w/VisualEditor:Sandbox, I get the following response in the web browser
> {"error":{"code":"parsoidserver","info":"Error contacting the Parsoid server"}}
and node js throws an error:
> /usr/local/var/www/mediawiki/extensions/Parsoid/js/lib/mediawiki.WikiConfig.js:25
> var nkeys = Object.keys( names );
> ^
> TypeError: Object.keys called on non-object
> at Function.keys (native)
> at Object.WikiConfig (/usr/local/var/www/mediawiki/extensions/Parsoid/js/lib/mediawiki.WikiConfig.js:25:21)
> at MWParserEnvironment.switchToConfig (/usr/local/var/www/mediawiki/extensions/Parsoid/js/lib/mediawiki.parser.environment.js:215:21)
> at ApiRequest.processListeners.processSome (/usr/local/var/www/mediawiki/extensions/Parsoid/js/lib/mediawiki.ApiRequest.js:53:4)
> at process.startup.processNextTick.process._tickCallback (node.js:244:9)
I really would like to get this working!
Thanks,
Stadtpirat
Fellow MediaWiki hackers!
After the pretty successful December release and some more clean-up work
following up on that we are now considering the next steps for Parsoid.
To this end, we have put together a rough roadmap for the Parsoid project at
https://www.mediawiki.org/wiki/Parsoid/Roadmap
The main areas we plan to work on in the next months are:
Performance improvements: Loading a large wiki page through Parsoid into
VisualEditor can currently take over 30 seconds. We want to make this
instantaneous by generating and storing the HTML after each edit. This
requires a throughput that can keep up with the edit rates on major
wikipedias (~10 Hz on enwiki).
Features and refinement: Localization support will enable the use of
Parsoid on non-English wikipedias. VisualEditor needs editing support
for more content elements including template parameters and extension
tags. As usual, we will also continue to refine Parsoid's compatibility
in round-trip testing and parserTests.
Apart from these main tasks closely connected to supporting the
VisualEditor, we also need to look at the longer-term Parsoid and
MediaWiki strategy. Better support for visual editing and smarter
caching in MediaWiki's templating facilities is one area we plan to look
at. We also would like to make it easy to use the VisualEditor on small
mediawiki installations by removing the need to run a separate Parsoid
service.
A general theme is pushing some of Parsoid's innovations back into
MediaWiki core. The clean and information-rich HTML-based content model
in particular opens up several attractive options which are discussed in
detail in the roadmap.
Please review the roadmap and let us know what you think!
Gabriel and the Parsoid team
--
Gabriel Wicke
Senior Software Engineer
Wikimedia Foundation
http://www.lua.org/wshop12.html
"the main goal of the workshop is to allow the Lua community to get
together and meet in person and talk about the Lua language, its uses,
and its implementation.... We shall have a plenary talk by Roberto
Ierusalimschy (Lua's chief architect) and several contributed talks....
If you'd like to speak at the workshop, please send a tentative title
and a short abstract to the organizers."
If you think it would be useful for Wikimedia to make an appearance, go
ahead and submit an abstract or suggest someone to go.
--
Sumana Harihareswara
Engineering Community Manager
Wikimedia Foundation
Dear all,
today I did some changes to simplify the math plug in a little bit and
as a preparation to the LaTeXML integration:
Please review and comment at:
https://gerrit.wikimedia.org/r/#/c/28824/
Best regards
Moritz
PS: I had some problems with the commit messages. They were not
accepted so the text is lost.
What I basically did that I separated the code of the divers ways of
rendering into separate classes that extend a common base class.
On Mon, Oct 15, 2012 at 12:13 AM, Moritz Schubotz
<physik(a)physikerwelt.de> wrote:
> HI DJ,
>
> first of all thanks a lot for your helpful comments.
>
> On Sun, Oct 14, 2012 at 10:31 PM, Derk-Jan Hartman
> <d.j.hartman+wmf_ml(a)gmail.com> wrote:
>> On 14 okt. 2012, at 17:45, Moritz Schubotz <physik(a)physikerwelt.de> wrote:
>>
>>> Summary: I'm looking for people to discuss about the parsing of math.
>>
>> The last person who really did some work on the extension was Brion Vibber. Almost all of the (limited) work that has gone into the extension over the last 1,5 year was actually based on the efforts of User:Nageh on the English wikipedia and his MathJax userscript. Beyond these MathJax changes, the extension has not seen much developer involvement for quite some while.
>>
>>> I came up with a proposal for a new version of the rendering of the
>>> <math> tag. I proposed to use LaTeXML to convert the LaTeX expressions
>>> in the math tag to MathML. If the browser is not capable of displaying
>>> MathML I use MathJaX to display the MathML output in the browser.
>>> My implementation (the LaTeXML branch) has only a few very little
>>> differences in contrast to the master branch. I have the feeling that
>>> the php code of the math extensions could be improved. For example I'd
>>> suggest to put all the texvc related stuff to another class.
>>> Furthermore I was thinking about an asynchronous rendering of the
>>> formula, which would speed up page loading time especially for major
>>> edits.
>>> At
>>> http://wiki.physikerwelt.de/images/text_math_search.pdf
>>> you find the draft of a paper where I describe in detail what
>>> I changed and why it is an improvement. The paper will appear soon in
>>> the postconference proceedings of CICM2012.
>>> Now I want to figure out, who is working on the development of the
>>> math extension, and who wants to discuss with me about our ideas.
>>> I'm open to any kind of suggestions and questions.
>>
>> Discussion is always welcomed and in this case probably best on the mailinglist.
>>
>> You seem to propose switching our current texvc -> image converter (by some users extended by using MathJax) with a LaTeX -> MathML presentational renderer that would use MathJax as a backup. The renderer would also output Content MathML to make the content easier to index and search. In general I would support this, definitely morally.
>
> I see MathJaX rather as a renering engine that supports browsers that
> do not such a good job in displaying MathML, rather than a fall back
> alternative.
>
>>
>> There are however some problems here as well.
>> - If a browser does not support MathML and not support (have enabled) Javascript, you would see nothing reading the article.
>>
>
> MathML is also capable of producing images. However this feature is
> more designed for the rendering of whole documents rather than for
> small latex snippets. So this would require a little customization.
>
>> - Browser support for MathML is still spotty. It would be nice if we can gather some actual numbers on this. There are apparently also significant bugs in rendering implementations. I believe this is one of the reasons that the default of MathJax is HTML+CSS now, and not MathML.
>>
>> - You are determining support serverside right now ? This would require fragmenting the cache when we move forward.
>
> I'm nor sure if I understand, what you mean by that. The math table
> becomes much larger as it is if you use texvc, since the matml column
> contains all the mathml code which is not really optimized with regard
> to space.
>
>>
>> - I think that currently a large group of users would fall into your MathJax group. Especially those with older browsers. However, we don't particularly like MathJax for that usergroup. It is incredibly slow on the client side and this greatly affects the userbase with less capable computers (of which we have quite a lot). This is one of the main reasons why we are not particularly pushing for this solution as a default right now (let alone as the primary backup method). I would seriously consider just skipping MathJax (at least by default) and just show images for the fallback option.
>
> MathJaX is slow if you want to display LateX stuff. If you want to
> display MathML with the help of MathJaX- that's what I propose-
> MathJaX is reasonably fast.
>>
>> - In order to provide proper MathML support, a proper font is required. Many Operating Systems do not yet have this font support. This is something that possibly could be solved with WebFonts, but will require a bit of work.
>>
>> - MathML does not, I believe, provide everything that is possible with texvc. This is another issue that we are currently seeing with MathJax, though we (Nagheh actually) has added most of them to MathJax by now. For MathML the solution seems not so simple however. This probably means we either need to phase out part of what we currently support (always complicated) or fragment the implementation for these particular situations.
>
> This would be really interesting to see which are those things, that
> can not be displayed with MathML.
>
>>
>> - I'm not aware of the current quality of LaTexML. Is there some information about the 'completeness' of the capabilities and quality of the implementation? Comparisons with other implementations ? That would be useful information.
>
> I'll provide some information soon.
>>
>> - Outputting both Presentational and Content MathML into the HTML will significantly increase the page size.
>
> This is true. I saw that in my experiments.
>
>>
>> - LaTeXML will have to be reviewed by one of the developers of the security team.
>>
>
> Yes. I think the major advantage of LaTeXML in contrast to texvc is
> that it can be run on a separate machine, with very restricted rights.
>
>> These would be the issues that I think would require assessment and solving at some point before a full solution can be deployed on Wikipedia. But of course, we can just start and expose the functionality at a later time.
>>
>> DJ
>>
>>
>> P.S. Seems that MathJax 2.1 will be released soon: http://www.mathjax.org/2012/10/01/news/mathjax-v2-1-beta-now-available-on-t…
>> _______________________________________________
>> MediaWiki-l mailing list
>> MediaWiki-l(a)lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
>
> Thanks again for the feedback.
>
> Best regards
> Moritz
>
> --
> Mit freundlichen Grüßen
> Moritz Schubotz
>
> Telefon (Büro): +49 30 314 22784
> Telefon (Privat):+49 30 488 27330
> E-Mail: schubotz(a)itp.physik.tu-berlin.de
> Web: http://www.physikerwelt.de
> Skype: Schubi87
> ICQ: 200302764
> Msn: Moritz(a)Schubotz.de
--
Mit freundlichen Grüßen
Moritz Schubotz
Telefon (Büro): +49 30 314 22784
Telefon (Privat):+49 30 488 27330
E-Mail: schubotz(a)itp.physik.tu-berlin.de
Web: http://www.physikerwelt.de
Skype: Schubi87
ICQ: 200302764
Msn: Moritz(a)Schubotz.de
Moritz, thanks for your note. Forwarding to the wikitext-l list, which
is specifically interested in MediaWiki's parsing.
--
Sumana Harihareswara
Engineering Community Manager
Wikimedia Foundation
On 10/14/2012 11:45 AM, Moritz Schubotz wrote:
> Summary: I'm looking for people to discuss about the parsing of math.
>
> Dear all,
>
> I came up with a proposal for a new version of the rendering of the
> <math> tag. I proposed to use LaTeXML to convert the LaTeX expressions
> in the math tag to MathML. If the browser is not capable of displaying
> MathML I use MathJaX to display the MathML output in the browser.
> My implementation (the LaTeXML branch) has only a few very little
> differences in contrast to the master branch. I have the feeling that
> the php code of the math extensions could be improved. For example I'd
> suggest to put all the texvc related stuff to another class.
> Furthermore I was thinking about an asynchronous rendering of the
> formula, which would speed up page loading time especially for major
> edits.
>
> At
> http://wiki.physikerwelt.de/images/text_math_search.pdf
> you find the draft of a paper where I describe in detail what
> I changed and why it is an improvement. The paper will appear soon in
> the postconference proceedings of CICM2012.
> Now I want to figure out, who is working on the development of the
> math extension, and who wants to discuss with me about our ideas.
> I'm open to any kind of suggestions and questions.
>
> Best regards
> Moritz
>
> --
> Mit freundlichen Grüßen
> Moritz Schubotz
>
> Telefon (Büro): +49 30 314 22784
> Telefon (Privat):+49 30 488 27330
> E-Mail: schubotz(a)itp.physik.tu-berlin.de
> Web: http://www.physikerwelt.de
> Skype: Schubi87
> ICQ: 200302764
> Msn: Moritz(a)Schubotz.de
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Sorry for the spam, but the ContentHandler changes especially may affect
you -- if you have any time this weekend or next week to do some
testing, we'd appreciate it. Thanks.
-Sumana
-------- Original Message --------
Subject: Please notice and report big glitches - changes coming
Date: Fri, 12 Oct 2012 17:14:05 -0400
From: Sumana Harihareswara <sumanah(a)wikimedia.org>
Organization: Wikimedia Foundation
To: Coordination of technology deployments across languages/projects
<wikitech-ambassadors(a)lists.wikimedia.org>
On Monday we start deploying a new version of MediaWiki, 1.21wmf2, to
the sites, starting with mediawiki.org and 2 test wikis
(https://www.mediawiki.org/wiki/MediaWiki_1.21/Roadmap). 1.21wmf2 will
have 3 big new things in it and we need your help to test on the "beta"
test site http://deployment.wikimedia.beta.wmflabs.org/wiki/Main_Page
now to see if there are any really critical bugs.
1) The new ContentHandler (
https://www.mediawiki.org/wiki/ContentHandler ) might affect handing of
CSS and JavaScript pages, import/export (including PDF export), and API
stuff, especially when rendering and editing. I'd suggest we also look
out for issues in template rendering, images and media handling,
localisation, and mobile device access. (merged on Oct 9)
2) High-resolution image support. This work-in-progress will try to
give higher-res images to high-density screens that can support it, like
new Retina displays. More info at
https://gerrit.wikimedia.org/r/#/c/24115/ . One of the bigger risks of
the high res stuff is load-based, since we may see substantial new load
on our image scalers. So *all* image scaling might be impacted. (merged
on Oct 11)
3) "Sites" is a new backend to represent and store information about
sites and site-specific configuration. This code is meant to replace
the current interwiki code, but does not do so just yet. Still, keep an
eye out for site-specific configuration or interwiki issues.
Right now the version of MediaWiki on the beta cluster dates from 9 Oct
and thus has ContentHandler but not the high-res image support or Sites.
So please test on the beta sites now and look out for these issues on
your sites in the weeks ahead.
https://www.mediawiki.org/wiki/Category:MediaWiki_test_plans has some
ideas on how to find errors.
Thanks! With your help we can find bugs early and get them fixed before
they affect lots of readers and editors.
--
Sumana Harihareswara
Engineering Community Manager
Wikimedia Foundation