Hello Conrad, It should be exactly the same, except written in Python. Maybe this should be my project, then. Actually... I've never used PHP for real programming, but how difficult would it be to write a really simple, stupid first pass at a DFA parser? I suspect I'd need much more than three months to make it useful, but would it be possible to implement some coherent subset of the features? E.g., building the LR0 automaton, at least? I suggested a Python port because http://www.mediawiki.org/wiki/Summer_of_Code_2010#MediaWiki_core lists it as a potential project idea. I was under the impression that people around here did not want to leave texvc in OCaml. Is this wrong? Yours, Damon Wang

2010/3/23 Aryeh Gregor <Simetrical+wikilist(a)gmail.com>om>: DFAs parse regular languages, which means those languages can also be expressed as regexes. In fact, the regexes accepted by the preg_*() functions allow certain extensions to the language theory definition of regular expressions, allowing them to describe certain non-regular languages as well. In short: preg_split() can do everything a DFA can do, and more. The only reason to use a DFA parser would be performance, but since the preg_*() functions are so heavily optimized I don't think that'll be an issue. Having it in a language no one understands is a bad thing and leads to maintenance not happening, so yeah, we definitely want it rewritten in PHP. If the PHP implementation turns out to be too slow to run on WMF, for instance, we could do a C++ port à la wikidiff2 (a C++ port of our ludicrously slow PHP diff implementation). Roan Kattouw (Catrope)

On Tue, Mar 23, 2010 at 1:00 PM, Roan Kattouw <roan.kattouw(a)gmail.com> wrote: This much I know, but is LaTeX actually a regular language? On Tue, Mar 23, 2010 at 1:13 PM, Conrad Irwin <conrad.irwin(a)googlemail.com> wrote: It's not security-critical. The worst you could possibly do is DoS, and any DoS could be instantly shut off by just turning off math briefly. Furthermore, the part that makes DoS impossible is a quite small portion of the code that would need to change effectively never. No, the problem is that most PHP programmers have never even heard of OCaml, let alone used it.

2010/3/23 Roan Kattouw <roan.kattouw(a)gmail.com>om>: Sorry, I got confused and wrote DFA when I should have written LALR. DFAs cannot parse even the allowed subset of AMS-LaTeX, because there are some permitted environments. Without claiming to know much formal language theory, a rule of thumb is that languages with matched delimiters were never regular, because of the pumping lemma: http://en.wikipedia.org/wiki/Pumping_lemma_for_regular_languages So, for example, it's theoretically impossible to check that parentheses nested correctly using regular expressions, and similarly it'd be impossible to check that the \begin and \end commands matched up. In practice there might be ways to hack around that by using multiple regular expressions and manually tracking how they nest, but at that point we're basically writing half of a bad LALR parser. Fortunately, though, Python has parser generators! And if we're really concerned about speed, there's PyBison, which does the parsing in C and apparently produces (at least) five-fold improvements over Python-native alternatives. Yours, Damon Wang

Conrad Irwin wrote: Just because a language is context-sensitive doesn't mean it will be hard to write a parser for it. That's just a myth propagated by computer scientists who, strangely enough given their profession, have a disdain for the algorithm as a descriptive framework. In the last few decades, pure mathematicians have been exploring the power of algorithms as a general description of an axiomatic system. And simultaneously, computer scientists have embraced the idea that the best way to process text is by trying to shoehorn all computer languages into some Chomsky-inspired representation, regardless of how awkward that representation is, or how inefficient the resulting algorithm becomes, when compared to an algorithm constructed a priori. -- Tim Starling

On Wed, Mar 24, 2010 at 9:16 AM, Trevor Parscal <tparscal(a)wikimedia.org> wrote: Also rewrite parser functions to use it? that would be interesting on en.wiki since they are always complaining about the syntax. jks -Peacvhey

On 03/23/2010 10:44 PM, Tim Starling wrote: Context free grammars have a strong advantage because they come with documentation built in. Given a BNF-esque description of a language, it is possible to understand, at a high level, how the language works. This means it's easy to write a parser (though much easier to get the parser written automatically), it's also more pleasant to verify properties of the language (no state to keep track of), and to reason about the consequences of modifications. Of course it is possible to give decent documentation for a context-sensitive language; from what I've seen, this just doesn't happen. Take for example Python and perl, or Markdown and MediaWiki. In both cases the former has a syntax that can be mainly modeled by a context free grammar and there are many implementations that all work. The latter of the pair has a context sensitive grammar defined only by the reference implementation, there are no other parsers with feature completeness. This is certainly not a technical limitation, rather a reflection on the ability of your average human programmer. Conrad

On Tue, Mar 23, 2010 at 6:28 PM, Conrad Irwin <conrad.irwin(a)googlemail.com> wrote: What does that mean? LaTeX can invoke external programs? Using what commands? Is this functionality actually enabled in practice in stock LaTeX installs?

If people won't remember, surely either the MediaWiki installer or the extension itself can be made to check this and simply refuse to enable <math> tags.

Aryeh Gregor wrote: Getting shell access is not a DoS or XSS. Specially for a large majority of installs where it compromises their only account. Does this mean that we shouldn't rewrite it? No. We should rewrite it, and make it more secure. We start it by having enough eyes on the code. I wouldn't be surprised if we found a vulnerability on texvc during the rewrite. Running the LaTeX interpreter under ulimit -u 1 should be provide a quite safe default against external launches. But take into account that file writes are also a dangerous vector.

Hello Rob, I understand, and that'd be very helpful. To be honest, I'm not passionately committed to any project at all. I've been writing projects for university and for a computer lab I work at, but it's mostly small, one-off sysadmin things and usually the emphasis is more on "xyz server has to be back up before we open tomorrow" than writing good, clean code. So, yes, I'd welcome other suggestions. There are ways to make this run faster if performance is a concern. For example, mod_python or mod_wscgi, or explicitly pulling the Python out into a standalone daemon that listens for requests from the webserver. Another possibility be writing it in C to avoid all interpreter overhead, and using a foreign function interface. Unfortunately, I'm not familiar with PHP's FFI. Google takes me to http://wiki.php.net/rfc/php_native_interface which seems to think that as of a year ago there weren't any good ones, but this doesn't look too painful: http://theserverpages.com/php/manual/en/zend.creating.php I'm most familiar with Python and C, for whatever that's worth coming from an undergrad who didn't know Python existed five years ago. I learned PHP to maintain the web interfaces of an in-house print system at work, but I haven't used it for anything as involved as what we're discussing here. So, in terms of productivity, yes, if I have to work in PHP my mentor will probably get asked a few more newbie questions. In terms of happiness, though, it'd be a great opportunity to dig into PHP and finally learn to use it as more than really smart CSS with a database connection. Although I prefer Python or even C because I think I'd be more useful, I wouldn't be very upset at all if it turned out you guys were willing to let me learn PHP on your time. I like the <math> parsing project because it seems well-suited for a third-year undergrad who knows LaTeX and reads a few other functional languages and has studied lex/yacc before in his coursework. The goals are clear, and I know how to break them down into smaller problems and how to tackle each one. It's a little isolated from the rest of Mediawiki, so I don't need to grok the entire code base. Basically, this looks like a way to make a concrete contribution despite being a newcomer to the project. That doesn't mean I'm not happy to entertain alternatives, just that they have a pretty high bar to clear. The second. I just want to tackle a problem that's near but not quite beyond my limits, and if I can help out a site I use daily, so much the better. Yours, Damon Wang

What visibility does this have? I thought it was some abandoned corner of the wiki that nobody has touched in the seven years since it was first written. What happens if I make a hash of this? Well, now my inner voice says, "I really don't want to make a hash of this texvc port!", so let me explain why I want to do it in Python rather than PHP. I agree that the performance will probably be just fine, and that it would be a great coup for maintainability and installation and usage. The problem is, I don't think PHP has a parser-generator package. So let me make sure I understand the problem here. You already have a texvc implementation that has worked just fine for the last seven years. TeX is pretty stable at this point, so chances are good you'd make it another seven years without problems. But you're still dissatisfied because OCaml is a hard language to find programmers for, and the existing implementation isn't really maintained. You want it ported to a different language that has more programmers available. (You also as a Mediawiki extension rather than a core feature; I'm going to do that, but I won't say anything more because it seems fairly uncontroversial.) Since the subset of TeX you need parsed has a context-free grammar, it needs an LALR parser, not just a bunch of regexes. I know three ways to get an LALR parser: (1) write a pushdown automaton manually (i.e., be yacc) (2) write input for a parser-generator (3) write a parser-generator, and give it input Option (2) is the most maintainable and feasible option, and it's precisely the one that cannot be done in PHP. As far as I know, PHP has no parser-generator package. (Please, please let me know if that's incorrect so I can stop embarrassing myself and get on with writing a GSoC proposal.) I could probably do (1), or some hackish kludge at half of it, by throwing custom control structures into a bucketload of regexes, but I don't think that's in the project's best interests. As has been pointed out, the OCaml implementation is really concise and elegant. A large fraction of that concision and elegance comes from not actually being a parser but rather only a context-free grammar written in a BNF-like syntax common to most parser-generators. I think it'd be easier to find a programmer who has worked with a parser-generator and can learn a little bit of OCaml, than it would be to find a PHP programmer who has to read himself into a manually implemented parser. After all, how many PHP programmers do you know who have experience mucking around inside an LALR parser? So that's why, while I'm happy to take it on in PHP as a learning experience for myself, I think it'd be better for Mediawiki to port texvc to Python. That gets us the larger pool of potential maintainers that comes with using a commonly known language, without sacrificing the amazing advantage of only needing to maintain a grammar rather than the parser itself. And as far as dependencies are concerned, Python is still a much easier dependency to satisfy, both for programmers working with the code and for sysadmins installing it. What do you guys think? Also, would anyone be interested in mentoring this project? Yours, Damon Wang

On Fri, Mar 26, 2010 at 10:48 PM, Damon Wang &lt;damonwang(a)uchicago.edu&gt; wrote: I actually disagree with this pretty strongly. It would be a regression in functionality for existing users -- if they upgrade, their wiki breaks unless they install a new extension. There's no reason to remove it from core that I see that outweighs this disadvantage. Okay, well, maybe you're right. I'd be interested to hear Tim Starling's opinion on this (using parser generators vs. writing by hand). Writing it in Python would certainly be a big step forward from OCaml -- any site with LaTeX accessible to MediaWiki will almost certainly have Python available, so Python vs. PHP should make no difference to end-users. And Python is probably the second-best-known language among MediaWiki hackers. The parsing part is unlikely to need much maintenance. There are other things currently in OCaml that make more sense to modify from time to time -- like the whitelist of commands, and (some of?) the code for non-image output formats. So for instance, MathML output is theoretically supported, but I don't know how good the support is. That might become more important in the future, since Firefox is likely to support inline MathML in text/html not too long from now. This sort of thing would be harder if it were Python rather than PHP. I don't think it would be a big deal if it were rewritten entirely in Python, though. It would be a big step forward in any case, and if it's easier for you, great. So personally I'd be okay with it, although it's perhaps not ideal. I probably wouldn't be of any help for this particular project, since I don't know anything about parsers, and my Python and TeX are passable but not great. We could probably come up with a mentor, though.

"Aryeh Gregor" &lt;Simetrical+wikilist(a)gmail.com&gt; wrote in message news:7c2a12e21003281059i551c4650p8a8e51e100b62479@mail.gmail.com... As opposed to their wiki breaking when they upgrade for all the other reasons that we document in the release notes? I have never built a wiki where texvc has been needed, wanted, or even thought harmless. Currently MW users have to compile and configure a binary from a language 99.99% of them cannot understand, and enable the functionality using config variables. Asking them instead to download and install an extension like every other non-ubiquitous feature in MediaWiki is far from being a regression. --HM

Can we make update.php ask the user if he wants to install the new extension? Is there any place we could get usage statistics for the math feature? I think the advantages for new installations justify inconveniencing some existing users, especially if we can automate installation of the new extension, but this discussion would be better with some data. Yours, Damon Wang

On Mon, Mar 29, 2010 at 12:12 PM, Aryeh Gregor &lt;Simetrical+wikilist(a)gmail.com&gt; wrote: What if it was written as an extension and moved to /extensions? Then we get the benefit of decoupling Math from the core software, but we don't require users to download a new extension to keep their existing functionality. As long as it's clearly indicated in the RELEASE-NOTES and relevant Manual pages that you might need to update the path to Math, I don't see a huge drawback. -Chad

Aryeh Gregor wrote: <math> was implemented directly in the parser a really long time ago. That's the reason it has always been in core, despite it not being available for 99% users. Only recently, I freed the 'math' from the parser (r57997), so it can be used by another tag hook to provide the same functionality (I gave up on math and wanted to use [[Extension:Mimetex_alternative]]) and then Tim moved it out creating CoreTagHooks (r61913). It logically is an extension, much more than eg. ParserFunctions. I bet there are more users of ParserFunctions than of math. Changing to python will also break for people that compiled math, update without reading the release notes and don't have python. We can as well move it to a separate extension instead of embedding it, although that wouldn't be much an issue. The only way to ensure it will work for everyone would be to provide a PHP implementation, in which case thousands of installs will get a toolbar button suddenly working.

* Bryan Tong Minh &lt;bryan.tongminh(a)gmail.com&gt; [Tue, 30 Mar 2010 17:22:09 +0200]: Fedora Linux has ocaml for ages. yum install ocaml or something like that. Compiling texvc is fast and easy - never had any problems. Since ocaml was developed in France, chances are bigger that it has wider spread over there. Dmitriy

On 29 March 2010 01:12, Aryeh Gregor &lt;Simetrical+wikilist(a)gmail.com&gt; wrote: I'd *like* to use it, but it's such an arse I've never yet got it working ... perhaps it's just me. - d.

On 23 March 2010 19:17, Rob Lanphier &lt;robla(a)robla.net&gt; wrote: Getting it off Ocaml is an excellent first step. I have tried and failed to get texvc working properly in MediaWiki myself more than a few times, because of Ocaml not wanting to play nice ... - d.

On Tue, Mar 30, 2010 at 14:34, Victor &lt;bobbie(a)ua.fm&gt; wrote: c is "better", so is Common Lisp, Scheme, Haskell, Clojure or a number of other languages. The problem is that worse is better. OCaml isn't widely known among programmers or as easy for PHP programmers to get into as say Perl, Python or Ruby. As a result the math/ directory has been untouched (aside from the stray doc+bug fix) since 2003. There are many long standing core issues with the texvc component in Bugzilla (http://bit.ly/bsSUPM) that noone is looking at. I don't think anyone would have a problem with it remaining in OCaml if it was being maintained and these long-standing bugs were being fixed.

Ævar Arnfjörð Bjarmason wrote: No it shouldn't! :)

On 03/31/2010 12:31 PM, Victor wrote: While I'll no doubt regret saying this, I am happy to fix some of these bugs. With the majority of these it's harder to decide "should we fix" and "how should we fix" rather than actually being hard to implement. What I don't want to do is fix things to find that it then gets immediately re-implemented in PHP, which seems to be what people want. Some of the issues have LaTeX dependencies, particularly "support Unicode", so fixing them could be dangerous unless we conditionally include support for them, i.e. by running a feature-test as part of the installation (we should probably do that anyway, so that we can provide nicer error messages to the user if they are missing one of the other dependencies). Conrad

On 30 March 2010 16:34, Victor &lt;bobbie(a)ua.fm&gt; wrote: .... Doing Math in any programming language or digital computer is a bad idea. Anyway. I could be worse, It could be Math in Javascript: v = (011 + "1" + 0.1)/3; 303.3666666666667 -- -- ℱin del ℳensaje.

Somebody ought to slap you for mixing four different types in such a horrendous manner to construct something that is supposed to make people who do not understand octal numbers and string concatenation to think JavaScript is insane. *shakes head*

Why such a contrived example?! This all boils down to new Boolean(false) == false returning true new Boolean(false) === false returning false The ?: operator is just not performing an implicit cast. The fact that the Boolean object is different from the Boolean primitive type is... ...well unfortunate. That's why it's use is as an Object is deprecated since JavaScript 1.3 You example gives the expected outcome if you call Boolean as a function: ([8, 32, 189].sort()[0]) === (Boolean(false) ? 189 : 8) So, can we drop this dreaded debate now?

On 30 March 2010 15:34, Victor &lt;bobbie(a)ua.fm&gt; wrote: Oh, I don't doubt that at all. It just didn't work for me :-) - d.

"Platonides" &lt;Platonides(a)gmail.com&gt; wrote in message news:hobfpi$4ud$1@dough.gmane.org... I took it to mean that he wanted to split the math parsing out as a **MediaWiki** extension, implementing <math> as a parser tag hook in the usual way. Which is definitely highly desirable. --HM