Exactly: The HTML to wikitext conversion is what makes Parsoid useful, and not only for VE. Thanks to Parsoid, ContentTranslation has a simple rich text editor with contenteditable (not a full VE, though this may change in the future). We are just starting to deploy it to production, but the users who tested in beta labs loved it. The Parsoid way of converting wikitext to HTML is useful, too, because it allows ContentTranslation to process the article that is being translated in a formal and expected way, understanding where are links, images, templates, timelines, references, etc., and adapting it automatically to the translated article. All of this is done with simple jQuery selectors and very little effort. בתאריך 20 בינו 2015 04:01, ‏"Matthew Flaschen" <mflaschen(a)wikimedia.org> כתב:

I believe Subbu will follow up with a more complete response, but I'll note that: 1) no plan survives first encounter with the enemy. Parsoid was going to be simpler than the PHP parser, Parsoid was going to be written in PHP, then C, then prototyped in JS for a later implementation in C, etc. It has varied over time as we learned more about the problem. It is currently written in node.js and probably is at least the same order of complexity as the existing PHP parser. It is, however, built on slightly more solid foundations, so its behavior is more regular than the PHP parser in many places -- although I've been submitting patches to the core parser where necessary to try to bring them closer together. (c.f. https://gerrit.wikimedia.org/r/180982 for the most recent of these.) And, of course, Parsoid emits well-formed HTML which can be round-tripped. In many cases Parsoid could be greatly simplified if we didn't have to maintain compatibility with various strange corner cases in the PHP parser. 2) Parsoid contains a partial implementation of the PHP expandtemplates module. It was decided (I think wisely) that we didn't really gain anything by trying to reimplement this on the Parsoid side, though, and it was better to use the existing PHP code via api.php. The alternative would be to basically reimplement quite a lot of mediawiki (lua embedding, the various parser functions extensions, etc) in node.js. This *could* be done -- there is no technical reason why it cannot -- but nobody thinks it's a good idea to spend time on right now. But the expandtemplates stuff basically works. As I said, it doesn't contain all the crazy extensions that we use on the main WMF sites, but it would be reasonable to turn it on for a smaller stock mediawiki instance. In that sense it *could* be a full replacement for the Parser. But note that even as a full parser replacement Parsoid depends on the PHP API in a large number of ways: imageinfo, siteinfo, language information, localized keywords for images, etc. The idea of "independence" is somewhat vague. --scott On Mon, Jan 19, 2015 at 11:58 PM, MZMcBride <z(a)mzmcbride.com> wrote: -- (http://cscott.net)

Some quick comments. As has already been alluded to, Parsoid does a couple different things. * It converts wikitext to html (in such a way that edits to the html can be serialized back to wikitext without introducing dirty diffs in the wikitext). * It converts html to wikitext (in such a way that edits to the wikitext preserves html semantics). There are caveats here in that Parsoid doesn't yet handle arbitrary html that you might throw at it, but insofaras the HTML conforms to the DOM spec [1], Parsoid should do a good job of serializing it to wikitext. This bidirectionality means that Parsoid can support clients that don't need to deal with wikitext directly knowing that Parsoid can go both ways. Amir has mentioned Content Translation. See the full list of clients here [3]. This support for bidirectional conversion between wikitext and HTML is non-trivial. See "Lossless conversion" section and other details in [2]. Getting to this stage where Parsoid is in terms of rendering and bidrectionality has required us to work through a lot of issues and edge cases given that editing requires HTML semantics and wikitext and transclusions is string-based. Parsoid can map a DOM node to a substring of wikitext that generated it, and that is also a non-trivial achievement. See the tech talk here [4]. I'm skipping the details of the different levels of testing that we implement to achieve this, but that has been a substantial part of getting to this part and being able to deploy seamlessly on a regular basis [5] largely without incident. As for the other part about preprocessing, yes, Parsoid currently relies on the Mediawiki API. The core parser has the following components: * preprocessing that expands transclusions, extensions (including Scribunto), parser functions, include directives, etc. to wikitext * wikitext parser that converts wikitext to html * Tidy that runs on the html produced by wikitext parser and fixes up malformed html Parsoid right now replaces the last two of the three components, but in a way that enables all of the functionality stated earlier. I'll skip the historical and technical reasons right now why we haven't put energy and resources into this component of Parsoid, but in brief, we found it more important to enable the bidirectional functionality and supporting clients and reuse the preprocessing functionality via the mediawiki API. But, there are several directions this can go from here (including implementing a preprocessor in Parsoid, for example). However, note that this discussion is not entirely about Parsoid but also about shared hosting support, mediawiki packaging, pure PHP mediawiki install, HTML-only wikis, etc. All those other decisions inform what Parsoid should focus on and how it evolves. Subbu. [1] http://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec [2] https://blog.wikimedia.org/2013/03/04/parsoid-how-wikipedia-catches-up-with… [3] http://www.mediawiki.org/wiki/Parsoid/Users [4] https://www.youtube.com/watch?v=Eb5Ri0xqEzk with slides @ https://commons.wikimedia.org/wiki/File:Parsoid.techtalk.apr15.2014.pdf [5] https://www.mediawiki.org/wiki/Parsoid/Deployments On 01/20/2015 08:02 AM, C. Scott Ananian wrote:

C. Scott Ananian wrote: Hrm. I guess this is the part that I'm still struggling with. If the PHP parser is/was already doing the job of converting to wikitext to HTML, why would that need to be rewritten in Node.js? Wouldn't it have been simpler to make the HTML output more verbose in the PHP parser so that it could cleanly round-trip? I'm still not clear where Node.js (or C or JavaScript) came into this. I heard there were performance concerns with the PHP parser. Was that the case? I'm mostly just curious... you can't un-milk the cow, as they say. Hrm. MZMcBride