Στις 23-01-2013, ημέρα Τετ, και ώρα 15:10 -0800, ο/η Gabriel Wicke έγραψε: On thing that jumped out at me is this: "We have also decided to narrow our focus a bit by continuing to use the PHP preprocessor to perform our template expansion." While I understand the reasoning and even sympathize with it, I had hoped that Parsoid, when complete, would facilitate the implementation of wikitext parsers apart from the canonical parser (i.e. MediaWiki), with clearly defined behavior for the language including templates. Is that idea dead then? Ariel

On 01/30/2013 12:36 AM, Ariel T. Glenn wrote: As it exists, Parsoid can tackle full template expansion -- but, since it does not support all parser functions natively, this is still incomplete, and we can bypass the need for the most part by relying on the PHP preprocessor to give us fully expanded wikitext which we process further. We are refocusing our efforts towards exploring HTML-based templating -- while supporting existing templates. Lua based templates already clean up a lot of template logic by have access to full conditional logic. By relying more on DOM-based templates (which would also be editable in a visual-editor like client), the expectation is that direct wikitext use itself will progressively diminish. Since most wikitext and probably Lua templates already return well-formed DOM (not all do), by simply adding a parse layer on top of them, they can be supported in a DOM-only templating framework. So, the first outcome of this effort would be to require templates to return DOM fragment always. In such a diminished-use scenario, we do not see the need for focusing a lot of energy and effort in attaining full compatibility entirely in Parsoid. We see Parsoid+PHP parser as providing legacy wikitext support while a large chunk of editing and storage happens in the HTML world. We can then take it from there based on how far this strategy takes us. If there still remains a need for a full replacement wikitext evaluation system to be in place (because of continuing popularity of wikitext or because of performance reasons or whatever else), that option remains open and is not closed at this time. Even so, there is still possibility of identifying "erroneous" or "undefined behavior" wikitext markup within Parsoid (in quotes, because anything that is thrown at the php parser and parsoid needs to be rendered always). We can detect, for example, missing opening/closing html tags (since we currently have to do that for roundtripping them properly without introducing dirty diffs), detect unbalanced tags in certain contexts by treating them as balanced-DOM contexts (image captions, extensions), and other such scenarios. We also have been adding a number of parser tests that try to specify edge case behaviors and make a call as to whether it is legitimate behavior or undefined behavior. All of this could be used in some mode to issue warnings in some lint-like mode, which can then serve to be a de-facto definition of legitimate wikitext since there is no possibility of a grammar-based definition for wikitext. So, while we are not focusing on attaining full replacement capability in Parsoid, our new directions do not entirely do away with the idea that you alluded to: (1) we are attempting to move towards templates (DOM/Lua/wikitext) that can only return DOM fragments (2) we retain the ability to provide some kind of linting ability in Parsoid (but this functionality is not at the top of our todo list since we are focused on reducing the scope of wikitext use over the long-term, while providing full compatibility in the immediate and short-term). Does that answer your question? Subbu. PS: The other primary reason for going with a new wikitext evaluator/runtime (more accurate than calling this a parser), possibly in c++, was performance -- but we are going at it in a different way already based on the notion that most edits on wiki pages are going to be "minor" edits (relative to the size of the page). If so, there is no sense in fully serializing it and fully reparsing it on every such minor edit -- it is a waste of server resources. Since we now have a fully RT-able HTML representation of wikitext, selective serialization (HTML->wikitext) selective reparsing (of wikitext-based edits that happen outside the VE), along with caching of DOM fragments (transclusions, etc) should take care of the performance issue -- these are addressed in the RFC.

Hey Aaron, thanks for your thoughts! You evidently kicked off the discussion ;) Awesome! Indeed. We have hard deadlines for the features needed by the VisualEditor, so the architectural work might be slowed down a bit if that gets tight. Conversion to HTML on save and HTML storage are important for user-perceived editing performance though, so it is fairly high priority. We will definitely do some statistics on this, and will discuss the storage strategy before starting implementation. Right now we are still researching the implementation options, should have more clue next week. The ContentHandler angle is something I have also been wondering about. For pure HTML wikis this should work as designed, with a single (HTML/RDFa) content model assigned per revision. For mixed wikis storing both HTML and wikitext however we need to support different content models (wikitext and HTML/RDFa) for each revision. Those two are isomorphic, but are handled differently. If there is interest in supporting multiple content models per revision within the ContentHandler framework, then now would probably be a good time to work that out. In any case, it seems to be a good idea to use the existing text storage logic in revision including its support for compression and external storage. Currently Parsoid converts each edited HTML document to wikitext, and then re-parses that wikitext while sanitizing attributes and tags with a port of the PHP Sanitizer class. Before we can store the HTML DOM edited by a client directly, we will need to rework sanitation to work on the DOM, and preferably also perform as much of the work on the way in instead of on the way out. Metadata embedded in the DOM beyond regular HTML can be divided into two categories: Public RDFa-based structures and private round-trip data. Public RDFa structures will need more solid verification, but are otherwise pretty straightforward (see the spec at [1]). We plan to move private round-trip data out of the DOM, which would prevent clients from messing with it. We will probably use some unique id attributes to aid the association of nodes with their metadata, but might also be able to get away without such ids by using a subtree hashing similar to the one described in XyDiff [2]. XyDiff would also be an improvement over the simplistic DOM diff algorithm we currently use for change detection. Gabriel [1]: http://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec [2]: http://gregory.cobena.free.fr/www/Publications/%5BICDE2002%5D%20XyDiff%20-%… -- Gabriel Wicke Senior Software Engineer Wikimedia Foundation