>> import mwparserfromhell >> text = "I has a template! {{foo|bar|baz|eggs=spam}} See it?" >> wikicode = mwparserfromhell.parse(text) >> templates = wikicode.filter_templates()

Il 24/07/2015 06:35, C. Scott Ananian ha scritto: Parsoid's expressiveness seems to convey useless information, overlook important details, or duplicate them in different places. If I want to resize an image, am I supposed to change "data-file-width" and "data-file-height"? "width" and "height"? Or "src"? I think what I'm looking for is sort of an 'enhanced wikitext' rather than 'annotated HTML'. and that would probably be a great addition to the parsoid package API. All edits made via visualeditoredit < https://www.mediawiki.org/w/api.php?action=help&modules=visualeditoredi… are tagged. I take it you would like an easy work flow to Right.

mwparserfromhell doesn't actually seem to have that functionality It is actually pretty easy to do with Pywikibot. But since Parsoid happens to work server-side, it makes sense to request and send back the structured tree directly. , but it > would also be nice to facilitate that use case if we can. > --scott > ​ > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l Thanks for your time. _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

wrote:

What if it isn't?

There is also section API support (you can get individual HTML Then I guess HTML would have to be converted into wikitext before saving? +1 API call

On Fri, Jul 24, 2015 at 12:34 AM, Ricordisamoa &lt;ricordisamoa(a)openmailbox.org These are great points, and reports from folks like you will help to improve our documentation. My goal for Parsoid's DOM[1] is that every bit of information from the wikitext is represented exactly *once* in the result.

In your example, `data-file-width` and `data-file-height` represent the *unscaled* size of the *source* image. Many image scaling operations want to know this, so we include it in the DOM. It is ignored when you convert back to wikitext. The `width` and `height` attributes are what you should modify if you want to resize an image, just like you would do for any naive html editor.

The `src` attribute is again mostly ignored (sigh); the 'resource' attribute specifies the url of the unscaled image. Of course if 'resource' is missing we'll try to make do with `src`; we really try hard to do something reasonable with whatever we're given. --scott [1] There is a tension between "don't repeat yourself" and the use of Parsoid DOM for read views. Certain attributes (like "alt" and "title") get duplicated by default by the PHP parser. So far I think we've been mostly successful in not letting this sort of thing infect the Parsoid DOM, but there may be corner cases we accomodate for the sake of ease-of-use for viewers.

HTML5+RDFa is a machine-readable format. But I think what you are asking for is either better documentation of the template-related stuff (did you read through the slides inhttps://phabricator.wikimedia.org/T105175 ?) or HTML template parameter support (https://phabricator.wikimedia.org/T52587) which is in the codebase but not enabled by default in production. --scott ​ _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Stephen Niedzielski: "it seems like, as soon as you get the HTML the first thing you want to do, perhaps a little bit ironically because it's called Parsoid, it's parse the output a little bit more" https://www.youtube.com/watch?v=3WJID_WC7BQ&t=35m14s

On 07/23/2015 01:07 PM, Ricordisamoa wrote: Parsoid and mwparserfromhell have different design goals and hence do things differently. Parsoid is meant to support HTML editing and hence provides semantic information as annotations over the HTML document. It effectively maintains a bidirectional/reversible mapping between segments of wikitext and DOM trees. You can manipulate the DOM trees and get back wikitext that represents the edited tree. As for useless information and duplicate information -- I think if you looked at the Parsoid DOM spec [1], you will know what to look for and what to manipulate. The information on the DOM is meant to (a) render accurately (b) support the various bots / clients / gadgets that look for specific kinds of information, and (b) be editable easily. If that spec has holes or needs updates or fixing, we are happy to do that. Do let us know. mwparserfromhell is an entirely wikitext-centric library as far as I can tell. It is meant to manipulate wikitext directly. It is a neat library which provides a lot of utilities and makes it easy to do wikitext transformations. It doesn't know about or care about HTML because it doesn't need to. It also seems to effectively gives you some kind of wikitext-centric AST. These are all impressions based on a quick scan of its docs -- so pardon any misunderstandings. Parsoid does not provide you a wikitext AST directly since it doesn't construct one. All wikitext information shows up indirectly as DOM annotations (either attributes or JSON information in attributes). As Scott showed, you can still do document ("wikitext") manipulations using DOM libraries, CSS-style queries, or directly by walking the DOM. There are lots of ways you can edit mediawiki pages without knowing about wikitext and using the vast array of HTML libraries. That happens to be our tagline: "we deal with wikitext so you don't have to". But, you are right. It can indeed seem cumbersome if you want to directly manipulate wikitext without the DOM getting in between or having to deal with DOM libraries. But that is not the use case we target. There are a vastly greater number of libraries in all kinds of languages (and developers) that know about HTML and can render, handle, and manipulate HTML easily than know how to (or want to) manipulate wikitext programmatically. Kind of the difference between the wikitext editor and the visual editor. They each have their constituencies and roles. All that said, as Scott noted, it is possible to develop a mwparserfromhell like layer on top of the Parsoid DOM annotations if you want a wikitext-centric view (as opposed to a DOM-centric view that most editing clients seem to want). But, since that is not a use case that we target, that hasn't been on our radar. If someone does want to take that on, and thinks it would be useful, we are happy to provide assistance. It should not be too difficult. Does that help summarize this issue and clarify the differences and approaches of these two tools? I am "on vacation" :-) so responses will be delayed. Subbu. [1] http://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec

I agree that we have not (to date) spent a lot of time on APIs supporting direct editing of the Parsoid DOM. I tend to do things directly using the low-level DOM methods myself (and that's how I presented my Parsoid tutorial at wikimania this year) but I can see the attractiveness of the `mwparserfromhell` API in abstracting some of the details of the representation. Thankfully you can have it both ways! Over the past week I've cloned the `mwparserfromhell` API, build on top of the Parsoid DOM. The initial patches have been merged, but there's a little work to do to get the API docs up on docs.wikimedia.org properly. Once that's done I'll post here with pointers.

Eventually I'd like to put the pieces together and implement something like a `pywikibot` clone based on this API and using the RESTBase APIs for read/write access to the wiki. As has been mentioned, the RESTBase API for saving edits is not yet quite complete ( https://phabricator.wikimedia.org/T101501); once that is done there should be no problem connecting the dots. (In the meantime you can use the API I just implemented to reserialize the wikitext and then use the standard PHP APIs, but that's a little bit clunky.) --scott ​ _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

On Sat, Aug 1, 2015 at 2:23 AM, Ricordisamoa&lt;ricordisamoa(a)openmailbox.org&gt; wrote: Clearly you're just trying to bait me into porting my code to python.

I assure you there is nothing JavaScript-specific about this; there are HTML DOM-manipulation libraries available in all major programming languages. HTML *is* an AST (in this case, at least). --scott

On 14 aug. 2015, at 00:20, C. Scott Ananian &lt;cananian(a)wikimedia.org&gt; wrote: Good news: https://doc.wikimedia.org/Parsoid/master/#!/guide/jsapi now documents the new friendlier API for Parsoid. --scott On Tue, Aug 11, 2015 at 3:03 PM, Ricordisamoa &lt;ricordisamoa(a)openmailbox.org&gt; wrote: -- (http://cscott.net) _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Hi Subbu, thank you for this thoughtful insight.

HTML is not a barrier by itself. The problem seems to be Parsoid being built primarily with VisualEditor in mind.

It is not clear to me how can a single DOM serving both view and edit modes avoid redundancy.

I see huge demand for alternative wikignome-style editors. The more Parsoid's DOM is predictable, concise and documented, the more users you get.

I hope we can meet in the middle :-)

You are right that there are some redundancies in information https://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec#Transclusion_cont… All template parameters are in data-mw but not parsed. Parameters ending up in the 'final' wikitext are parsed separately.

I see huge demand for alternative wikignome-style editors. The more For example, to find images I have to search elements where typeof is one of mw:Image, mw:Image/Thumb, mw:Image/Frame, mw:Image/Frameless, then see if it's a figure or a span, and expect either a <figcaption> or data-mw accordingly. Add that the img tag's parent can be <a> or <span>... Instead, this is what I'd expect a proper structure to look like: