On Tue, Aug 11, 2015 at 5:16 PM, Trevor Parscal <tparscal(a)wikimedia.org> wrote: It is possible to do this in Parsoid (or any node service) with this line: var sanerHTML = domino.createDocument(input).outerHTML; However, performance is about 2x worse than current tidy (116ms vs. 238ms for Obama), and about 4x slower than the fastest option in our tests. The task has a lot more benchmarks of various options. Gabriel -- Gabriel Wicke Principal Engineer, Wikimedia Foundation

On Tue, Aug 11, 2015 at 5:24 PM, Trevor Parscal <tparscal(a)wikimedia.org> wrote: Mainly a pure-JS DOM implementation (domino) not being quite the same speed as C or Rust with all optimizations turned on. The deltas are roughly in line with language benchmarks like http://benchmarksgame.alioth.debian.org/. Gabriel

Some years back I was importing a large number of complex templates to a wiki that didn't have tidy enabled. The results were nothing short of horrendous in a substantial number of cases. Wiki authors will generally stop worrying about their code as long as the results look right. For good or ill, tidy does a remarkable job of localizing unclosed tags, and often that is enough to effectively fix the appearance of broken HTML syntax so it doesn't spill over into other sections. Without Tidy (or its equivalent) there will be a lot of template garbage that needs to be repaired. The garbage in -> garbage out approach might seem appealing in principle, but any transition to such a condition is going to dredge up a lot of malformed HTML code created by wiki editors that we've been hiding for many years. If one is going to replace Tidy with something substantially different in execution, I would suggest that one needs a significant test suite of complex pages in order to judge how bad the collateral damage is likely to be, and ideally some set of tools to help editors fix it. -Robert Rohde On Thu, Aug 13, 2015 at 7:51 AM, Brian Wolff <bawolff(a)gmail.com> wrote:

On Saturday, August 15, 2015, MZMcBride <z(a)mzmcbride.com> wrote: I dont know about that. Viz editor is targeting ordinary tasks. Its the complex things that mess stuff up. missing Failing fast and loud is good in lots of contexts. I dont think wiki editing is one of them. Afaik, anchors are disallowed because spammers commonly insert them. Its trivial to sanitize and allow them if we so desired. -- bawolff

On 08/17/2015 10:15 PM, MZMcBride wrote: Wikitext is string-based and generates a html string and in the general case, it need not be well-formed HTML. There is a lot of broken wikitext out there and if you remove Tidy and don't introduce a HTML5 parser based balancer, you are going to see a lot of breakage. * Unclosed HTML tags (very common) * Misnested tags * Misnesting of tags (ex: links in links .. [http://foo.bar this is a [[foobar]] company]) * Fostered content in tables (<table>this-content-will-show-up-outside-the-table<tr><td>....</td></tr></table>) ... this has been one of the biggest source of complexity inside Parsoid ... in combination with templates, this is nasty. * Other ways in which HTML5 content model might be violated. (ex: <small>\n*a\n*b\n</small>) * Look at the parser tests file and see all the tests we've added with annotations that say "php parser relies on tidy" [[ Tangent: We have a linting option in Parsoid that we can turn on in production that can dump information about all these broken forms of wikitext (we have this information because we have to break the wikitext in the same ways when we convert html to wikitext). We haven't turned it on in production yet because we haven't yet had the time to hook this into project wikicheck .. we had initial conversations, but we couldn't follow up on our end. ]] Besides these, there is also other unrelated-to-html5-semantics behavior that wikis have come to rely on. * Stripping of empty tags -- correct page rendering rely on the fact that Tidy strips empty elements from HTML. We had to explicitly add this behavior to Parsoid so pages render identically. We could rip this out as long as all those templates are fixed up. The infobox on itwiki:Luna relies on this, to give you a specific example. * Some behaviors found in https://phabricator.wikimedia.org/T4542 * I am sure there are a bunch of other behaviors that I am missing / don't know about. So, you cannot just rip out Tidy and not replace it with something in its place. Even replacing it with a HTML5 parser (as per the current plan) is not entirely straightforward simply because of all the other unrelated-to-html5-semantics behavior. Part of the task of replacing Tidy is to figure out all the ways those pages might break and the best way to handle that breakage. Going forward, we are thinking about how to enforce stricter constraints on what templates (and extensions) can produce so impacts from broken wikitext is contained. That will give you some of what you are asking ("fail fast", but in a different form). That requires a functioning html5 treebuilder / parser to be in place which is what this RFC is about. Subbu.

On 08/18/2015 07:58 AM, MZMcBride wrote: Are you suggesting that you get rid of wikitext editing? If not, you cannot assume editors are going to write perfect markup. What is needed is a way to define DOM scopes in wikitext and enforce well-formedness within scopes. So, for example, template output can be considered a DOM scope (either opt-in or opt-out). If we felt bold, we can define a list to be a DOM scope .. or a table to be a DOM scope ... or a image caption to be a DOM scope, and so on. Rather than expect editors to write perfect markup, we should be thinking about sane semantics for them like scoping that delimit effects of broken markup. With proper semantics, it is easier to reason about markup and not rely on whimsical behavior of whatever tool we used yesterday or use today or might use tomorrow. We are working towards these kind of scoping semantics and the first step on the way is to get a HTML5 treebuilder / parser in place. Subbu.

On 18 August 2015 at 04:15, MZMcBride &lt;z(a)mzmcbride.com&gt; wrote: Or when it turns a complex task into a simple one, e.g. table editing (one click to remove a column). - d.

On Tue, Aug 18, 2015 at 11:48 PM, Derk-Jan Hartman < d.j.hartman+wmf_ml(a)gmail.com&gt; wrote: In my experience, mismatched tags are quite often used on purpose. For example, Cyberpower678 has two unmatched div tags at the end of his StandardLayout template <https://en.wikipedia.org/wiki/User:Cyberpower678/StandardLayout>, used to put a shaded border round the posts on his talk page <https://en.wikipedia.org/wiki/User_talk:Cyberpower678>. There are no corresponding closing div tags at the end of the talk page, as they would be moved by the talk page archive bot, and Tidy takes care of the invalid HTML anyway.

On 13/08/15 15:43, MZMcBride wrote: The HTML 5 spec makes a distinction between valid, balanced HTML and error recovery algorithms. Browsers are basically the only clients able to handle moderately bad HTML, and as I've previously said in discussions of HTML 5 output, I don't think it is acceptable to screw over all non-browser clients by sending output that relies on obscure details of the HTML 5 spec. I think XHTML or something close to it is an appropriate machine-readable output format. Have you looked at my survey on the bug? Compliant HTML 5 parsers are 10-30k source lines and are in pretty short supply. Wikitext is not meant to be easily machine-readable, it is meant to be easily human-writable. Unbalanced tags in HTML are errors, but in wikitext they are allowed. This is a design choice. Most humans don't really care about the spec, they just want the machine to figure out what they meant. And, as several others have noted, you can't just disable Tidy, since the effects of unclosed tags are not confined to the content area, and there is a large amount of existing content that depends on it. I have seen the effects of Tidy being accidentally disabled on the English Wikipedia, it is not pleasant. Am I correct in saying that MZMcBride is the only person in this thread in favour of the idea of getting rid of HTML cleanup? By the way, you can see my work in progress on an HTML reserializer web service in the mediawiki/services/html5depurate project on Gerrit: <https://gerrit.wikimedia.org/r/#/q/status:open+project:mediawiki/services/html5depurate+branch:master,n,z> -- Tim Starling

On Wed, Aug 19, 2015 at 1:22 PM, MZMcBride &lt;z(a)mzmcbride.com&gt; wrote: My $0.02 from the peanut gallery: If we fixed up the bulk of the most common cases we can (where the bad HTML is not the result of an edit error), could we keep a Tidy/HTML5 type of thing around, but move it to edit validation rather than render output processing? We could start by leaving the current output-side code alone, and warning (to the user as a minor info blurb on edit submission, and in our logs) about edits that fail validation, so that we can get some idea of the scope and causes of the problem, fix what we can, and then evaluate whether we can eventually start flat-out rejecting the minority of edits that fail validation and then eventually remove the tidy on the output side. That ignores the whole problem of existing bad html already in the DB, of course, but that could probably be fixed with a one-time job...

On 08/19/2015 08:22 AM, MZMcBride wrote: In response to both these queries, see this snippet from my earlier post on this thread ( https://lists.wikimedia.org/pipermail/wikitech-l/2015-August/082806.html ) "Even replacing it with a HTML5 parser (as per the current plan) is not entirely straightforward simply because of all the other unrelated-to-html5-semantics behavior. Part of the task of replacing Tidy is to figure out all the ways those pages might break and the best way to handle that breakage." Also see https://phabricator.wikimedia.org/T89331#1499979 about how we might go about evaluating this. So, we aren't saying we'll implement those Tidy behaviors here. Part of the solution might very well be to break some of that Tidy behavior and have the pages be fixed up (bots, manually, however). In any case, the first step is to understand those impacts. Subbu.

On 20/08/15 01:21, Erwin Dokter wrote: That's not a solution to the problems that we are trying to solve. As I said in my original post, my number one problem with Tidy is that it changes. So I am very happy that it is not in active development. Switching to a fork that is actively maintained would be much worse. It would be like the switch from Tidy to the proposed HTML reserializer web service, except that the pain would be repeated every time we upgrade our Linux distribution. The other problem with Tidy is that it is poorly specified and has only one implementation. Switching to a fork of it doesn't improve the situation. HTML 5 has not significantly relaxed the rules about block elements inside inline elements. The terminology has changed: now instead of inline elements we have "phrasing content" and instead of block elements we have "flow content". You're still not allowed to put a <div> inside a <span>, because <span> is phrasing content and <div> isn't. The "children" column here has a summary: http://www.w3.org/TR/html5/index.html#elements-1 -- Tim Starling