I'm going to take the bait and respond in part, to defend the teams and projects I work with:

On Tue, Nov 8, 2016 at 2:47 PM, Rogol Domedonfors domedonfors@gmail.com wrote:

they are summarised by the four words *under-ambitious, under-resourced, under-managed and under-performing*. The VE/Parsoid/Flow complex suffers from scope mismatch. As a vehicle for delivering a WYSIWYG editor and discussion board it is over-complex,

I'll stop here. I think it is poorly understood in the community how complex wikitext markup has been allowed to grow over the decades it has been under development. There *is no specification for wikitext*. We have informal guides which omit most of the interesting corner cases, like, say, priority between conflicting markup. Take a look at http://spec.commonmark.org/ to see what a precise specification for a *much simpler* markup language would look like. As you read through the cases in that spec, consider that if you translated most of the examples into wikitext, *literally no one knows what the expected output would be*. The implementation is too organic, with bits glommed on ad-hoc, and we'd probably have to run them through the parser to find out what happens: there is no way for a human to reason about it.

So, yes: VE/Parsoid are very complex (I'll leave Flow for a little further down, be patient). Part of this was a learning process: when I joined the project in 2013 there was a perhaps-naïve sense that we could easily implement a sort of "cleaned up wikitext" and migrate any markup that broke. Over the intervening years (and in response to heated criticism of Visual Editor when the "cleaned up wikitext" failed to exactly match editors' expectations), Parsoid has grown and grown, until it has almost as many barnacles as the core PHP parser.

This is not only a challenge for engineering. This is a challenge to the community. In order to actually achieve simplicity in our parser and editor we will need to make aggressive changes to our content markup in order to reverse years (decades) of poorly-understood workarounds to buggy or underspecified wikitext syntax. Yes, you can fault WMF management (and I'll personally take some blame) for not aggressively pushing the community into this action, but the community in its turn has been strongly resistant to proposed changes in wikitext, which they characterize as "for the sake of Visual Editor", not fully realizing the extent to which the obscurity of the current wikitext syntax imperils the editability, archivability and future usability of the content we are creating.

We need to tackle this challenge together. Only when we commit to aggressively updating the wikitext used in our projects, will we be able to drop the "unnecessary" complexity of the parser-editor implementation. There are a number of different proposals for "wikitext 2.0", more than one of which I expect will be the subject of sessions at the developer summit this year ( https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Handling_wiki...). But again: the engineers can't do it alone. It's not just an engineering problem. We *started* with the engineering solution (simple parsoid/VE with no compatibility barnacles) and had to spend the next three years in further development because of social/human factors --- namely, every time that VE or Parsoid "broke" something the solution was to complicate the parser and editor, not to fix the wikitext. If we want to scale down ongoing development, we need to tackle the community side of the larger issue of wikitext maintenance and migration.

I think you actually agree: you finish your open letter with, "Your staff need to engage with users as equals, not de haut en bas. The community is here to work with you if you will let it." But this goes both ways -- as a community we need to stop treating our existing content as sacrosanct, and recognize that the community needs to accept change if they want engineering to finish projects and move the platform forward.

while VE and Flow are under-ambitious. Can these really be regarded as cutting edge in 2016?

Agreed. We have collaborative features being rolled out in VE now, but that just brings us to the state of the art circa 2005/6, when Google Docs (nee Writely) was written (or ~2009, when AbiWord gained its collaborative features in version 2.8.0). Yes, we're late. But the actual implementation of a collaborative editor has always been the "easy part" (see https://www.mediawiki.org/wiki/Extension:TogetherJS for example). The hard parts were:

1. Always generating exactly the wikitext a human editor would generate for a given edit (which varies by projects).

2. Open social collaboration among tens of millions of users on WMF projects (~30 million on enwiki alone), while respecting privacy and preventing harassment.

I discussed #1 above. #2 is still an open problem -- not just for us, but for the "state of the art". If you are aware of a similar WMF-scale collaborative editor, do let me know! In particular, balancing openness, privacy, anti-harassment and anti-vandalism is a hard problem. We are working on this, but this will also require buy-in from the community. We have to embrace, together, a real-time UX for Wikipedia without reflexively pushing back just because "it's not how editing on Wikipedia used to be". We probably need to be open to trying things, learning what works and what doesn't, and then trying other things -- without blaming WMF for the shortcomings of the first attempt. This was a proposed topic for the dev summit, has some proposed sessions as well ( https://phabricator.wikimedia.org/T149661 and https://phabricator.wikimedia.org/T149663), and I've been leading sessions at Wikimania on this since 2014 to encourage discussion in the community (see https://en.wikipedia.org/w/index.php?title=User:Cscott&action=edit#Wikim... ).

Perhaps this is actually your point: we need the community and engineering to work together. We need the community to embrace change, enabling engineering to implement that change.

If correctly scoped they could and should have been delivered and finalised long ago, if not brought in from pre-existing open source projects.

Again, if we just adopted (say) markdown or pure-HTML for our content, we could have used one of the existing editors. But instead the community wanted something which looked just like traditional wikitext. If there's a failure in vision there, it cuts both ways.

The execution of the VE project has been lacklustre. Such a straightforward product should have been finished long since, and it was clearly inadequately resourced and managed.

Again, there's has been a *lot* of background work to make a project "look" straightforward. Inadequately resourced I could agree with -- there was a WMF reorg under the previous ED that reassigned most of VE's engineers. It still doesn't really have enough resources to tackle an aggressive project like collaborative editing; especially considering that "adequate" resourcing would probably involve all parts of engineering to develop an integrated UX for collaboration, not just resources in the editing team. Calls to cut resources and "finish" VE don't help.

The current culture appears not

to pritorise such issues as timeliness and grip.

I'd say that the current culture rewards short-term projects with immediately realizable and quantifiable goals that are highly likely to be successful. That's not a bad thing, per se, but you seem to be arguing for the adoption of larger scale projects with "vision". WMF has become scared of large scale projects after the community rejection of Visual Editor, Media Viewer, and Flow. It has interpreted the community's reaction to these projects as a directive to "think smaller" and concentrate on simple reactions to user bug reports.

This is what led to VE/Parsoid spending years tweaking the support of corner case legacy constructs and trying to generate output that looks more and more like traditional wikitext, instead of "finishing" the project. If you want projects to end cleanly, the community has to agree one a point at which things are "done enough" and that future issues should be accommodated by content changes or behavior changes, not by endless refinements to the code.

Finally, Workflow has already failed. In the absence of resources to undertake the required research into the huge complexity of work flows within the projects, whatever is designed will be designed in ignorance, and hence simply can not succeed. Ever. It is already a waste of time.

I think you are suggesting that we devote the resources required to properly understand the workflow problem and adequately solve the problem?

What we are currently doing is actually what you indicate: we have stopped development on Flow and given up on the general workflow problem, instead looking at small tools for specific work flows -- like the addition of ORES scores to various review processes. We are not currently working on the general workflow problem.

In terms of the wider project, the view of an editing and rendering engine as handling only unidirectional linear text fails to capture even the richness of current projects let alone future knowledge modalities, such as complex text (hieroglyphics, chemistry, mathematics, music), higher-dimensional data (genomic, proteomic, 3D printing), data in time (audio, video), interactive, computational, ... all of which could and should have been scoped out by your innovation work.

I'm not sure why you think that VE/Parsoid only supports unidirectional linear text? Our RTL support is actually quite robust. And VE supports WikiHiero -- see for example https://www.mediawiki.org/wiki/VisualEditor/Basic_example_worksheet#WikiHier.... Further extensions are limited mostly by the community's willingness to write support: the infrastructure to add extensions to Visual Editor and Parsoid are both present.

I do think we should spend time thinking about non-text content.

Hence https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Handling_wiki... , https://phabricator.wikimedia.org/T149554, and https://phabricator.wikimedia.org/T90914#2759122 for instance. I think the general rule of Wikimedia Incubator projects holds -- if the community generated really compelling content in new media, WMF engineering would be increasingly motivated to provide tools to support that new media. It's a bit of a chicken-and-egg problem. We tried to envision what "cool content" might look like at the last Wikimania (see "Integrated video" and "Integrated VR/panorama" at https://en.wikipedia.org/wiki/User:Cscott/Ideas), but I think engineering is rightly reluctant to spend a lot of resources developing these sorts of features w/o content requiring them.

To conclude, I would like to echo your call for increased collaboration between community and engineering. But in addition to the "engineering should build what the community wants" direction, we need to also build the "community should embrace changes arising from WMF efforts to evolve the platform" direction. The necessary balance and interplay between these factors has been missing, and that has resulted in an engineering department which is too under-resourced to actually execute a vision for the future. And in the absence of a vision for the future, the community has called for engineering to scale down (further) and "focus".

This is not healthy for the future of the project. --scott