On Mon, Jul 22, 2013 at 8:44 PM, Tim Starling tstarling@wikimedia.org wrote:
and the results from Aaron Halfaker's study [2]
As noted at the top of the page, the analysis is still in progress.
Importantly, there were many confounding variables in the test, some of which are already documented. This includes users being assigned to the test group that received VisualEditor whose browser did not properly support it (it would have literally just not worked if the user attempted to edit); these issues were fixed later. See https://meta.wikimedia.org/wiki/Research:VisualEditor%27s_effect_on_newly_re... for some of these issues, but like I said, analysis is still in progress and we'll need to see what conclusions can actually be drawn from the data.
A proponent of source editing would claim that the steep learning curve is justified by the end results. A visual editor is easier for new users, but perhaps less convenient for power users. So Aaron Halfaker's study took its measurements at the point in the learning curve where you would expect the benefit of VE to be most clear: the first edit.
Actually, as noted in the draft, because the test group was assigned at the point of account creation, we're not taking into account any prior experience using wikitext as an IP editor. 59% of respondents in the 2011 editor survey stated that they had edited as IPs prior to making an account, so we should assume that this is not an insignificant proportion: https://meta.wikimedia.org/wiki/Research:Wikipedia_Editors_Survey_November_2...
Round-trip bugs
If you have, like I have, spent hours looking at VisualEditor diffs, you'll know that these are relatively rare at this point. The bug category of "round-trip bugs" is sometimes used for issues that aren't accurately be described this way, e.g. users typing wikitext into VisualEditor, having difficulty updating a template parameter, or accidentally deleting content (sometimes due to bugs in VE).
Perhaps the main problem is performance. Perhaps new users are especially likely to quit on the first edit because they don't want to wait 25-30 seconds for the interface to load (the time reported in [3]). Performance is a very common complaint for established users also.
You're quoting a user test from June 10 which was performed on the following page, which I've temporarily undeleted:
https://www.mediawiki.org/wiki/Golden-Crowned_Sparrow
Editing this page in Firefox on a 6-year-old system only slightly faster than the tester's specs today takes about 5 seconds to initialize. In Chrome it takes about 3 seconds, in the ballpark of reloading the page into the source editor. Note that Gabriel put major caching improvements into production around June 7, which may not have been in effect for this user / this page yet.
Still, I think that the hypothesis that any actual negative impact of VE on new users is due to performance issues is very supportable. Performance is the single biggest issue overall for VE right now, and performance on long pages can absolutely be prohibitively poor. Improving it is the highest priority for the team.
Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation