[Wikimedia-l] a compromise proposal for visual editor dogfooding

[Andrew Gray]

On 25 July 2013 06:29, James Salsman <jsalsman at gmail.com> wrote:
> Why not make the visual editor the default with opt-out for 5% of
> newly registered editors and anonymous IP page loads, and opt-in for
> everyone else until there is evidence that it is not decreasing the
> number of edits?

Meant to reply to this earlier and it got lost:

I think there's a problem with any kind of A/B testing on the main
site interface for more than small changes - it's that we only have
one set of documentation. Wikipedia's help pages (certainly on enwiki)
aren't amazing, but they are used, and people will fall back to them
if they have problems using the site.

Having a substantially different editing interface for a fraction of
users means that there's now one more layer of confusion before (some
or all) people can get help, probably leading to more abandoned edits
*compared to all-in VE with documentation*. So the results would be
skewed downwards; it might be a small effect, but if we're looking for
a statistical difference on 5% of new edits, it might be enough to
give a spurious negative result.

Again, I don't think A/B testing is inherently bad, but we'd need to
test an integrated environment. Where people aren't going to consult
help pages (say, the login page) it's much simpler.

-- 
- Andrew Gray
  andrew.gray at dunelm.org.uk