On Thu, Jul 17, 2014 at 4:24 AM, Andrew Gray andrew.gray@dunelm.org.uk wrote:
A/B testing for major interface changes isn't such a good idea, I think. It's great for small things like login fields that people aren't likely to notice, but once you get into major interface elements you're going to get quite a few people confused or frustrated.
"Why is your Wikipedia different to mine?", "Why does this not look like the help page says?", etc.
Regarding help pages ...
It is safe to say that a decent segment of the editing community wants to keep the old interface, and the old interface is needed to edit the page, and I think it is safe to assume that the MediaViewer isnt going away - at worst it will go back to being a Beta Feature that is enabled for a large segment of the userbase.
Therefore, the help pages need to cover both. This is a cost of introducing a feature that isnt a complete replacement of prior functionality - and has a high maintenance cost that the devs and community must bear. The solution to that is to hold the feature in Beta until it is an acceptable replacement for the existing functionality.
If this A/B testing goes ahead, and I think it should, the relevant help pages should have a note at the top that explains that testing is occurring and their user pref may have been preset to either one.