On Fri, Jul 26, 2013 at 5:52 PM, James Salsman jsalsman@gmail.com wrote:
Do you intend to measure the total number of edits per day prior to and after the visual editor roll-out?
It appears that you have not analyzed or presented any data associated with those statistics.
We run A/B tests precisely so we don't need to rely on that kind of analysis. Pre/post launch comparisons are notoriously subject to confounding effects. I really wish we could just look at that kind of data, because running properly designed and instrumented experiments is really hard. But we can't, if we want to really know what caused an increase or decrease in edits.
When you look at these kinds of numbers just on a pre/post basis, it's very hard to discern what causes a drop or increase in any given metric. We know, for instance, that as the summer progresses, editing activity drops and climbs again in the fall. We also have no idea what the impact of other deployments during that week might be (even small improvements or regressions in performance have big effects, for example). The list of unknown potential confounds go on.
For the interested: Dario covered why this kind of pre/post analysis is faulty in his discussion of cohort analysis and analytics tools at a Metrics & Activities meeting.[1][2]
Steven
1. https://meta.wikimedia.org/wiki/Metrics_and_activities_meetings/2013-03-07 2. Slides: https://docs.google.com/a/wikimedia.org/presentation/d/12HWRzf8XHsWC9zE3onyi...