[Wikimedia-l] Updates on VE data analysis

[Steven Walling]

On Fri, Jul 26, 2013 at 5:52 PM, James Salsman <jsalsman at gmail.com> wrote:

> Do you intend to measure the total number of edits per day prior to
> and after the visual editor roll-out?
>
> It appears that you have not analyzed or presented any data associated
> with those statistics.
>

We run A/B tests precisely so we don't need to rely on that kind of
analysis. Pre/post launch comparisons are notoriously subject to
confounding effects. I really wish we could just look at that kind of data,
because running properly designed and instrumented experiments is really
hard. But we can't, if we want to really know what caused an increase or
decrease in edits.

When you look at these kinds of numbers just on a pre/post basis, it's very
hard to discern what causes a drop or increase in any given metric. We
know, for instance, that as the summer progresses, editing activity drops
and climbs again in the fall. We also have no idea what the impact of other
deployments during that week might be (even small improvements or
regressions in performance have big effects, for example). The list of
unknown potential confounds go on.

For the interested: Dario covered why this kind of pre/post analysis is
faulty in his discussion of cohort analysis and analytics tools at a
Metrics & Activities meeting.[1][2]

Steven

1.
https://meta.wikimedia.org/wiki/Metrics_and_activities_meetings/2013-03-07
2. Slides:
https://docs.google.com/a/wikimedia.org/presentation/d/12HWRzf8XHsWC9zE3onyi6eeA_J98bPxoknLY_be-vUc/edit#slide=id.p