It would be interesting to test for random dates in the past how often the prediction went outside predicted range without any significant intervention happening at that time. It's so easy to accept favorable deviations as a prove the intervention worked, while deviations probably happen all the time. We are in a very complex world, with infinite causes influencing our stats. BTW ru.wikipedia from Russia runs into negative numbers in four months from now. -----Original Message----- From: analytics-bounces(a)lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Tilman Bayer Sent: Thursday, September 17, 2015 22:31 To: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. Subject: [Analytics] Intervention analysis (Re: Wikimedia traffic forecast application) This app is really cool. I wonder if beside future predictions, it could be modified to support another use case: Assessing the impact of past events and software changes on our pageviews. As many of us are aware, the Wikimedia movement has been struggling for a long time to understand the effects of our work (and of outside events) on our readership. And while WMF engineering teams are getting better about doing, say, A/B tests, it's often not possible to provide a controlled environment for such experiments. There's an established statistical technique aimed at such situations, called "Intervention Analysis", see e.g. [1]. It requires modeling the time series (here: monthly pageviews) with an ARIMA model just like it has been done in the app. One then basically does a backdated forecast from the time of the intervention, and uses the difference between that forecast and the actual development to model the effect of the intervention. I've been wondering recently if this has ever been used for Wikipedia pageviews; yesterday while attending Morten's research showcase talk about their "misalignment" paper I noticed that that paper has indeed been applying it (to views of individual articles, where it may be easier to isolate effects).[2] Is anyone aware of other examples? Would it be possible to modify the app to support such backdated forecasts, as a first step, and also for calculating their difference to the actual development? [1] https://onlinecourses.science.psu.edu/stat510/node/76 [2] http://www-users.cs.umn.edu/~morten/publications/icwsm2015-popularity-quali… (p.8) On Tue, Sep 15, 2015 at 5:28 PM, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote: -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics