[Analytics] Intervention analysis (Re: Wikimedia traffic forecast application)

This app is really cool. I wonder if beside future predictions, it could be modified to support another use case: Assessing the impact of past events and software changes on our pageviews. As many of us are aware, the Wikimedia movement has been struggling for a long time to understand the effects of our work (and of outside events) on our readership. And while WMF engineering teams are getting better about doing, say, A/B tests, it's often not possible to provide a controlled environment for such experiments. There's an established statistical technique aimed at such situations, called "Intervention Analysis", see e.g. [1]. It requires modeling the time series (here: monthly pageviews) with an ARIMA model just like it has been done in the app. One then basically does a backdated forecast from the time of the intervention, and uses the difference between that forecast and the actual development to model the effect of the intervention. I've been wondering recently if this has ever been used for Wikipedia pageviews; yesterday while attending Morten's research showcase talk about their "misalignment" paper I noticed that that paper has indeed been applying it (to views of individual articles, where it may be easier to isolate effects).[2] Is anyone aware of other examples? Would it be possible to modify the app to support such backdated forecasts, as a first step, and also for calculating their difference to the actual development? [1] https://onlinecourses.science.psu.edu/stat510/node/76 [2] http://www-users.cs.umn.edu/~morten/publications/icwsm2015-popularity-quali… (p.8) On Tue, Sep 15, 2015 at 5:28 PM, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote: -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB