This app is really cool. I wonder if beside future predictions, it could be modified to support another use case: Assessing the impact of past events and software changes on our pageviews.
As many of us are aware, the Wikimedia movement has been struggling for a long time to understand the effects of our work (and of outside events) on our readership. And while WMF engineering teams are getting better about doing, say, A/B tests, it's often not possible to provide a controlled environment for such experiments.
There's an established statistical technique aimed at such situations, called "Intervention Analysis", see e.g. [1]. It requires modeling the time series (here: monthly pageviews) with an ARIMA model just like it has been done in the app. One then basically does a backdated forecast from the time of the intervention, and uses the difference between that forecast and the actual development to model the effect of the intervention. I've been wondering recently if this has ever been used for Wikipedia pageviews; yesterday while attending Morten's research showcase talk about their "misalignment" paper I noticed that that paper has indeed been applying it (to views of individual articles, where it may be easier to isolate effects).[2] Is anyone aware of other examples?
Would it be possible to modify the app to support such backdated forecasts, as a first step, and also for calculating their difference to the actual development?
[1] https://onlinecourses.science.psu.edu/stat510/node/76 [2] http://www-users.cs.umn.edu/~morten/publications/icwsm2015-popularity-qualit... (p.8)
On Tue, Sep 15, 2015 at 5:28 PM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
An updated version of a pageview forecasting application written by Ellery (Research & Data team) has just been released:
https://ewulczyn.shinyapps.io/pageview_forecasting https://twitter.com/WikiResearch/status/643942154549592064
The data is refreshed monthly and it includes breakdowns by country and platform.
Dario
Dario Taraborelli Head of Research, Wikimedia Foundation wikimediafoundation.org • nitens.org • @readermeter
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
It would be interesting to test for random dates in the past how often the prediction went outside predicted range without any significant intervention happening at that time. It's so easy to accept favorable deviations as a prove the intervention worked, while deviations probably happen all the time. We are in a very complex world, with infinite causes influencing our stats.
BTW ru.wikipedia from Russia runs into negative numbers in four months from now.
-----Original Message----- From: analytics-bounces@lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Tilman Bayer Sent: Thursday, September 17, 2015 22:31 To: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. Subject: [Analytics] Intervention analysis (Re: Wikimedia traffic forecast application)
This app is really cool. I wonder if beside future predictions, it could be modified to support another use case: Assessing the impact of past events and software changes on our pageviews.
As many of us are aware, the Wikimedia movement has been struggling for a long time to understand the effects of our work (and of outside events) on our readership. And while WMF engineering teams are getting better about doing, say, A/B tests, it's often not possible to provide a controlled environment for such experiments.
There's an established statistical technique aimed at such situations, called "Intervention Analysis", see e.g. [1]. It requires modeling the time series (here: monthly pageviews) with an ARIMA model just like it has been done in the app. One then basically does a backdated forecast from the time of the intervention, and uses the difference between that forecast and the actual development to model the effect of the intervention. I've been wondering recently if this has ever been used for Wikipedia pageviews; yesterday while attending Morten's research showcase talk about their "misalignment" paper I noticed that that paper has indeed been applying it (to views of individual articles, where it may be easier to isolate effects).[2] Is anyone aware of other examples?
Would it be possible to modify the app to support such backdated forecasts, as a first step, and also for calculating their difference to the actual development?
[1] https://onlinecourses.science.psu.edu/stat510/node/76 [2] http://www-users.cs.umn.edu/~morten/publications/icwsm2015-popularity-qualit... (p.8)
On Tue, Sep 15, 2015 at 5:28 PM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
An updated version of a pageview forecasting application written by Ellery (Research & Data team) has just been released:
https://ewulczyn.shinyapps.io/pageview_forecasting https://twitter.com/WikiResearch/status/643942154549592064
The data is refreshed monthly and it includes breakdowns by country and platform.
Dario
Dario Taraborelli Head of Research, Wikimedia Foundation wikimediafoundation.org • nitens.org • @readermeter
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
_______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
I'm particularly interested in this area. Is there a team of researchers within the organization that analyzes these numbers and external influences on search stats? For my research, I define external influences as nation-state sponsored censorship during high tension periods, which are usually short periods of time (a day or few hours). Of corse, I'm operating under the assumption that many people search for political candidates or figures on Wikipedia during a) election cycles and/or b) times of conflict. Any other researchers here focused on geopolitical conditions and its effects on digital search behavior and its stats?
Thank you, Tilman, for sharing the link to Intervention Analysis + Time Series.
Feel free to connect with me at riabaldevia@gmail.com.
Ria
On Thu, Sep 17, 2015 at 10:30 AM, Tilman Bayer tbayer@wikimedia.org wrote:
This app is really cool. I wonder if beside future predictions, it could be modified to support another use case: Assessing the impact of past events and software changes on our pageviews.
As many of us are aware, the Wikimedia movement has been struggling for a long time to understand the effects of our work (and of outside events) on our readership. And while WMF engineering teams are getting better about doing, say, A/B tests, it's often not possible to provide a controlled environment for such experiments.
There's an established statistical technique aimed at such situations, called "Intervention Analysis", see e.g. [1]. It requires modeling the time series (here: monthly pageviews) with an ARIMA model just like it has been done in the app. One then basically does a backdated forecast from the time of the intervention, and uses the difference between that forecast and the actual development to model the effect of the intervention. I've been wondering recently if this has ever been used for Wikipedia pageviews; yesterday while attending Morten's research showcase talk about their "misalignment" paper I noticed that that paper has indeed been applying it (to views of individual articles, where it may be easier to isolate effects).[2] Is anyone aware of other examples?
Would it be possible to modify the app to support such backdated forecasts, as a first step, and also for calculating their difference to the actual development?
[1] https://onlinecourses.science.psu.edu/stat510/node/76 [2] http://www-users.cs.umn.edu/~morten/publications/icwsm2015-popularity-qualit... (p.8)
On Tue, Sep 15, 2015 at 5:28 PM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
An updated version of a pageview forecasting application written by
Ellery (Research & Data team) has just been released:
https://ewulczyn.shinyapps.io/pageview_forecasting https://twitter.com/WikiResearch/status/643942154549592064
The data is refreshed monthly and it includes breakdowns by country and
platform.
Dario
Dario Taraborelli Head of Research, Wikimedia Foundation wikimediafoundation.org • nitens.org • @readermeter
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics