Hey all,
heads up that a refreshed Wikipedia Clickstream dataset is now available for March 2016, containing 25 million (referer, resource) pairs extracted from about 7 billion webrequests.
https://dx.doi.org/10.6084/m9.figshare.1305770.v16
Ellery (the author of the dataset) is cc'ed if you have any questions, or you can chime in on the talk page of the dataset entry on Meta https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream.
Show us what you do with this data, if you use it in your research.
Dario
*Dario Taraborelli *Head of Research, Wikimedia Foundation wikimediafoundation.org • nitens.org • @readermeter http://twitter.com/readermeter
Hi Dario,
This data is super interesting! How realistic is it that your team make it available through the Wikimedia REST API [1]? I would then in turn love to add it to Wikipedia Tools [2], just imagine how amazing it would be to be able to ask a spreadsheet for…
=WIKI{OUT|IN}BOUNDTRAFFIC("en:London", TODAY()-2, TODAY()-1)
…(or obviously the API respectively) and get the results back immediately without the need to download a dump first. What do you think?
Cheers, Tom
-- [1] https://wikimedia.org/api/rest_v1/?doc [2] http://bit.ly/wikipedia-tools-add-on
Hey Thomas,
yes, I agree this dataset is really valuable (just looking at the sheer number of downloads [1] and requests for similar data we've received). I can see the value of making it more easily accessible via an API.
Ellery and I have been talking about the idea of – at the very least – scheduling the generation of new dumps, if not exposing the data programmatically. Right now, I am afraid this is not within my team's capacity and Analytics has a number of other high-priority areas to focus on. We were planning to talk to Joseph et al anyway and decide how to move forward (hi Joseph!), we'll report back on the lists as soon as this happens.
Dario
[1] https://figshare.altmetric.com/details/3707715
On Mon, May 2, 2016 at 3:12 AM, Thomas Steiner tomac@google.com wrote:
Hi Dario,
This data is super interesting! How realistic is it that your team make it available through the Wikimedia REST API [1]? I would then in turn love to add it to Wikipedia Tools [2], just imagine how amazing it would be to be able to ask a spreadsheet for…
=WIKI{OUT|IN}BOUNDTRAFFIC("en:London", TODAY()-2, TODAY()-1)
…(or obviously the API respectively) and get the results back immediately without the need to download a dump first. What do you think?
Cheers, Tom
-- [1] https://wikimedia.org/api/rest_v1/?doc [2] http://bit.ly/wikipedia-tools-add-on
-- Dr. Thomas Steiner, Employee (http://blog.tomayac.com, https://twitter.com/tomayac)
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg, Germany Managing Directors: Matthew Scott Sucherman, Paul Terence Manicle Registration office and registration number: Hamburg, HRB 86891
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.29 (GNU/Linux)
iFy0uwAntT0bE3xtRa5AfeCheCkthAtTh3reSabiGbl0ck0fjumBl3DCharaCTersAttH3b0ttom hTtPs://xKcd.cOm/1181/ -----END PGP SIGNATURE-----
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Ellery and I have been talking about the idea of – at the very least –
scheduling the generation of new dumps, if not exposing the data programmatically. Right now, I am afraid >this is not within my team's capacity and Analytics has a number of other high-priority areas to focus on.
Right. Most of our work in the next couple quarters revolves around editing data, we need that to be in as good shape as pageview data is now in the near future so I think we will be able to take projects like this one again in probably couple quarters.
I have created a phab task with your awesome use case. https://phabricator.wikimedia.org/T134231
On Mon, May 2, 2016 at 4:51 PM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
Hey Thomas,
yes, I agree this dataset is really valuable (just looking at the sheer number of downloads [1] and requests for similar data we've received). I can see the value of making it more easily accessible via an API.
Ellery and I have been talking about the idea of – at the very least – scheduling the generation of new dumps, if not exposing the data programmatically. Right now, I am afraid this is not within my team's capacity and Analytics has a number of other high-priority areas to focus on. We were planning to talk to Joseph et al anyway and decide how to move forward (hi Joseph!), we'll report back on the lists as soon as this happens.
Dario
[1] https://figshare.altmetric.com/details/3707715
On Mon, May 2, 2016 at 3:12 AM, Thomas Steiner tomac@google.com wrote:
Hi Dario,
This data is super interesting! How realistic is it that your team make it available through the Wikimedia REST API [1]? I would then in turn love to add it to Wikipedia Tools [2], just imagine how amazing it would be to be able to ask a spreadsheet for…
=WIKI{OUT|IN}BOUNDTRAFFIC("en:London", TODAY()-2, TODAY()-1)
…(or obviously the API respectively) and get the results back immediately without the need to download a dump first. What do you think?
Cheers, Tom
-- [1] https://wikimedia.org/api/rest_v1/?doc [2] http://bit.ly/wikipedia-tools-add-on
-- Dr. Thomas Steiner, Employee (http://blog.tomayac.com, https://twitter.com/tomayac)
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg, Germany Managing Directors: Matthew Scott Sucherman, Paul Terence Manicle Registration office and registration number: Hamburg, HRB 86891
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.29 (GNU/Linux)
iFy0uwAntT0bE3xtRa5AfeCheCkthAtTh3reSabiGbl0ck0fjumBl3DCharaCTersAttH3b0ttom hTtPs://xKcd.cOm/1181/ -----END PGP SIGNATURE-----
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
--
*Dario Taraborelli *Head of Research, Wikimedia Foundation wikimediafoundation.org • nitens.org • @readermeter http://twitter.com/readermeter
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hi Dario and Ellery :) As Nuria said there is a lot of work ongoing on Edit data, but I'll be very interested in discussing how to better productionize / serve the data. Cheers ! Joseph
On Tue, May 3, 2016 at 6:17 AM, Nuria Ruiz nuria@wikimedia.org wrote:
Ellery and I have been talking about the idea of – at the very least –
scheduling the generation of new dumps, if not exposing the data programmatically. Right now, I am afraid >this is not within my team's capacity and Analytics has a number of other high-priority areas to focus on.
Right. Most of our work in the next couple quarters revolves around editing data, we need that to be in as good shape as pageview data is now in the near future so I think we will be able to take projects like this one again in probably couple quarters.
I have created a phab task with your awesome use case. https://phabricator.wikimedia.org/T134231
On Mon, May 2, 2016 at 4:51 PM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
Hey Thomas,
yes, I agree this dataset is really valuable (just looking at the sheer number of downloads [1] and requests for similar data we've received). I can see the value of making it more easily accessible via an API.
Ellery and I have been talking about the idea of – at the very least – scheduling the generation of new dumps, if not exposing the data programmatically. Right now, I am afraid this is not within my team's capacity and Analytics has a number of other high-priority areas to focus on. We were planning to talk to Joseph et al anyway and decide how to move forward (hi Joseph!), we'll report back on the lists as soon as this happens.
Dario
[1] https://figshare.altmetric.com/details/3707715
On Mon, May 2, 2016 at 3:12 AM, Thomas Steiner tomac@google.com wrote:
Hi Dario,
This data is super interesting! How realistic is it that your team make it available through the Wikimedia REST API [1]? I would then in turn love to add it to Wikipedia Tools [2], just imagine how amazing it would be to be able to ask a spreadsheet for…
=WIKI{OUT|IN}BOUNDTRAFFIC("en:London", TODAY()-2, TODAY()-1)
…(or obviously the API respectively) and get the results back immediately without the need to download a dump first. What do you think?
Cheers, Tom
-- [1] https://wikimedia.org/api/rest_v1/?doc [2] http://bit.ly/wikipedia-tools-add-on
-- Dr. Thomas Steiner, Employee (http://blog.tomayac.com, https://twitter.com/tomayac)
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg, Germany Managing Directors: Matthew Scott Sucherman, Paul Terence Manicle Registration office and registration number: Hamburg, HRB 86891
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.29 (GNU/Linux)
iFy0uwAntT0bE3xtRa5AfeCheCkthAtTh3reSabiGbl0ck0fjumBl3DCharaCTersAttH3b0ttom hTtPs://xKcd.cOm/1181/ -----END PGP SIGNATURE-----
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
--
*Dario Taraborelli *Head of Research, Wikimedia Foundation wikimediafoundation.org • nitens.org • @readermeter http://twitter.com/readermeter
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
I have created a phab task with your awesome use case. https://phabricator.wikimedia.org/T134231
Thanks, subscribed. Looking forward to the feature :-)