Hi Michal,
it seems that what you want is a data set, which would be very similar
to what I recently issued a request for: see this phabricator item
https://phabricator.wikimedia.org/T128132
There has been a public data set for the year 2007, part of which you
publicly available [1]. See [2] for a study using the 2007 data set.
My focus has been on simulating the performance of WMF's caching
servers, for which the 2007 data set is insufficient. However, a
different research domain might require a slightly different focus of
capturing the data set.
The 2007 data set was captured with a sampling rate of 1:10. For my
project, such a high sampling rate would be perfect (1:100 might also
work). However, I learned that the current request rate is much higher
so we'd have to narrow the scope of the data set (e.g., by focussing on
specific WMF projects, like the English Wikipedia). You can find a
discussion on the phabricator page linked above.
What would be the lowest sampling rate allowable for your project? I
assume the publicly available hourly access data [3], [4] would be
insufficient?
Feel free to comment on the phabricator item, maybe we can compile a
single data set that works for both of our research domains and other
helps other people?
Best,
Daniel
[1]
http://www.wikibench.eu/?page_id=60
[2]
http://www.distributed-systems.net/papers/2009.comnet-wiki.pdf
[3]
https://wikitech.wikimedia.org/wiki/Analytics/Data/Pagecounts-all-sites
[4]
http://dumps.wikimedia.org/other/pagecounts-all-sites/
On 03/21/2016 10:11 AM, Michal Bystricky wrote:
We would like to have URI addresses of requests for
some time of usage
- let's say 1 month.
According to the data format
<https://wikitech.wikimedia.org/wiki/Analytics/Data/Webrequest>, the
attributes of Webrequests we need are following:
http_method,
uri_host,
uri_path,
uri_query,
ts,
access_method,
agent_type,
pageview_info,
page_id
Do we need to go through NDA process or it is possible to get the data
right away from the public dataset?
Thank you,
M.
Can you be more specific about what you need, Michal? If you truly
need access to the private data that we keep in wmf.webrequest for a
limited time, then you'd have to go through a process to sign an
NDA. But if you tell us what you need, there may be a public dataset
that you can use.
On Thu, Mar 3, 2016 at 2:48 PM, Michal Bystricky
<michal.bystricky(a)stuba.sk> wrote:
Hello Analytics Team,
We would like to have one-time access to wmf.webrequest data.
What is the correct way of accessing the data?
In our research group, we want to simulate the requests for
specific version of WikiMedia.
Thanks,
Michal Bystricky
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org <mailto:Analytics@lists.wikimedia.org>
https://lists.wikimedia.org/mailman/listinfo/analytics
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics