Cristian Consonni, 11/11/2015 15:09:
I am working with a student on scientific citation on
Wikipedia and,
very simply put, we would like to use the pageview dataset to have a
rough measure of how many times a paper was viewed thanks to
Wikipedia.[*]
The full dataset is, as of now, ~ 4.7TB in size.
I have two questions:
* if we download this dataset this would entail, from a first
estimation, ~ 30 days of continuous download (assuming an average
download speed of ~ 2MB/s, which was what we measured over the
download of a month of data (~ 64GB)). Here at my University (Trento,
Italy) this kind of downloads have to be notified to the IT
department. I was wondering if this would be a useful information for
the WMF, too.
No need to notify such small downloads.
* given the estimation above I was wondering if it is
possible to
obtain this data over FedEx Bandwith (cit. [1]). i.e. via shipping of
a physical disk, I know that in some fields (e.g. neuroscience) this
is the standard way to exchange big dataset (in the order of TBs).
This assumes that some point of the network has faster download from
that machine. The server is very slow for pretty much anyone except rare
exceptions (
https://phabricator.wikimedia.org/T45647 ), possibly even
inside the cluster. Copying to a hard drive might take many days.
You have two more alternatives:
* scp from Labs, /public/dumps/pagecounts-all-sites/ (sometimes reaches
3-4 MB/s for me);
*
archive.org for pagecounts-raw
https://archive.org/search.php?query=wikipedia_visitor_stats (you can
start download of all months at once and use torrent, will hopefully
saturate your bandwidth because you will download from dozens servers
rather than one).
Nemo
Thanks in advance for your help.
Cristian
[*] I know these are pageviews and not unique visitors, furthermore
there is no guarantee that viewing a citation means anything. I am
approaching to this data the same way "impressions" versus
"clicktroughs" are treated in the online advertising world.
[1]
https://what-if.xkcd.com/31/
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l