Hi all,
For all Hive users using stat1002/1004, you might have seen a deprecation
warning when you launch the hive client - that claims it's being replaced
with Beeline. The Beeline shell has always been available to use, but it
required supplying a database connection string every time, which was
pretty annoying. We now have a wrapper
<https://github.com/wikimedia/operations-puppet/blob/production/modules/role…>
script
setup to make this easier. The old Hive CLI will continue to exist, but we
encourage moving over to Beeline. You can use it by logging into the
stat1002/1004 boxes as usual, and launching `beeline`.
There is some documentation on this here:
https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Beeline.
If you run into any issues using this interface, please ping us on the
Analytics list or #wikimedia-analytics or file a bug on Phabricator
<http://phabricator.wikimedia.org/tag/analytics>.
(If you are wondering stat1004 whaaat - there should be an announcement
coming up about it soon!)
Best,
--Madhu :)
We’re glad to announce the release of an aggregate clickstream dataset extracted from English Wikipedia
http://dx.doi.org/10.6084/m9.figshare.1305770 <http://dx.doi.org/10.6084/m9.figshare.1305770>
This dataset contains counts of (referer, article) pairs aggregated from the HTTP request logs of English Wikipedia. This snapshot captures 22 million (referer, article) pairs from a total of 4 billion requests collected during the month of January 2015.
This data can be used for various purposes:
• determining the most frequent links people click on for a given article
• determining the most common links people followed to an article
• determining how much of the total traffic to an article clicked on a link in that article
• generating a Markov chain over English Wikipedia
We created a page on Meta for feedback and discussion about this release: https://meta.wikimedia.org/wiki/Research_talk:Wikipedia_clickstream <https://meta.wikimedia.org/wiki/Research_talk:Wikipedia_clickstream>
Ellery and Dario
I've added the following unanswered questions at
https://wikitech.wikimedia.org/wiki/Talk:Analytics/Data/Webrequest/RawIPUsa…
1. Is the ability to rerun metrics more important than protecting
reader privacy?
2. On what basis is the decision on the previous question made, or if
there is no decision on the question yet, who has the authority to
establish that basis?
3. Can Ops use access logs in which the article names have never been
stored on permanent, non-RAM media?
4. Can the users who require logs of article names use those in which
the IP address, proxy information, and geolocation has never been
stored on permanent media?
Laurentiu,
I am cc-ing analytics@ public e-mail lists where you can direct this type
of questions.
>for this goal we need time series of the http requests (pagecount, traces,
and so on) the resolution of milliseconds.
This is really not super clear. I think you need to provide a more through
description of the data you need.
For example, this describes internal data that we retain for ad most 90
days:
https://wikitech.wikimedia.org/wiki/Analytics/Data/Webrequest
Given that you need ms resolution it is not likely we might have a dataset
you can use but maybe someone on this list can point you to a released
dataset that might be suitable.
Thanks,
Nuria
On Fri, Jan 27, 2017 at 3:00 PM, Laurentiu Checiu <lchec097(a)uottawa.ca>
wrote:
> Dear Ms. Nuria Ruiz,
>
> I am a PhD student at the University of Ottawa and my research is focused
> on the cloud computing stochastic models.
>
> I found "Page view statistics for Wikimedia projects":
> http://dumps.wikimedia.org/other/pagecounts-raw and from this source I
> can construct time series of http requests on a hourly base. Based on these
> time series we can estimate a model for a cloud computing system. However,
> this hourly rate of the requests is not quite suitable for our intended
> model. We are aiming to a model able to react at level of seconds or even
> faster and for this goal we need time series of the http requests
> (pagecount, traces, and so on) at the resolution of milliseconds. We are
> interested only on the number of the requests on the time unit (ms) and not
> on the actual source or the destination of these http requests.
>
> May I ask you for help in this matter ?
>
> Best regards,
> Laurentiu Checiu
>
Hullo,
How do we go about purging data from statsd? We recently stopped
incrementing a handful of keys (see below) that we haven't used in quite
some time and would like to tidy up after ourselves.
Thanks,
-Sam
---------- Forwarded message ----------
From: Sam Smith <samsmith(a)wikimedia.org>
Date: Wed, Jan 25, 2017 at 8:22 PM
Subject: Removing wfIncrStats from MobileFrontend's Special:MobileOptions
page
To: mobile-l <mobile-l(a)lists.wikimedia.org>, Internal communication for WMF
Reading team <reading-wmf(a)lists.wikimedia.org>
Hullo,
Some time ago, the Reading Web team noticed that the Special:MobileOptions
page is instrumented in two distinct ways: using EventLogging and
wfIncrStats. We had a little spare bandwidth this sprint and got
around to removing
the <https://phabricator.wikimedia.org/T132134>wfIncrStats
<https://phabricator.wikimedia.org/T132134>-based logging from
<https://phabricator.wikimedia.org/T132134>Special:MobileOptions
<https://phabricator.wikimedia.org/T132134>.
As of today's deploy, we won't be incrementing the following keys in statsd:
- mobile.options.views
- mobile.options.errors
- mobile.options.saves
We'll be looking into and documenting other wfIncrStats-based logging in
the MobileFrontend codebase soon.
Thanks!
-Sam
(including analytics@ public list)
Rafael:
As I think we have mention before please be so kind to e-mail analytics@
rather than individual people.
>Again, we are exclusively looking for the absolute number of Wikipedia
updates per year per county.
I think you mean "edits" to wikipedia. If so we are currently working on a
project that sets up to deliver an estimate (likely an interval) of edits
happening a country. we do not have an ETA for such a deliverable but the
major project that you can follow is this one:
https://phabricator.wikimedia.org/T130256
You can ping us again by the end of next quarter (April 2017) by which we
can probably give you more specific information.
Thanks,
Nuria
On Wed, Nov 2, 2016 at 12:53 PM, Rafael Escalona Reynoso <re32(a)cornell.edu>
wrote:
> Dear Dan,
>
> I hope you are doing fine and that you remember me. I am the lead
> researcher at The Global Innovation Index (GII). I contacted you last year
> searching for data on Wikipedia uploads per country. I believe that this
> request got assigned a task number at some point. Here is what I know:
>
>
>
>
>
> Can you please let me know of this request’s status?
>
>
>
> Also, if any legal issues seem to be obstructing the compilation of this
> data, can you please refer us to someone from your legal department to
> explore the possibility of tailoring a contract/confidentiality agreement
> between the GII and Wikimedia?
>
>
>
> Again, we are exclusively looking for the absolute number of Wikipedia
> updates per year per county. These used to be available via Wikimedia here:
>
> https://stats.wikimedia.org/wikimedia/squids/
>
>
>
> Hope to hear from you soon.
>
>
>
> Sincerely,
>
> Rafael Escalona Reynoso, PhD, MPA.
>
> Lead Researcher at The Global Innovation Index
>
> Samuel Curtis Johnson Graduate School of Management
>
> 207 Sage Hall
> Cornell University
> Ithaca, NY 14853-6201
>
> Phone: +1 (607) 262-0983
>
> Email: re32(a)cornell.edu <soumitra.dutta(a)cornell.edu>
>
> http://www.johnson.cornell.edu
>
> [image: cid:image001.png@01CB662F.A467E740]
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Mon, Apr 4, 2016 at 10:58 PM, Dan Andreescu <dandreescu(a)wikimedia.org>
> wrote:
>
> Yes, once this data is properly anonymized, it should continue to be
> released in the same shape. We just have to make sure it's properly safe
> first.
>
>
>
> *From: *Rafael Escalona Reynoso
>
> *Sent: *Monday, April 4, 2016 19:21
>
> *To: *Dan Andreescu
>
> *Cc: *Jordan Litner; sacha.wunschvincent(a)wipo.int
>
> *Subject: *RE: On Wikipedia edits archive per county.
>
>
>
> Dan,
>
> Thank you for the update. This is kind of what we were expecting. I have a
> follow-up question: Would the data be collected in the same fashion for
> subsequent years (2016, 2017, etc.)? Or will this be a single time
> exercise? Do let me know whenever you can.
>
>
>
> Best,
>
>
>
>
>
> Rafael Escalona Reynoso, PhD, MPA.
>
> Lead Researcher at The Global Innovation Index
>
> Samuel Curtis Johnson Graduate School of Management
>
> 207 Sage Hall
>
> Cornell University
>
> Ithaca, NY 14853-6201
>
>
>
> Phone 1: +1 (607) 262-0983
>
> Phone 2: +1 (607) 255-9245
>
> Email: re32(a)cornell.edu
>
> http://www.johnson.cornell.edu
>
>
>
> [image: cid:image001.png@01CB662F.A467E740]
>
>
>
> See www.globalinnovationindex.org
>
>
>
>
>
>
>
> *From:* Dan Andreescu [mailto:dandreescu@wikimedia.org]
> *Sent:* Monday, April 04, 2016 6:13 PM
>
> *To:* Rafael Escalona Reynoso
> *Cc:* Jordan Litner; sacha.wunschvincent(a)wipo.int
> *Subject:* Re: On Wikipedia edits archive per county.
>
>
>
> Hey Rafael,
>
>
>
> We haven't been able to prioritize this work yet. It's been moved here:
>
>
>
> https://phabricator.wikimedia.org/T131280
>
>
>
> It has two stakeholders but no resources to get it done due to privacy
> issues. So we won't be able to get the 2015 data cleaned up before your
> deadline. But we're meeting about it again tomorrow and we will still do
> it so you can have this data for either next year's report or an amendment.
>
>
>
> On Mon, Apr 4, 2016 at 11:49 AM, Rafael Escalona Reynoso <re32(a)cornell.edu>
> wrote:
>
> Dear Dan,
>
> Hope you are doing fine. Just is quick note to follow up on the Wikipedia
> data. When do you think this data will be available? We are about to close
> the model and would very much like to have 2015 data included. Let me know.
>
>
>
> Best,
>
>
>
> Rafael Escalona Reynoso, PhD, MPA.
>
> Lead Researcher at The Global Innovation Index
>
> Samuel Curtis Johnson Graduate School of Management
>
> 207 Sage Hall
>
> Cornell University
>
> Ithaca, NY 14853-6201
>
>
>
> Phone 1: +1 (607) 262-0983
>
> Phone 2: +1 (607) 255-9245
>
> Email: re32(a)cornell.edu
>
> http://www.johnson.cornell.edu
>
>
>
> [image: cid:image001.png@01CB662F.A467E740]
>
>
>
> See www.globalinnovationindex.org
>
>
>
>
>
>
>
> *From:* Dan Andreescu [mailto:dandreescu@wikimedia.org]
> *Sent:* Thursday, February 18, 2016 8:19 PM
>
>
> *To:* Rafael Escalona Reynoso
> *Cc:* Jordan Litner; sacha.wunschvincent(a)wipo.int
> *Subject:* Re: On Wikipedia edits archive per county.
>
>
>
> That's perfect, I added it to the request: https://phabricator.
> wikimedia.org/T127409
>
> On Thursday, February 18, 2016, Rafael Escalona Reynoso <re32(a)cornell.edu>
> wrote:
>
> Dan,
>
> These were my thoughts exactly. Let me then elaborate on the value of the
> report, the index and why we feel that Wikipedia data is essential.
>
>
>
> The report is co-published by Cornell University, INSEAD, and the World
> Intellectual Property Organization (WIPO, a specialized agency of the
> United Nations), with the collaboration of three Knowledge Partners: the
> Confederation of Indian Industry, du, and A.T. Kearney and IMP³rove –
> European Innovation Management Academy. Now in its ninth edition, the
> report has established itself as a premier reference among innovation
> metrics and as a tool to facilitate public-private dialogue and
> evidence-based policymaking.
>
>
>
> The Global Innovation Index (GII) is a ranking of 141 economies in terms
> of their innovation capabilities and results. A total of 79 metrics in the
> form of data-based indicators are at its core. These rich metrics can be
> used —on the level of the index, the sub-indices, or as individual
> variables—to monitor performance over time and to benchmark developments
> against their peers. These can also help study country profiles over time,
> and to identify their relative strengths and weaknesses from the rich and
> unique GII dataset.
>
>
>
> Each year the GII results are presented within the framework of a
> top-level international event:
>
> • 2013 Geneva, Switzerland at the Opening Session of
> the United Nations Economic and Social Council (ECOSOC) High-Level Segment,
> organized by WIPO;
>
> • 2014 Sydney, Australia in the context of the B20/G20
> preparations; and
>
> • 2015 London, United Kingdom before the Minister of
> Innovation and Industry.
>
>
>
> This year the launch is scheduled for the summer in Beijing, China
> preceding the preparations for the 2016 G20 summit.
>
>
>
> Recognizing the need for a broad horizontal vision of innovation
> applicable to developed and emerging economies alike, the GII includes
> indicators that go beyond the traditional measures such as expenditure in
> research and development. That said, an area that is of great relevance and
> limited to the GII is that of creative outputs. Within it, *Wikipedia
> monthly page edits (per million population 15-69 y/o)* is a key metric.
> This indicator, along with others that measure the number of generic
> top-level and country-code top-level domains and video uploads in YouTube,
> helps capture what we define as online creativity.
>
>
>
> Lastly, we believe that the GII can be an important vehicle to signal that
> Wikipedia is a critical lever to innovation and a factor contributing to a
> new understanding of the digital information landscape and innovation
> globally.
>
>
>
> Based on all the above, we would like to request that our petition to
> collect data on Wikipedia monthly page edits per country, reported
> quarterly per year be given priority within your tasks.
>
>
>
> Sincerely,
>
>
>
>
>
> Rafael Escalona Reynoso, PhD, MPA.
>
> Lead Researcher at The Global Innovation Index
>
> Samuel Curtis Johnson Graduate School of Management
>
> 207 Sage Hall
>
> Cornell University
>
> Ithaca, NY 14853-6201
>
>
>
> Phone 1: +1 (607) 262-0983
>
> Phone 2: +1 (607) 255-9245
>
> Email: re32(a)cornell.edu
>
> http://www.johnson.cornell.edu
>
>
>
> [image: cid:image001.png@01CB662F.A467E740]
>
>
>
> See www.globalinnovationindex.org
>
>
>
>
>
>
>
> *From:* Dan Andreescu [mailto:dandreescu@wikimedia.org
> <dandreescu(a)wikimedia.org>]
> *Sent:* Thursday, February 18, 2016 11:17 AM
> *To:* Rafael Escalona Reynoso
> *Cc:* Jordan Litner; sacha.wunschvincent(a)wipo.int
> *Subject:* Re: On Wikipedia edits archive per county.
>
>
>
> Where the request is coming from, with all due respect, does not matter.
> We aim to be neutral in how we make knowledge available (namely, we try to
> make it available to everyone, for free).
>
>
>
> But, we have to prioritize somehow, and that process definitely takes into
> consideration the value our work has to the world. So, if you tell me more
> about what this data could help you accomplish, we could use that to argue
> that prioritizing your request might save lives, serve the mission of open
> knowledge, etc.
>
>
>
> But to answer your other question directly, yes, a letter to Jimmy Wales
> would not have any effect on this priority process and might be seen by the
> community we serve as an attempt to circumvent our planning process.
>
>
>
> On Thu, Feb 18, 2016 at 10:54 AM, Rafael Escalona Reynoso <
> re32(a)cornell.edu> wrote:
>
> Dan,
>
> Let me share with you the following thought. I just had a call with the
> Dean at the business school here at Cornell, who is the creator of the
> Global Innovation Index (and my direct boss). I explained the situation
> with the Wikipedia uploads data and how methodological changes are now
> making it impossible for us to collect it in the fashion that we were used
> to. He mentioned that he is an acquaintance with Jimmy Wales and offered to
> direct him a letter explaining what we need and the importance of the
> indicator for our index. My notion here is that the issue has more to do
> with a shortage of labor hand and quite large a backlog than with where the
> request is coming from. Also, I do not want the letter to come across as an
> imposition or to give the wrong message. Based on the above, would this
> letter help prioritize the collection of this data?
>
>
>
> Let me know what you think.
>
>
>
> Best,
>
>
>
> Rafael Escalona Reynoso, PhD, MPA.
>
> Lead Researcher at The Global Innovation Index
>
> Samuel Curtis Johnson Graduate School of Management
>
> 207 Sage Hall
>
> Cornell University
>
> Ithaca, NY 14853-6201
>
>
>
> Phone 1: +1 (607) 262-0983
>
> Phone 2: +1 (607) 255-9245
>
> Email: re32(a)cornell.edu
>
> http://www.johnson.cornell.edu
>
>
>
> [image: cid:image001.png@01CB662F.A467E740]
>
>
>
> See www.globalinnovationindex.org
>
>
>
>
>
> *From:* Dan Andreescu [mailto:dandreescu@wikimedia.org
> <dandreescu(a)wikimedia.org>]
> *Sent:* Wednesday, February 17, 2016 2:57 PM
> *To:* Rafael Escalona Reynoso
> *Cc:* Jordan Litner; sacha.wunschvincent(a)wipo.int
> *Subject:* Re: On Wikipedia edits archive per county.
>
>
>
> As much as I love to help a fellow Cornellian, we are too small of a team
> to create one-off solutions like that. We either publish it for everyone
> or no-one. But even if we did that, we'd still have a lot of work to check
> whether cross-referencing that data with other data wouldn't hurt privacy.
>
>
>
> What would help is if you filed a task in Phabricator and tagged it with
> the "Analytics" project, and described very precisely what data you need,
> at what time granularity, and what you need it for. We'll use that as
> proof that we need to prioritize the work sooner than later.
>
>
>
> On Wed, Feb 17, 2016 at 2:45 PM, Rafael Escalona Reynoso <re32(a)cornell.edu>
> wrote:
>
> Dan,
>
> One last thing. We also report scaled data from Google on YouTube uploads
> and, as you mention, they have to protect privacy. However, we prepare for
> them an Excel sheet where they simply need to upload the totals for each
> country (which we never get to see) and they report back to us exclusively
> the normalized scores (0-100) and rankings for all countries we request
> information for. Using this procedure it becomes impossible to
> reverse-engineer the raw values used to obtain these totals and – again –
> we never get to see the actual data. Is there a chance that we could
> establish a similar type of arrangement with Wikimedia? Let me know.
>
>
>
> Best,
>
>
>
> Rafael.
>
>
>
> *From:* Dan Andreescu [mailto:dandreescu@wikimedia.org
> <dandreescu(a)wikimedia.org>]
> *Sent:* Wednesday, February 17, 2016 2:23 PM
> *To:* Rafael Escalona Reynoso
> *Subject:* Re: On Wikipedia edits archive per county.
>
>
>
> Thank you for this, again. Sorry to pester you with this again but, do you
> know of any other data (from a source different from Google) where online
> activity could be measured? Any leads would be quite appreciated.
>
>
>
> mmm, not geolocated that I know of, and it's unlikely that you'd find
> that. Because either
>
>
>
> * an organization is for-profit, in which case they would sell that data
>
> * an organization is non-profit, in which case they'd likely need to
> protect their users and bump up against the same *hard* problems we did
>
>
>
> But I could be wrong, good luck in your search and do report back if you
> find any, and especially if you find approaches that help to protect
> privacy.
>
>
>
>
>
>
>
>
>
>
>
Please participate in the discussion about the "Creation and renewal of
the Committee" section. This is not to approve it yet, just a discussion:
https://www.mediawiki.org/wiki/Talk:Code_of_Conduct/Draft#.22Creation_and_r…
Most of this text has been around for a while, but I made a few changes
to the draft last week, so I wanted to give people an opportunity to
comment, suggest changes, etc.
I'll give it a few more days, then start the formal approval discussion.
Thanks,
Matt Flaschen
Hi,
We have the following task blocked on whoever maintains the PHP side of
EventLogging:
http://phabricator.wikimedia.org/T150269
Basically some of the EventLogging schema ResourceLoader modules can take a
very long time to generate. This doesn't happen with any other
ResourceLoader module.
Can someone take ownership of that task? Timo can help answer any questions
about ResourceLoader to move it forward.
Thanks
Saw this on reddit:
https://theintercept.com/2016/04/28/new-study-shows-mass-surveillance-breed…
>From the paper
<https://poseidon01.ssrn.com/delivery.php?ID=7530740711240640670151241081260…>
:
"This case study uses data on English language Wikipedia article view
counts from the online service stats.grok.se, a portal maintained by a
Wikimedia Foundation member. This portal provides access to a range of
Wikipedia analytics, stats, and data.86 In particular, the portal
aggregates Wikipedia article view data on a daily and monthly basis.87 This
data at stats.grok.se has been used in a range of research, including
studies involving market trends, health information access, and
social-political change.88”
Just thought it might be of interest, especially considering WMF’s NSA
lawsuit <https://blog.wikimedia.org/2015/03/10/wikimedia-v-nsa/>.
-Ao