Are there any reasons to not replace HTTP GET request IP addresses and proxy information with their SHA-512 secure hash prior to writing them to permanent media?

So, just to give context, our HTTP requests take this path: * varnish log (very small buffer, not permanent) * varnishkafka * kafka (small buffer, I think 7 days) * camus * refine process (we use IPs at this point to geolocate) * webrequest table on hdfs (this is the first time they're stored on permanent media, for 60 days) * other datasets like hourly pageviews aggregates, (IPs are not passed on to these) So if we wanted to not store them in kafka buffers even, we'd have to give up geolocating. I think a lot of people find this very useful (fundraising, research, ops, reading), so it's unlikely to be removed. I don't have as clear a reason for why we store the plain IP in webrequest. I think we could count uniques and all that other stuff with the IP hash. It's a good question, tentative +1 unless I'm forgetting something. But even so, it's not so bad, it's only stored for 60 days and we have no other plain IPs anywhere else (like we removed them from Event Logging for example). On Tue, Nov 8, 2016 at 4:26 PM, James Salsman <jsalsman(a)gmail.com> wrote: _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Dan Andreescu, 10/11/2016 16:00: I support any decrease of the storage of plain IP addresses. See also < https://www.mediawiki.org/wiki/Thread:Talk:Requests_for_com ment/Structured_logging/IP_address_and_other_personal_identi fying_information> for more references. Nemo _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

https://www.mediawiki.org/wiki/Thread:Talk:Requests_for_com ment/Structured_logging/IP_address_and_other_personal_identi fying_information> for more references. To be clear: on our end we need buffer time that allows us to know that should there be a bug we can reprocess pageviews if needed (this does happen). That buffer time is now 60 days and perhaps it could be a bit smaller but it is still going to be a matter of weeks, not days for which the raw data needs to be available. As mentione earlier in the thread we need raw IPs to geolocate requests, once that is done IPs are discarded. On Fri, Nov 11, 2016 at 12:00 AM, Federico Leva (Nemo) &lt;nemowiki(a)gmail.com _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

On Fri, Nov 11, 2016 at 9:25 AM, Leila Zia &lt;leila(a)wikimedia.org&gt; wrote: I'd be happy to have Legal and Analytics take a look at what could be done to tighten the screws a bit on who has access to other data in the logs such as UAs. (To follow up on a comment from Wikimedia-l: I'm also very wary of letting people outside of WMF and the community have access to this kind of information, even with a signed NDA.)

Pine _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

​Hi Pine, On Fri, Nov 11, 2016 at 10:39 AM, Pine W &lt;wiki.pine(a)gmail.com&gt; wrote: I'm not a supporter of the narrative that non-staff folks who have access to the webrequest logs should be limited more than staff members who can have access to the logs. Some of these folks are highly trained individuals (sometimes even more than staff members) and some of them are less experienced but work very closely with a staff member who is experienced in dealing with sensitive data. They understand the importance of the data they work with, and we do our share in onboarding them and making sure we are all on the same page about what data they're working with and how they should handle it. Let's step back: * Subpoena related concerns: the best way to handle this from the data storage perspective is to not have the data at all. That is why very sensitive data is purged after 60 days at the moment in webrequest logs. As Nuria said, this length of time may be shortened by a little, but at least because of operational constraints, we won't be able to not store this data at all. * Error related concerns: One way to reduce the errors is to constrain the number of people who can access the data (which is already happening, we're talking about increasing restrictions here). In this case, there is very little difference between staff and non-staff folks who have access to webrequest logs at the moment. Mistakes can happen by people in each group. I may make a mistake and give an output in my GitHub account with the top 10 IP addresses that have accessed WP in the last hour. This mistake can happen, by anyone accessing this data. The logical thing to do is to reduce the number of people who don't /have to/ have access to that data. If I don't /need/ to see the IPs for my work, I shouldn't see them, whether I'm a staff member or non-staff under NDA to access this data. If I should, then we should accept that mistakes can happen, but we will do our best to reduce them. * I also want to point out prioritization here, which is something Nuria and her team should handle (and this will affect Security, Research, and Legal as well): the Analytics team has been allocating resources to transition wikistats. This has been a gigantic endeavour by the team. We know that if wikistats data is not generated for a few months, we will have a lot of unhappy people around. If we are to step back and allocate resources to spend hours on rethinking how we handle webrequest logs (and I can assure you that this will require at least 2 full-time work week of many people, likely spread over months), we will have to slow down some other effort. Also, consider Security who needs to be involved in this process. You know better than I that Security has a lot to do with very few people. imho, we are doing a very good job with the way we are handling webrequest logs data at the moment given our constraints. Sure, we can and should improve some steps over time. Best, Leila _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

On Fri, Nov 11, 2016 at 2:16 PM, Leila Zia &lt;leila(a)wikimedia.org&gt; wrote: It is worth considering this in context of https://twitter.com/ Pinboard/status/797167026481442816 That is, not storing the data is nice, but do we have any plans in place in case a government decides to place a recording device in our data center beside our servers? We may have the best of intentions, but "we don't store it" could in fact be misleading comfort if there is a third-party who *is* storing it. This is perhaps a broader question (and more in line with James' initial inquiry?), as it suggests that we reconsider what sort of protections we can actually provide to our editors, and make sure they know if we can't protect them from state-level monitoring. --scott -- (http://cscott.net) _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

to the webrequest logs should be limited more than staff members who can have access to the logs. Some >of these folks are highly trained individuals (sometimes even more than staff members) and some of them are less experienced but work very closely with a staff member who is of the data they work with, and we do our share in onboarding them and making sure we are all on the >same page about what data they're working with and how they should handle it. +1, researchers collaborating with research team in data-centered projects are probably much more aware of data and privacy issues than the average WMF person that accesses data just sporadically. On Fri, Nov 11, 2016 at 11:16 AM, Leila Zia &lt;leila(a)wikimedia.org&gt; wrote: _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

By the way, to the best of my knowledge, all recordings to "permanent media" are overwritten or destroyed after 60 days. I thought that was specified in either the Privacy Policy or Terms of Use but I can't find the specific reference, and that bothers me. Can someone at Legal explain why this isn't specified in either in the PP or ToU? (Feel free to fork this question if it becomes a distraction to the original thread.) Thanks, Pine On Tue, Nov 8, 2016 at 1:26 PM, James Salsman &lt;jsalsman(a)gmail.com&gt; wrote:

.... on our end we need buffer time that allows us to know that should there be a bug we can reprocess pageviews if needed (this does happen). That buffer time is now 60 days and perhaps it could be a bit smaller but it is still going to be a matter of weeks, not days for which the raw data needs to be available.

Pine wrote: I am not suggesting removing the IP addresses or proxy information from POST requests as checkuser requires. We need to anonymize both IP addresses and proxy information with a secure hash if we want to keep each GET request's geolocation, to be compliant with the Privacy Policy. The Privacy Policy is the most prominent policy on the far left on the footer of every page served by every editable project, and says explicitly that consent is required for the use of geolocation. The Privacy and other policies make it clear that POST requests and Visual Editor submissions aren't going to be anonymized. However, geolocations for POST edit and visual editor submissions still require explicit consent which we have no way to obtain at present. Editors' geolocations as they edit are very useful for research, but by the same token have the most serious privacy concerns. Obtaining consent to store geolocation seems like it would interfere with, complicate, and disrupt editing. If geolocation is stored with anonymized IP addresses for GETs but not POSTs or Visual Editor submissions, both could easily be recovered because of simultaneously interleaved GET and POST requests for the same article are unavoidable. Do we have any privacy experts on staff who can give these issues a thorough analysis in light of all the issues raised in https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1450006 ? If Ops needs IP addresses, they should be able to use synthetic POST requests, as far as I can tell. If they anticipate a need for non-anonymous GET requests, then perhaps some kind of a debugging switch which could be used on a short term basis where an IP range or mask could be entered to allow matching addresses to log non-anonymously before expiring in an hour would solve any anticipated need?

First, I don't know who or where to ask such questions of the Ops team.

Second, is the suggestion to discard before storing as the default behavior with a manual switch that can be turned on by Ops to store temporary raw logs with IP for debugging if and when needed by Ops for a limited time -- say an hour or two -- with automatic zeroing deletion at the end of that

period a viable solution to this contingency?

Nuria Ruiz wrote: long First, I don't know who or where to ask such questions of the Ops team. Second, is the suggestion to discard before storing as the default behavior with a manual switch that can be turned on by Ops to store temporary raw logs with IP for debugging if and when needed by Ops for a limited time -- say an hour or two -- with automatic zeroing deletion at the end of that time period a viable solution to this contingency? _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

We need to anonymize both IP addresses and proxy information with a secure

The Privacy Policy is the most prominent policy on the far left on the

every editable project, and says explicitly that consent is required for

Do we have any privacy experts on staff who can give these issues a

If Ops needs IP addresses, they should be able to use synthetic POST

Pine wrote: I am not suggesting removing the IP addresses or proxy information from POST requests as checkuser requires. We need to anonymize both IP addresses and proxy information with a secure hash if we want to keep each GET request's geolocation, to be compliant with the Privacy Policy. The Privacy Policy is the most prominent policy on the far left on the footer of every page served by every editable project, and says explicitly that consent is required for the use of geolocation. The Privacy and other policies make it clear that POST requests and Visual Editor submissions aren't going to be anonymized. However, geolocations for POST edit and visual editor submissions still require explicit consent which we have no way to obtain at present. Editors' geolocations as they edit are very useful for research, but by the same token have the most serious privacy concerns. Obtaining consent to store geolocation seems like it would interfere with, complicate, and disrupt editing. If geolocation is stored with anonymized IP addresses for GETs but not POSTs or Visual Editor submissions, both could easily be recovered because of simultaneously interleaved GET and POST requests for the same article are unavoidable. Do we have any privacy experts on staff who can give these issues a thorough analysis in light of all the issues raised in https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1450006 ? If Ops needs IP addresses, they should be able to use synthetic POST requests, as far as I can tell. If they anticipate a need for non-anonymous GET requests, then perhaps some kind of a debugging switch which could be used on a short term basis where an IP range or mask could be entered to allow matching addresses to log non-anonymously before expiring in an hour would solve any anticipated need? _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics