Nuria, regarding the IP addresses specifically (not the proxy, for which, I'll need more time to go through the use-cases we've had and see if we can find work-arounds if we hash proxy information):

Have we considered in the past to create at least two levels of access when it comes to the IP addresses? From what you describe, it is clear to me that your team will need to have access to raw IPs for a certain period of time. It may be the case that no one else uses that information (for all of the use-cases of the research I've been involved in, hashed IP works as well, as long as we have geolocation available to us). By creating two layers of access, we can make sure that your team has access to raw IP while everyone else doesn't. Is this an option?

And one suggestion: if we want to reconsider the way we provide access to IP address, I'd like to suggest that we step back and reconsider the way we give access to other fields in the webrequest logs as well. This will be a longer process, but it may be worthwhile. For example, if we decide that access to raw IP should be limited even further, do we want to have the same restrictions applied to access to UAs? It's not obvious to me that the answer should be no.

Best,
Leila


On Fri, Nov 11, 2016 at 8:31 AM, Nuria Ruiz <nuria@wikimedia.org> wrote:
>I support any decrease of the storage of plain IP addresses. See also <https://www.mediawiki.org/wiki/Thread:Talk:Requests_for_comment/Structured_logging/IP_address_and_other_personal_identifying_information> for more references.

To be clear: on our end we need buffer time that allows us to know that should there be a bug we can reprocess pageviews if needed (this does happen). That buffer time is now 60 days and perhaps it could be a bit smaller but it is still going to be a matter of weeks, not days for which the raw data needs to be available. As mentione earlier in the thread we need raw IPs to geolocate requests, once that is done IPs are discarded. 



On Fri, Nov 11, 2016 at 12:00 AM, Federico Leva (Nemo) <nemowiki@gmail.com> wrote:
Dan Andreescu, 10/11/2016 16:00:
I don't have as clear a reason for why we store the plain IP in
webrequest.  I think we could count uniques and all that other stuff
with the IP hash.  It's a good question, tentative +1 unless I'm
forgetting something.

I support any decrease of the storage of plain IP addresses. See also <https://www.mediawiki.org/wiki/Thread:Talk:Requests_for_comment/Structured_logging/IP_address_and_other_personal_identifying_information> for more references.

Nemo


_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics


_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics