Hi, [ I discussed this issue with Ironholds some weeks back. Also he reached out to Ops about this some more weeks back. So I guess he can chime in with more details. Only chiming it with my half-knowledge, as kevinator asked me to ] Warning: Longish-email :-/ On Tue, Jan 20, 2015 at 10:33:01PM +0530, Ananth RK wrote: That looks like (if X-Forwarded-For is “valid”) your jumping straight to considering the X-Forwarded-For header. You might want to check beforehand whether the client ip is a “trusted” proxy. And only if it is, you should walk the X-Forwarded-For. Also, it sounds a bit like you're going through X-Forwarded-For from left to right. Make sure to walk it from right to left, as proxies are expected to append (not prepend) the client IP they forward for. The way you combine the looping and if looks like you might be skipping across entries that you cannot parse. But if you find invalid entries while backtracking the IPs, you're probably in parts of the X-Forwarded-For value that you shouldn't trust. (X-Forwarded-For can easily be spoofed by clients) Handwavy pseudo-code only to illustrate the general pattern: list_of_ips = append client_ip to X-Forwarded-For for ip in reverse( list_of_ips ) do if ip is not a trusted proxy or the iterator does not have more elements then return ip fi done (That should give invalid IP addresses for some requests. That seems to be the correct thing from my point of view. But if you rather geolocate wrong than not geolocate at all, you can choose to pick the last known good IP address instead. I guess that's more a matter of taste.) Different stakeholders at the WMF use different lists of what kind of proxies they consider. So things are not really clear cut. (Do not treat the below list as authorative or complete. It's merely a collection of the pieces I know. And I did not check since quite some time. So it might be horribly outdated.) * General In general, I'd say one can assume that WMF-servers do not append bogus data to X-Forwarded-For. So when backtracking IPs, you can consider the hosts in $all_networks from https://git.wikimedia.org/blob/operations%2Fpuppet.git/9f97e3c2c5bc012ba5c3… as proxy, if they claim to have forwarded the request. One might handle Labs special in there. Not sure. * MediaWiki MediaWiki considers $wgSquidServersNoPurge https://git.wikimedia.org/blob/operations%2Fmediawiki-config.git/af1cc64bb5… and https://git.wikimedia.org/blob/mediawiki%2Fextensions%2FTrustedXFF/ec180fbd… as proxies. * Wikipedia Zero Wikipedia Zero does not follow the MediaWiki pattern when tagging, but is stripping WMF servers and considers sets of IP addresses from https://zero.wikimedia.org/wiki/Zero:-NOKIAPROD https://zero.wikimedia.org/wiki/Zero:-NOKIAQA https://zero.wikimedia.org/wiki/Zero:-OPERA (that's a private wiki. :-/ ) as proxies. * Legacy analytics software Legacy analytics software often used hardcoded lists of IP addresses they considered proxies. Those lists became stale quickly. I'd just ignore legacy software and not model what they modeled. ------------------------------------------------------------ If you want for a simple solution that at least allows determining the IP that made the request to the WMF servers, and you do not need to model a behaviour of some given component, go for “General”. If you want to mimic MediaWiki as closely as possible, go for the “MediaWiki” item. If you care more about Wikipedia Zero, and carrier/geotagging there, go for the “Wikipedia Zero” item. If you are looking for a general purpose, versatile approach, add a parameter that allows to select between the different strategies to resolve X-Forwarded-For. Yes, that sucks :-/ Have fun, Christian P.S.: Note that the repositories that the above links point to are not fully static. While their proxy configuration does not change every other day, they do change every now and then. -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

I'd actually suggest just moving them to the site-wide config files we use for all other recognised proxies.

I don't grok that message, I'm afraid; there's a config list of "these IPs are IPs we accept as trusted XFF providers": it's used in CheckUser and similar tools to avoid ending up with everyone geolocating to Virginia. I'm suggesting zero IP ranges that are providing XFFs get added to that so that there's only one thing to maintain and only one thing to check. On 22 January 2015 at 09:39, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote: to to -- Oliver Keyes Research Analyst Wikimedia Foundation _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

IF proxy_ip does not start with "127.0" or "192.168" or "10." There are other special/private IP blocks that you'll probably want filtered. This RFC contains the full list for IPv4: https://tools.ietf.org/html/rfc5735#section-4 And the equivalent for IPv6: https://tools.ietf.org/html/rfc5156