On Wed, Jul 10, 2013 at 9:54 PM, Noneof MicrosoftsBusiness <phonenumberofthebeast(a)hotmail.com> wrote: but day. advice let I think at this point that would either require a change to the format of the domas (anonymized) stats or an NDA and maybe some other approvals. (or kraken! but rumor is that's not yet ready for the general public) -Jeremy _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Hmm, This one really has me stumped: http://stats.grok.se/en/latest90/Yahoo! That is not a wikibump, but some sort of structural thing. The only thing I can think of is that some sort of popular band, manga character, or porn queen in China has been named Yahoo!

Hi, On 11.07.2013, at 10:37, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote: Yupp, i also think this is a weird anomaly… the causes can be very weird though, as we found out back in 2010-11 when the views to the "initial" page were suddenly very skewed: http://infodisiac.com/blog/2010/11/page-views-anomaly-in-october-resolved/#… Domas was able to find the cause by sampling some of the requests and found that all had the same referrer. Turned out it was an online ads page that had an error in their html which tried to load the page as background image. It's hard to analyze / clean this stuff while maintaining privacy. I remember there was a survey sent out to several (linked) open data researchers a while ago how the Wikimedia foundation could provide better stats. My reply was something along these lines: Provide more stats with every line in the hourly pageview stats: - how many different IP addresses cause the accesses (better: how many accesses per IP address (avg + stddev)) - how many different referrers cause the accesses (better: how many accesses per referrer (avg + stddev)) - how many accesses come from wikimedia IPs (toolserver, some bots) - ip address count for top 5 (or 10) originating countries (get some geolocation in) I think the top 3 aren't really computationally expensive but would really improve our ability to clean the view stats. The 4th always was on my wishlist, but would require some more work for reverse ip->geolocation lookup. Cheers, Jörn _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Is there a way to do this? Forgive me, I'm not exactly computer-illiterate, but this under-the-hood stuff is not something I'm familiar with.

Jeremy,**** ** ** Some background:**** ** ** So we are talking about search engine crawlers here, right?**** ** ** Here are most active crawlers:**** http://stats.wikimedia.org/wikimedia/squids/SquidReportCrawlers.htm**** for Google there is special page with more depth:**** http://stats.wikimedia.org/wikimedia/squids/SquidReportGoogle.htm**** * * It's been a long standing request to filter crawler data from page views. **** We almost did it a year ago, and planned to have two sets of counts in Domas' files (one with crawlers included, one without).. **** I'm not sure what came in the way. Diederik can tell you more about that, and current status.**** ** ** It would cut our page views by about 20%. **** The test we planned to implement is pretty simple: test 'user agent' field for 'crawler/spider/bot/http' reject if any occurs.**** Note user agent string is completely unregulated, but an informal rule is to include url only on crawler requests. **** ** ** BTW crawlers 'bots' not to be confused with Mediawiki bots:**** http://stats.wikimedia.org/EN/BotActivityMatrixEdits.htm**** http://stats.wikimedia.org/EN/BotActivityMatrixCreates.htm**** ** ** Erik Zachte**** ** ** ** ** *From:* analytics-bounces(a)lists.wikimedia.org [mailto: analytics-bounces(a)lists.wikimedia.org] *On Behalf Of *Toby Negrin *Sent:* Thursday, July 11, 2013 6:55 AM *To:* A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics. *Subject:* Re: [Analytics] Wikipedia Top 25**** ** ** We have some bot information from wikistats here<http://stats.wikimedia.org/#bots>ts>. I don't think it's particularly actionable for what you are doing, but it might be interesting directionally.**** ** ** -Toby**** ** ** On Wed, Jul 10, 2013 at 3:09 PM, Jeremy Baron &lt;jeremy(a)tuxmachine.com&gt; wrote:**** On Wed, Jul 10, 2013 at 9:54 PM, Noneof MicrosoftsBusiness &lt;phonenumberofthebeast(a)hotmail.com&gt; wrote: but day. advice let I think at this point that would either require a change to the format of the domas (anonymized) stats or an NDA and maybe some other approvals. (or kraken! but rumor is that's not yet ready for the general public) -Jeremy _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics**** ** ** _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics