Analytics

analytics@lists.wikimedia.org

2 participants
1825 discussions

udp2log added fields + geocoding
by Andrew Otto 20 Jun '12

20 Jun '12

Hiya all! We're getting ready to make some changes to the udp2log sources. The first change will be the addition of two new fields at the end of the log line: Accept-Language and X-Carrier. We also would like to consider enabling geocoding via udp-filter for more of the udp2log files. Erik Z, I've attached a some sample log lines of what these changes might generate. They include the two new fields and a geocoded and then anonymized IP address. (Note that the IPs in my sample were all 127.0.0.1, so they couldn't be properly geocoded. But, the format will be the same.) Can you check this out and tell Diederik and me if this will cause any problems with your scripts? Thanks! -Andrew Otto

1 0

Fwd: Profile of Facebook Data Science Team
by Dario Taraborelli 14 Jun '12

14 Jun '12

From a company with not exactly the same privacy standards or data mining needs as Wikimedia, but still an interesting read: http://www.technologyreview.com/featured-story/428150/what-facebook-knows/ Begin forwarded message: > Date: June 14, 2012 11:05:22 AM PDT > Subject: Profile of Facebook Data Science Team > Source: FlowingData > Author: Nathan Yau > > MIT Technology Review profiles the Facebook Data Science Team, described as a gathering of grad students at a top school and headed by Cameron Marlow, the "young professor." > > Back at Facebook, Marlow isn't the one who makes decisions about what the company charges for, even if his work will shape them. Whatever happens, he says, the primary goal of his team is to support the well-being of the people who provide Facebook with their data, using it to make the service smarter. Along the way, he says, he and his colleagues will advance humanity's understanding of itself. That echoes Zuckerberg's often doubted but seemingly genuine belief that Facebook's job is to improve how the world communicates. Just don't ask yet exactly what that will entail. "It's hard to predict where we'll go, because we're at the very early stages of this science," says Marlow. "The number of potential things that we could ask of Facebook's data is enormous." > > Related > > Facebook status updates: young people are self-centered and old ramble > Data Science is catching on > Taking a Look at Facebook Statistics from All Facebook > > Read more… >

1 0

Benchmarking Kraken
by Andrew Otto 08 Jun '12

08 Jun '12

Alright! We've got a 10 node CDH3 hadoop cluster set up. I am experimenting with (and learning about!) hadoop as we go. We plan on doing some benchmarking of CDH3 vs. DataStax Enterprise (and vs. CDH4?) on this cluster before we make decisions. Right now is playtime! I just added some notes to this Etherpad on some variable tweaking I will be doing. My new notes start at about line 187. (Can I link to a specific line in Etherpad?) I've also created a google spreadsheet where I am keeping track of my benchmarking runs. Let me know if need access to it. https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0AvpRkIqSY9hNdE… If anyone on this list (who's on this list, anyway?!) has some insight or experience with hadoop benchmarking, feel free to chime in. We'd love the help! Thanks all, -Andrew Otto

1 0

External links statistics
by Lars Aronsson 03 Jun '12

03 Jun '12

(Hi everybody! Apparently I was already on this list, but had forgotten, since the volume is so low. I'm a volunteer and a former board member of the Swedish chapter. I also run Project Runeberg, the Scandinavian e-text archive, runeberg.org.) Here at the Berlin hackathon, I've improved the script I wrote in December for compiling statistics on external links. My goal is to learn how many links Wikipedia has to a particular website, and to monitor this over time. I figure this might be intresting for GLAM cooperations. This is found in the external links table, but since I want to filter out links from talk and project pages, I need to join it with the page table, where I can find the namespace. I've tried the join on the German Toolserver, and it works fine for the minor wikis, but it tends to time out (beyond 30 minutes) for the ten largest Wikipedias. This is not because I fail to use indexes, but because I want to run a substring operation on millions of rows. Even an optimized query takes some time. As a faster alternative, I have downloaded the database dumps, and processed them with regular expressions. Since the page ID is a small integer, counting from 1 up to a few millions, and all I want to know for each page ID is whether or not it belongs to a content namespace, I can do with a bit vector of a few hundred kilobytes. When this is loaded, and I read the dump of the external links table, I can see if the page ID is of interest, truncate the external link down to the domain name, and use a hash structure to count the number of links to that domain. It runs fast and has a small RAM footprint. In December 2011 I downloaded all the database dumps I could find, and uploaded the resulting statistics to the Internet Archive, see e.g. http://archive.org/details/Wikipedia_external_links_statistics_201101 One problem though is that I don't get links to Wikisource, Wikiquotes this way, because they are not in the external links table. Instead they are interwiki links, found in the iwlinks table. My improvement here in Berlin is that I now also read the interwiki prefix table and the iwlinks table. It works fine. One issue here, is the definition of content namespaces. Back in December, I decided to count links found in namespaces 0 (main), 6 (File:), Portal, Author and Index. Since then, the concept of "content namespaces" has been introduced, as part of refining the way MediaWiki counts articles in some projects (Wiktionary, Wikisource), where the normal definition (all wiki pages in the main namespace that contain at least one link) doesn't make sense. When Wikisource, using the ProofreadPage extension, adds a lot of scanned books in the Page: namespace, this should count as content, despite these pages not being in the main namespace, and whether or not the pages contain any link (which they most often do not). One problem is that I can't see which namespaces are "content" namespaces in any of the database dumps. I can only see this from the API, http://en.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=namespa… The API only provides the current value, which can change over time. I can't get the value that was in effect when the database dump was generated. Another problem is that I want to count links that I find in the File: (ns=6) and Portal: (mostly ns=100) namespaces, but these aren't marked as content namespaces by the API. Shouldn't they be? Is anybody else doing similar things? Do you have opinions on what should count as content? Should I submit my script (300 lines of Perl)somewhere? -- Lars Aronsson (lars(a)aronsson.se) Aronsson Datateknik - http://aronsson.se Project Runeberg - free Nordic literature - http://runeberg.org/

1 0

Liberating data @ Wikipedia presentation at Strata webconference
by Diederik van Liere 15 May '12

15 May '12

Apologies for cross-posting. Hi all, This is a quick headsup that I will be doing a short presentation at the Strata Online Webconference Data that Matters that is titled "Liberating data @ Wikipedia". For more info: http://strataconf.com/strata-may2012?intcmp=il-strata-stny12-widget-link-to… Best, Diederik

2 1

Re: [Analytics] [glam] Analytics-related mailing list, and request for GLAM-related metrics docs
by Lori Phillips 11 May '12

11 May '12

Thanks Sumana, I'd compiled all of my GLAM analytics notes and thoughts into a Google Doc (noting that I'm not a tech-person, so it may not make much sense - but I just happen to speak to museum technologists a good bit and had insights to share). This was sent to the analytics team, but I never heard anything back. https://docs.google.com/document/d/1kptd2EUgZ5gSuh9PFP7SUl4wyL5JJGKp2eYxrQo… I'll now add it to this page, maybe it will get more visibility. On Fri, May 11, 2012 at 1:25 PM, Sumana Harihareswara <sumanah(a)wikimedia.org > wrote: > I know a lot of you want statistics, metrics, and analytics: > > https://lists.wikimedia.org/mailman/listinfo/analytics > > Diederik van Liere's inaugural post says it all: > > > Welcome to the the inaugural Analytics Mailing list email. > > > > Here all your analytics wishes comes true, > > > > > > so proposals, ideas, crazy ideas, crazy crazy ideas are welcome here! > > as long as we can count something it is welcome. > > So please feel free to join that, and to put > https://www.mediawiki.org/wiki/Analytics on your watchlist! Also check > out https://www.mediawiki.org/wiki/Analytics/Pageviews/GLAM . "The > purpose of this page is to gather all the different documents, > discussions and ideas related to GLAM Analytics and to define a list > with the MOST important metrics. Please feel free to add any missing > data, this is document is almost certain to be incomplete." > > -- > Sumana Harihareswara > Volunteer Development Coordinator > Wikimedia Foundation > > _______________________________________________ > glam mailing list > glam(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/glam > -- Lori Phillips Web Content Specialist | Wikipedian in Residence The Children's Museum of Indianapolis US Cultural Partnerships Coordinator Wikimedia Foundation 703.489.6036 | http://loribyrdphillips.com/

1 0

API + sha1
by Aaron Halfaker 10 May '12

10 May '12

Hey guys, I'm trying to use the sha1 hashes of Wiki content for the first time (woot! Props to D et al. for seeing it through) but I'm having some trouble actually getting them out of the API/databases. It looks like the checksums only go back to April 19th. Is this true of all pages? Is there any plan to propagate the metric backwards? -Aaron

5 7

Re: [Analytics] Using tab as delimiter instead of space in the log files
by Andrew Otto 10 May '12

10 May '12

(Moving this to analytics list, cause uhhh, why not?) Ok some more info! I was testing some of my changes to udp-filter on my test labs instance, and hey! whadyaknow! The test failed. One of the failures was due to the content type field not matching correctly. It looked like this: text/html; charset=UTF-8 And actually, my test didn't catch the charset portion of this, my regexp is matching via whitespace. It was the semi-colon in the text/html string that caused the test to fail. I started sleuthing, and determined that this only shows up in my logs if I am hitting mediawiki pages. I grepped mediawiki core, and found that there are tons of places that are manually setting the Content-Type header with the charset. So I'm pretty sure the setting of the value of this header is coming directly from Mediawiki. But, that isn't really our issue. Headers are allowed to have spaces in the values. Here's some example log output with spaces in headers. https://gist.github.com/2648312 Summary: Nginx: - Does not escape spaces in Content-Type - Does not escape spaces in Accept-Language - Does not escape spaces in any header (afaict*). Varnish: - Does not escape space in Content-Type + Escapes spaces in Accept-Langage + Escapes spaces in any header (afaict). Squid: ~ Removes charset from Content-Type header (?) + Escapes spaces in Accept-Language. + Escapes spaces in any header (afaict). The remaining question we need to answer: Do we want to patch source code for these guys in order to fix this problem? I'd rather not if we don't have to. Do we have to? Uhhh, dunno! -Ao *As Far As I Can Tell On May 9, 2012, at 11:42 AM, Erik Zachte wrote: > For scripts it makes no difference: split("\t") is fine > > For browsing lines in an editor: I can disable tab expansion (or pipe to sed first) > > For human readability of pasted log lines in e.g. mail, it's no big deal (just slightly more ambiguous than a printable char) > > So, I'm fine which whatever works best, inter-process wise. > > Downward compatibility is another issue, but subscribers to our logs could get tabs replaced by blanks before data are forwarded. > Anyway please don't change this overnight. > > -- > > *KISS* > > Then again. Diederiks filter could easily and cheaply change all spaces to underscores, then all tabs to spaces, and no-one would even have to think twice, or make any changes. > > Erik > > > > From: Diederik van Liere [mailto:dvanliere@wikimedia.org] > Sent: Wednesday, May 09, 2012 3:59 PM > To: Andrew Otto > Cc: Erik Zachte; Rob Lanphier > Subject: Re: Using tab as delimiter instead of space in the log files > > Even I thought 12 months ago, "how hard can it be to use spaces only as delimiters" in a server log line. And 12 months later I now *know* it is extremely hard and those spaces will keep coming back, and we will keep discarding those log lines because they do not match the expected number of fields. Right now, these space issues are giving me headaches to come up with reasonable WikipediaMobile app usage for the Mobile team. > > But think a couple of months ahead when we want to stream our data into the cluster, do we really want a gazillion if else if statements to make sure that the data is in the right format before we store in the cluster? This will create a huge performance problem. > > By keep fixing spaces we will always behind the facts, because it takes a while before we discover the issue, and then we need to make changes to the config and get it approved or we fix it for once and for all. > > Because the user agent info is freeform I think it's safest to go with a tab control character but I am open for other suggestions. > > Best, > D > > On Wed, May 9, 2012 at 9:40 AM, Andrew Otto <otto(a)wikimedia.org> wrote: > > we have a lot of software scattered all over the place processing > > this stuff (much of which is outside the purview of this group), and > > we're probably going to break at least some of it with this change. > If so ja, could be a problem. How widespread is this? We'd like to keep track of these uses going forward. > > >>> Spaces, tabs and any > >>> non-URL character in a URL needs to be escaped by > >>> Varnish/nginx/Squid/whatever > >> the source can be located within hours and > >> patched. > > I really don't think we should start patching server software (like we did with nginx user agent string) to replace spaces. > Diederik, I agree. If we can figure out how to always escape things properly in the log format config directives for each log source (nginx,squid,varnishncsa), then no problem. But patching and recompiling every time we find a new edge case like this would be really annoying and kinda dangerous. The more changes we make to upstream sources the more we have to remember to maintain and the more bug prone we make crucial software. > > Hmm, if we can figure out how to patch each of theses sources only once to make all whitespace characters in log fields escaped, then that might be ok. Mayyyybe, I dunno. What if we add a new source? Then we'd have to figure out how to patch it too. > > >> is there any printable > >> char that could be used? > Technically I'm sure we could find one, but I think we'd have more complaints about readability in emails and editors if we don't use whitespace than if we just used a tab. > > What if we used multiple spaces (2? 4? 5?) as a delimiter? I'm not sure all the log format config directives would allow us to, but this might be less prone to editor weirdness, and also might cause problems for fewer users. Then they can split(' '). Although, hm, I guess that isn't much better than split("\t"), eh? > > -A > > > > On May 9, 2012, at 9:21 AM, Diederik van Liere wrote: > > > > > The tab would be inserted by varnish/squid/nginx when writing the log, we don't need to worry about URL transport or anything like that. > > We keep running into new cases where spaces show up where we don't expect them, and I really don't think we should start patching server software (like we did with nginx user agent string) to replace spaces. > > That locks us into specific versions of the software and the alternative is easy and robust. > > > > Erik: I don't think that editor settings are a convincing argument against the use of tabs. > > > > I am going to send this proposal to Wikitech mailing list. > > > > > > D > > On 2012-05-09, at 7:29 AM, Erik Zachte wrote: > > > >> I'm not strongly pro or con but here are some thoughts: > >> > >> I always get confused by tabs as they will be presented differently > >> depending on editor settings (and different editors have different > >> defaults). > >> If someone pastes some log lines in an email, tabs might already have been > >> expanded. > >> > >> Also Diederik's new filter will validate the line, and count fields. > >> If it signals invalid lines, the source can be located within hours and > >> patched. > >> How many sources could there be? We have 3 streams? > >> In theory I am in favor of validating at the earliest point. > >> > >> -- > >> > >> I know this is tricky, but if we decide to change, is there any printable > >> char that could be used? > >> > >> There are some characters that are out-of-bounds for urls, but some of those > >> are unsafe for other reasons > >> http://www.ietf.org/rfc/rfc1738.txt 2.2 section par. 'unsafe:' > >> > >> <quote> > >> Other characters are unsafe because gateways and other transport agents are > >> known to sometimes modify such characters. > >> These characters are "{", "}", "|", "\", "^", "~", "[", "]", and "`". > >> <unquote> > >> > >> One that comes to mind, but may have other quirks is 127/7F/ > >> > >> Another 2 cents. > >> Erik > >> > >> > >> -----Original Message----- > >> From: Diederik van Liere [mailto:dvanliere@wikimedia.org] > >> Sent: Wednesday, May 09, 2012 3:33 AM > >> To: Rob Lanphier > >> Cc: Andrew Otto; Erik Zachte > >> Subject: Re: Using tab as delimiter instead of space in the log files > >> > >> a) Not sure why this wouldn't solve the problem, the problem is that we > >> consistently are encountering spaces in fields while we are also using space > >> as delimiter. By changing the delimiter we are no longer affected by spaces > >> in the fields. Varnish does not need to escape tabs, those tabs are > >> configured in the varnish/squid/nginx config files. Andrew, can you confirm > >> this? > >> > >> b) Who else is consuming the log files besides the analytics team and Erik > >> Zachte? And changing to a tab as delimiter is going to be a small change, so > >> yes it needs to be communicated to the people who consume these log files > >> but not fixing it or alternative writing all kinds of patches seems a worse > >> solution. > >> > >> > >> My 2 cents. > >> D > >> > >> On 2012-05-08, at 9:25 PM, Rob Lanphier wrote: > >> > >>> On Tue, May 8, 2012 at 6:06 PM, Diederik van Liere > >>> <dvanliere(a)wikimedia.org> wrote: > >>>> Instead of continuing patching our servers to fix these spaces issues I > >> strongly suggest that we move away from the space as delimiter and start > >> using the tab (\t) character. > >>> > >>> Eeek! I'm pretty sure that this will: > >>> a) not really solve the problem. If the Varnish isn't escaping > >>> spaces, what makes us think it's escaping tabs? Spaces, tabs and any > >>> non-URL character in a URL needs to be escaped by > >>> Varnish/nginx/Squid/whatever > >>> b) we have a lot of software scattered all over the place processing > >>> this stuff (much of which is outside the purview of this group), and > >>> we're probably going to break at least some of it with this change. > >>> > >>> Regardless, we'll need to have a much wider conversation about such a > >>> change than this group. > >>> > >>> Rob > >> > >> > > >

5 12

Account registration dashboards merged
by Dario Taraborelli 09 May '12

09 May '12

(cc'ing analytics for information) http://toolserver.org/~dartar/reg2/ • merged the new account registration dashboards and the old Y2Y comparison into a single report • added cronjob to refresh the Y2Y stats hourly

1 0

FYI: weekend work on locke
by Rob Lanphier 06 May '12

06 May '12

Hi folks, Dunno if there's a private thread on this already, but I figured it'd be good to utilize our underutilized public list for this. I see Ariel has done some work on locke, presumably to deal with some disk full problem that always seems to crop up at the worst time on the weekend (thank you Ariel for being on top of this). Quoting from the server admin log[1]: 07:03 apergos: manually rotates udplogs on locke, copying destined_for_storage3 off to hume:/archive/emergencyfromlocke/ (jeff, this note's for you in particular) Andrew, this is probably worth following up on when you get a chance. Thanks Rob [1] Server admin log: http://wikitech.wikimedia.org/view/Server_admin_log

2 1

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Analytics