Re: [Wiki-research-l] 2012 top pageview list

I got a couple of private replies to this thread, so I figured I would just answer them publicly for the benefit of the list: (1) Do I only parse/store English Wikipedia? Yes; for scalability reasons and because that is my research focus. I'd consider opening my database to users with specific academic uses, but its probably not the most efficient way to do a lot of computations (see below). Plus, I transfer the older tables to offline drives, so I probably only have ~6 months of the most recent data online. (2) Can you provide some insights into your parsing? First, I began collecting this data for the purposes of: http://repository.upenn.edu/cis_papers/470/ Where I knew the revision IDs of damaging revisions and wanted to reason about how many people saw that article/RID in its damaged state. This involved storing data on EVERY article at the finest granularity possible (hourly) and then assuming uniform intra-hour distributions. See the URL below for my code (with the SQL server credentials blanked out) that does this work. A nightly [cron] task fires the Java code. It goes and downloads an entire days worth of files (24) and parses them. These files contain data for ALL WMF projects and languages, but I use a simple string match to only handle "en.wp" lines. Each column in the database represents a single day and contains a binary object wrapping (hour, hits) pairs. Each table contains 10 consecutive days of data. Much of this design was chosen to accommodate the extremely long tail and sparseness of the view distribution; filling a DB with billions of NULL values didn't prove to be too efficient in my first attempts. I think I use ~1TB yearly for the English Wikipedia data. I would appreciate if anyone ends up using this code that my original work above would get a cite/acknowledgement. However, I imagine most will want to do a bit more aggregation, and hopefully this can provide a baseline for doing that. Thanks, -AW CODE LINK: http://www.cis.upenn.edu/~westand/docs/wp_stats.zip On 12/29/2012 11:06 PM, Andrew G. West wrote: -- Andrew G. West, Doctoral Candidate Dept. of Computer and Information Science University of Pennsylvania, Philadelphia PA Email: westand(a)cis.upenn.edu Website: http://www.andrew-g-west.com