Analytics March 2014

analytics@lists.wikimedia.org

33 participants
29 discussions

Analytics Development Sprint Update
by Toby Negrin 10 Mar '14

10 Mar '14

Hi all, We finished our sprint on Tuesday and made plans for the next one on Thursday and I wanted to let you know the updates. We've been in the middle of annual planning, so I'm going to provide the update on Epics next week. We finished the two projects that we had committed to for the sprint: Fundraising Visualization Syncup<https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/1449> Measuring ULSFO Site Performance<https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/1450> We were also able to finish: Productionize Wikimetrics<https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/1185> We generated the dashboards for the March metrics meeting We wrote a utility (kraken-toolbelt) that allows read/write from/to snappy compressed hdfs files We fixed the following defects: Adapt to tag change in comScore csv files<https://bugzilla.wikimedia.org/show_bug.cgi?id=62044> Merge editors stats for all wikis failed No sampled-1000 tsv file for 2014-02-06 on stat1002<https://bugzilla.wikimedia.org/show_bug.cgi?id=60955> We also worked on, but did not finish a few other issues: Run Scheduled Reports via Wikimetrics<https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/1376> Unit testing Hive queries Kafka-Tee testing/deployment For the new sprint, we needed to pivot and migrate our existing services to the new labs cluster in the Virginia datacenter. We've tasked this out, but there are too many unknowns and possible dependencies to properly estimate, so we will work on this until it's done. -Toby

1 0

drop in pageviews from Jan 2013 versus Jan 2014...
by Lane Rasberry 07 Mar '14

07 Mar '14

Hello, I watch pageview traffic to about 100 popular Wikipedia health articles. In January 2014 as compared to January 2013, stats.grok.se reports about a 50% drop in pageviews. Does someone have a ready explanation for this? The obvious explanation would be that Wikipedia's traffic has halved, which does not seem right to me, but it could be so. Can someone direct me to previous or current conversations about this? yours, -- Lane Rasberry user:bluerasberry on Wikipedia 206.801.0814 lane(a)bluerasberry.com

3 2

Phasing out stat1001's mobile_device_props-daily.tsv, and mobile_platform-daily.tsv
by Christian Aistleitner 06 Mar '14

06 Mar '14

Hi, with the Analytics Development team's decision to move from dclass to ua-parser, life support for http://stat1001.wikimedia.org/public-datasets/analytics/mobile/mobile_devic… http://stat1001.wikimedia.org/public-datasets/analytics/mobile/mobile_platf… would become a burden. As we additionally do not know of any user of those files up to now, we are considering to stop generating those two files [1]. If you are using any of the above two files, please let us know by 2014-03-05, so we can discuss how to move forward. Best regards, Christian [1] It's really only those two files listed above. We keep on generating all the other mobile files as for example the mobile-sampled-100 as usual for the time being. -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- OpenPGP key transition from 0xEF78CCDE to 0x13C1072F: http://quelltextlich.at/openpgp-transition-0xEF78CCDE-to-0x13C1072F.txt

1 1

Analytics participation in GSoC
by Aaron Halfaker 05 Mar '14

05 Mar '14

Hey folks, I was wondering if WMF analytics has ever participated in Google Summer of Code and/or if there is interest in participating in the near future. See: - https://www.mediawiki.org/wiki/Google_Summer_of_Code_2014 - https://www.mediawiki.org/wiki/Mentorship_programs/Possible_mentors -Aaron

3 2

Discrepancy in edit count
by Denny Vrandečić 04 Mar '14

04 Mar '14

Hi, this table: http://stats.wikimedia.org/wikispecial/EN/TablesWikipediaWIKIDATA.htm#editd… seems to think that there have only been 15,238,669 edits by non-IPs in the article namespace on Wikidata. Considering Wikidata crossed 100M edits overall, I would be extremely suprised if either IPs or non-article edits would make up the difference. Can someone check what is going on there? Especially with regards to how trustworthy the rest of the linked table is? Cheers, Denny

2 1

Kafkatee files
by Andrew Otto 04 Mar '14

04 Mar '14

Yoyo, Who generates the mobile and zero limn dashboards? I am currently generating log files from Kafka that should be useable in place of the ones that udp2log generates. It’d be cool to compare the output of the two. Christian has already done a little bit of comparison, now it’d be nice to see the output used for something! :)

2 3

[abogott@wikimedia.org: [Wikitech-l] Labs migration starts on Tuesday]
by Christian Aistleitner 04 Mar '14

04 Mar '14

Hi, it seems that actions on our end are necessary within a month for our labs instances to survive the data center migration (see below). I am all for migrating our instances (at least limn0, as it is required for Wikipedia Zero). But since I cannot find a Mingle card for the labs migration, and it was not brought up before: Do we want to migrate our instances? Best regards, Christian ----- Forwarded message from Andrew Bogott <abogott(a)wikimedia.org> ----- To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>, "A list for announcements and discussion related to the Wikimedia Labs project." <labs-l(a)lists.wikimedia.org> From: Andrew Bogott <abogott(a)wikimedia.org> Date: Fri, 28 Feb 2014 23:59:00 +0800 Subject: [Wikitech-l] Labs migration starts on Tuesday User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 List-Id: Wikimedia developers <wikitech-l.lists.wikimedia.org> Starting on Tuesday, March 4th, the new Labs install in the eqiad data center will be open for business. Two dramatic things will happen on that day: Wikitech will gain the ability to create instances in eqiad, and wikitech will lose the ability to create new instances in pmtpa. About a month from Tuesday, the pmtpa labs install will be shut down. If you want your project to still be up and running in April, you must take action! We are committed to not destroying any instances or data during the shutdown, but projects that remain untouched by human hands during the next few weeks will be mothballed by staff: the data will be preserved but most likely compressed and archived, and instances will be left in a shutdown state. (Note: Toollabs users can sit tight for a bit; Coren will provide specific migration instructions for you shortly.) I've written a migration guide, here: https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration_Howto It's a work in progress, so check back frequently. Please don't hesitate to ask questions on IRC, make suggestions as to guide improvements, or otherwise question this process. Quite a few of the suggested steps in that guide require action on the part of a Labs op -- for that purpose we've created a bugzilla tracking bug, 62042. To add a migration bug that links to the tracker, use this link: https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs&compo… At the very least, please visit this page and edit it with your project migration plans: https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration_Progress Projects that have no activity on that page will be early candidates for mothballing. If you want me to delete your project, please note that as well -- that will allow us to free up resources for future projects. I am cautiously optimistic about this migration. Most of our testing has gone fairly well, so a lot of you should find the process smooth and easy. That said, we're all going to be early adopters of this tech, so I appreciate your patience and understanding when inevitable bugs shake out. I look forward to hearing about them on IRC! -Andrew _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ----- End forwarded message ----- -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

2 1

If it did not happen on the list
by Christian Aistleitner 03 Mar '14

03 Mar '14

Dear fellow Analytics Developer team members, over the past few weeks, it seems we at least twice discussed that maybe we want to adopt “If it didn't happen on the list, it didn't happen”. It appeared to me that both times, we actually wanted to move forward with that ... but it never got posted to the mailing list :-D So I'm being bold: Either slap me with a large trout by 2014-02-28 or agree with me that the Analytics development team adopts “If it didn't happen on the list, it didn't happen.” (With “list” being analytics(a)lists.wikimedia.org for public things, and “list” being analytics-internal(a)lists.wikimedia.org for not so public things.) To state the obvious: meetings in real life, hangouts, or personal email do not qualify as “list” :-) Have fun, Christian /me hands a large trout to all of you to slap me with and ducks. -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- OpenPGP key transition from 0xEF78CCDE to 0x13C1072F: http://quelltextlich.at/openpgp-transition-0xEF78CCDE-to-0x13C1072F.txt

4 6

Update on Analytics Development Epics
by Toby Negrin 03 Mar '14

03 Mar '14

Hi Everyone, The development team is in the middle of the sprint that I sent you an update on last week; At the end of next week I'll update this list with what we accomplished during the sprint. For this week, here is the current status of the Epics<https://www.mediawiki.org/wiki/Analytics/Prioritization_Planning>we are working on: Context ====== List of Epics is here<https://www.mediawiki.org/wiki/Analytics/Prioritization_Planning> . Analytics Mingle board is here<https://wikimedia.mingle.thoughtworks.com/projects/analytics/overview> Epics ==== Production Issues - Resolved on No sampled-1000 tsv file for 2014-02-06 on stat1002<https://bugzilla.wikimedia.org/show_bug.cgi?id=60955> - Wikipedia Zero team investigated ~30% increase of number of lines zero tsvs between 20140218 and 20140220 file<https://bugzilla.wikimedia.org/show_bug.cgi?id=61660> - Wikipedia Zero team investigated on light drop in zero requests around 2014-02-08 <https://bugzilla.wikimedia.org/show_bug.cgi?id=61274> - Data for ULSFO Cache performance<https://trello.com/c/dNODZnr0/108-measuring-ulsfo-s-impact-on-site-performa…>prepared for Ops blog post Metric Definition Standardization - Current deliverables from Research and Data (New Editor Metrics) and Development (Page Views) internally reviewed and waiting start of community review process Accurate Pageviews for Wikipedia Zero - Dashboards updated to cover all active partners - Port of legacy processing scripts to Hadoop prioritized and will begin next sprint Editor Engagement Vital Signs - Sync-up meeting confirmed metrics for first release (New Editors) finished by development - WIkimetrics changes to support scheduled queries under development - Met with FR Tech to discuss results of visualization library review/Wikimetrics -- loosely coupled collaboration seems likely on both. Metrics about Mobile Usage - Prototype of mobile browser report running on unsampled mobile data on Hadoop - Decision made to use UA Parser <https://github.com/tobie/ua-parser> to parse out sanitized UA information for above report - Worked on implementing Hive unit tests<https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/1293> Simplify Limn Dashboard Deployment - Proposal from research to simplify dashboard creation for certain "counting' use cases reviewed; will be assessed for next sprint. Page View API - We have specs for new server from Henrik for stats.grok.se; Currently working with vendor to purchase. - Worked on implementation of new JVM deployment system<https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/1374> - Worked on puppetizing Camus kafka-HDFS integration tool<https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/1313> - Worked on Kaftatee to support legacy analytics systems<https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/1425> Please let me know if you have any questions -Toby

2 2

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Analytics March 2014