Analytics October 2014

analytics@lists.wikimedia.org

44 participants
43 discussions

Wikimedia Research showcase – October 15 2014, 11.30 PT
by Dario Taraborelli 17 Oct '14

17 Oct '14

After a break in September, we’re resuming our monthly Research and Data showcase. The next showcase will be live-streamed tomorrow Wednesday October 15 at 11.30 PT. As usual you can join the conversation via IRC on freenode.net by joining the #wikimedia-research channel. We look forward to seeing you there, Dario This month: Emotions under Discussion: Gender, Status and Communication in Wikipedia By David Laniado: I will present a large-scale analysis of emotional expression and communication style of editors in Wikipedia discussions. The talk will focus especially on how emotion and dialogue differ depending on the status, gender, and the communication network of the about 12000 editors who have written at least 100 comments on the English Wikipedia's article talk pages. The analysis is based on three different predefined lexicon-based methods for quantifying emotions: ANEW, LIWC and SentiStrength. The results unveil significant differences in the emotional expression and communication style of editors according to their status and gender, and can help to address issues such as gender gap and editor stagnation. Wikipedia as a socio-technical system By Aaron Halfaker: Wikipedia is a socio-technical system. In this presentation, I'll explain how the integration of human collective behavior ("social") and information technology ("technical") has lead to a phenomena that, while being massively productive, is poorly understood due to lack of precedence. Based on my work in this area, I'll describe five critical functions that healthy, Wikipedia-like socio-technical systems must serve in order to continue to function: allocation, regulation, quality control, community management and reflection. Next I'll argue the Wikimedia Foundation's analytics strategy currently focuses on outcomes related to a relatively narrow aspect of system health and all but completely ignores productivity. Finally, I'll conclude with an overview of three classes of new projects that should provide critical opportunities to both practically and academically understand the maintenance of Wikipedia's socio-technical fitness.

9 16

Mobile page views available
by Toby Negrin 16 Oct '14

16 Oct '14

Hi all -- I have some good news to share. At the beginning of the month, we announced that mobile page views were available on our servers. Somewhat belatedly, I have follow up information available here: <goog_271963740> https://wikitech.wikimedia.org/wiki/Analytics/Pagecounts-all-sites Thanks to the good work of the analytics development and WMF operations team who made this happen. We are in process of reaching out to developers who consume this data to let them know about the new stream. We have a follow up question -- we are considering leaving up the original stream up until the end of the quarter. Does this seem reasonable? It is a maintenance burden as it is created via different infrastructure. thanks, -Toby

7 7

Re: [Analytics] [Ops] POODLE SSL 3.0 vulnerability
by Mark Bergsma 16 Oct '14

16 Oct '14

(+CC: Analytics) On Wed, Oct 15, 2014 at 8:54 AM, Giuseppe Lavagetto <glavagetto(a)wikimedia.org> wrote: > So, the somewhat-anticipated new SSL vulnerability is out, CVE-2014-3566. > To make the long story short: > - - SSL v 3.0 operating with CBC-mode chiphers is vulnerable to this > attack and could allow an attacker to get plaintext info on the > traffic being exchanged with the server > - - It's fairly easy for an attacker to induce a downgrade from more > modern version of TLS to SSL 3.0 > > There are some patches that would eliminate the downgrading issues > *for chrome users only at the moment*, but I'm not that happy with the > idea of patching openssl and maintaining the patch. > > Another possibility would be to force use of RC4 in SSL 3.0, which as > Brandon put it on IRC is almost as using rot13. > > So, the easiest (and best) way of getting rid of this vulnerability > (and a bunch of others, to be honest) would be to drop SSL 3.0 > support. That would mean dropping HTTPS support for IE6 users, which > is a decision we can't make lightly, but keeping SSL 3.0 exposes the > vast majority of our users to this vulnerability. > > What should we do? This is not as serious as heartbleed but shouldn't > be taken lightly anyway. So to sum it up again: SSL 3.0 is vulnerable/weak, enabling RC4 doesn't help much. Keeping SSL 3.0 enabled with RC4 for clients that only support SSL 3.0 (mostly IE 6 and below it seems) -might- be acceptable, if we could warn those users when they use it, and if we can prevent every other browser from downgrading to it. But that doesn't seem possible at the moment; Google's TLS_FALLBACK_SCSV needs all clients and servers patched, which will obviously take a long while and has maintainability issues. Also depends on whether OpenSSL and/or Debian will incorporate this nonstandard patch... at least this affects a lot of people. Disabling SSL 3.0 could break SSL completely for something in the order of 1.5% of HTML page requests, according to http://stats.wikimedia.org/wikimedia/squids/SquidReportClients.htm Perhaps the Analytics team could do some additional, specific investigation on this to aid this decision? I think if we can't allow secure HTTPS by keeping SSL 3.0 enabled, we should probably disable it even if it breaks a small number of requests. 1.5% doesn't seem insignificant though. Obviously we'd need to communicate that very well, as we don't seem to have good options for doing any sort of fallback to HTTP. I guess it also depends on what the public awareness of this issue will be...

11 21

Dropping requests for [uU]ndefined from webstatscollector's page view definition
by Christian Aistleitner 16 Oct '14

16 Oct '14

Hi, recently, we again saw a few million requests to the 'undefined', and 'Undefined' pages. While some requests for Undefined are real proper requests, by far most of them seem to stem from gone-wrong-JavaScript. (See bug 66352 [1]) While the fact that we there is gone-wrong-JavaScript is an issue on it's own, I am preparing patches to drop [uU]ndefined from the webstatscollector pageview definition [2], and hence no longer count them in the pagecounts-raw [3] and pagecounts-all-sites [4] files. If you think it's worth keeping them, please let us know soonish. (I'll try to get the changes merged on 2014-10-09.) Have fun, Christian [1] https://bugzilla.wikimedia.org/show_bug.cgi?id=66352 [2] https://wikitech.wikimedia.org/wiki/Analytics/Webstatscollector#Used_Page_V… [3] https://wikitech.wikimedia.org/wiki/Analytics/Pagecounts-raw http://dumps.wikimedia.org/other/pagecounts-raw/ [4] https://wikitech.wikimedia.org/wiki/Analytics/Pagecounts-all-sites http://dumps.wikimedia.org/other/pagecounts-all-sites/ -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

1 2

Reportcards
by Jon Robson 16 Oct '14

16 Oct '14

Just curious... I just discovered http://reportcard.wmflabs.org/ which somehow I had no idea existed. Is there any reason it doesn't link to http://mobile-reportcard.wmflabs.org and vice versa and why there is a separate mobile report card instance?

4 5

Adventures in Clusterland 2014-10-06--2014-10-12
by Christian Aistleitner 15 Oct '14

15 Oct '14

Hi, in the week from 2014-10-06–2014-10-12 Andrew, Jeff, and I worked on the following items around the Analytics Cluster and Analytics related Ops: * ULSFO outage affecting webrequest logs (Bug 71876, Bug 71879) * Revoked default Push grant for Analytics on gerrit's analytics/* projects * Wikimetrics showing many requests to internal files * Counting pageviews for the pages “undefined” / “Undefined” (Bug 66532) * Counting redirect pageviews for Webstatscollector (Bug 71790) * Reworking webstatscollector's build system * Puppetization of MaxMind's Connection Type databases * Wikihadoop now available on the Analytics Cluster * Analytics Mini-Hackathon in San Francisco (details below) Have fun, Christian * ULSFO outage affecting webrequest logs (Bug 71876, Bug 71879) It seems there have been connection issues from ULSFO, which caused a minor hiccup in the webrequest logs on both udp2log and kafka [1]. Due to kafka's buffering, kafka could nicely bridge the shorter dropouts, and in total only a few minutes of data have been lost on kafka, while udp2log was shaky for up to 2 hours. * Revoked default Push grant for Analytics on gerrit's analytics/* projects Per default, all Analytics members had Push permission on all of gerrit's analytics/* project. As accidental pushes caused pain again, we now revoked the default Push grant, and made sure that our bots still had necessary permission to do their duty. * Wikimetrics showing many requests to internal files A fix for the mis-redirection of those monitoring requests has been implemented (but it's not yet deployed). * Counting pageviews for the pages “undefined” / “Undefined” (Bug 66532) A short increase on requests for the pages “undefined” and “Undefined” impacted pageview trend graphs. So after the initial push-back that bug 66532 received, it was picked up again, and we prepared patches for both the C and Hive implementation of webstatscollector's pageview definition to not count such requests. Deployment of those patches is likely to happen around 2014-10-15. * Counting redirect pageviews for Webstatscollector (Bug 71790) Ever since, the webstatscollector pageview definition has been counting redirects, and was hence overcounting. Since, we're about to deploy a webstatscollector anyways, we prepared changes to fix this longstanding miscounting. * Reworking webstatscollector's build system Fresh compilations of webstatscollector's C implementation gave executables that segfaulted. So we fixed some NULL dereferences, fixed the build system, made it capable of compiling with optimization turned on, and built a rudimentary testsuite for the collector process. Thereby, we can now again build the collector executable, and can automatically verify that it's working. * Puppetization of MaxMind's Connection Type databases MaxMind's Connection Type (NetSpeed) databases have been puppetized. They are available for example on stat1002, and stat1003 at /usr/share/GeoIP/GeoIPNetSpeedCell.dat /usr/share/GeoIP/GeoIPNetSpeed.dat . * Wikihadoop now available on the Analytics Cluster This allows for easier parsing of Mediawiki xml revision dumps. * Analytics Mini-Hackathon in San Francisco During this week, the Analytics Mini-Hackathon took place, and more prototyping around ** Scoop and Oozification ** Streaming data into HDFS happened, and some time was spend on hunting down the kafkatee issues. -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

1 0

Adventures in Clusterland 2014-09-29--2014-10-05
by Christian Aistleitner 15 Oct '14

15 Oct '14

Hi, in the week from 2014-09-29–2014-10-05 Andrew, Jeff, and I worked on the following items around the Analytics Cluster and Analytics related Ops: * LDAP server renaming * Esams bits having problems to produce to kafka * Wikimetrics showing many requests to internal files * Released new dataset pagecounts-all-sites (details below) Have fun, Christian * LDAP server renaming WMF changed its LDAP server names. For production machines, this was handled nicely through puppet. Same for labs instances that run of puppet head. But many of our instances fall in neither of those two categories. So we had to bring in the necessary changes by hand, and make sure nothing broke. * Esams bits having problems to produce to kafka (Bug 71435) We were seeing both duplicates and missing log lines from esams bits caches on 2014-10-28 and 2014-10-29. It appeared that big caches for Kafka and our ACK setup back-fired upon too high load. So Kafka caching got adjusted a bit, and we took a closer look at the ACKs handling. This problem might occur again, so we might do further adjustments there. * Wikimetrics showing many requests to internal files (Bug 71606) Reoccurring requests to internal resources of Wikimetrics have been found in the logs. They turned out to be mislead requests from monitoring of puppet's apache default setup. * Released new dataset pagecounts-all-sites We released the first Analytics cluster generated dataset pagecounts-all-sites. Announcement is at https://lists.wikimedia.org/pipermail/analytics/2014-October/002597.html -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

1 0

Happy Ada Lovelace Day
by Pine W 15 Oct '14

15 Oct '14

Ada Lovelace Day is celebrated on October 14 this year. Augusta Ada King, Countess of Lovelace (born in the year 1815) was a mathematician and computer programmer who worked on Charles Babbage's Analytical Engine. She foresaw how computers could evolve into devices that perform tasks more sophisticated than simple calculations. She is controversially credited with authoring the world's first computer program, and certainly worked extensively with Babbage. [1] Ada Lovelace Day celebrates women's contributions to science, technology, engineering, and mathematics. Wikimedia Commons, English Wikipedia, and Persian Wikipedia have designated a watercolor portrait of Lovelace as a featured picture. [2] Happy Ada Lovelace Day, Pine [1] https://en.wikipedia.org/wiki/Ada_Lovelace [2] https://commons.m.wikimedia.org/wiki/File:Ada_Lovelace_portrait.jpg

1 0

Analytics services reliability question
by Pine W 13 Oct '14

13 Oct '14

I've noticed in my time on this list that it seems like the reliability of Wikimedia Analytics services is a bit spotty. I've worked in IT services, and our web and email servers' reliability seemed pretty good, comparable to the Wikimedia content delivery services. I'm curious if there is something about the nature of Analytics services that makes them inherently fragile, or if there is something about Wikimedia's particular configuration that is an issue. This isn't intended as criticism; I'm just curious. Thanks for your work keeping this place running. Pine

5 7

Fwd: [Wikitech-l] Tech Talk: The Dashboarding Problem: October 6
by Pine W 11 Oct '14

11 Oct '14

Thanks for this. I'm forwarding to the Analytics and Research lists. Pine ---------- Forwarded message ---------- From: Rachel Farrand <rfarrand(a)wikimedia.org> Date: Mon, Oct 6, 2014 at 1:12 PM Subject: Re: [Wikitech-l] Tech Talk: The Dashboarding Problem: October 6 To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org> Thank you for the great turnout today! If you would like to view the recording of the talk, here is the link: http://www.youtube.com/watch?v=hzMwwLfvh5g If you have any questions about today's talk please feel free to get in touch with Dan Andreescu <dandreescu(a)wikimedia.org> and Nuria Ruiz < nuria(a)wikimedia.org> You can check out past tech talk recondrings at the MediaWiki YouTube page here: http://www.youtube.com/channel/UCg4wlhlN8RjP6_e_vMC4CTA If you would like to nominate future tech talks or see what we have coming up, go here: https://www.mediawiki.org/wiki/Project:Calendar/How_to_schedule_an_event/Te… Thanks! On Mon, Oct 6, 2014 at 11:03 AM, Rachel Farrand <rfarrand(a)wikimedia.org> wrote: > Reminder: This tech talk starts in 1 hour > > On Wed, Oct 1, 2014 at 12:01 PM, Rachel Farrand <rfarrand(a)wikimedia.org> > wrote: > >> Please join us for the following tech talk: >> >> Tech Talk: *The Dashboarding Problem* >> Date: October 6 >> Time: 1900 UTC >> < http://www.timeanddate.com/worldclock/fixedtime.html?msg=Tech+Talk%3A+The+D… > >> Link to live YouTube stream <http://www.youtube.com/watch?v=hzMwwLfvh5g> >> IRC channel for questions: #wikimedia-office >> Google+ page >> < https://plus.google.com/u/0/b/103470172168784626509/events/ch8uuivq05nqejql…>, another >> place for questions >> >> Talk description: >> The Analytics team has been busy exploring dashboarding and visualizing >> editor engagement data. We found that while most people focus on >> visualization, data access and information architecture are just as >> important and separate problems. >> Mike Bostock solved visualization and the design team took care of >> information architecture, so we built a dashboard around their work. >> In this talk we share our learnings from developing dashiki, our new >> dashboard stack. We will talk about why we believe a server-less javascript >> app was the right architecture for the problem, how with about 900 lines of >> javascript we transform data into Vega grammar, and how knockout components >> helped us stay modular. >> >> While we'll look at some javascript, the talk is high level, about 30 >> minutes long, and everyone that is interested in dashboarding, >> visualization, and modularity is welcome to attend. >> >> Dashiki Code: https://github.com/wikimedia/analytics-dashiki >> >> Editor Dashboard: https://metrics.wmflabs.org/static/public/dash/ >> > > _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Analytics October 2014