November 2014 - Analytics - lists.wikimedia.org

Adding “php” key to X-Analytics header
by Christian Aistleitner 13 Jan '15

13 Jan '15

Hi, just a quick heads up, that Ops are about to add a “php” key to the X-Analytics header (i.e.: for sampled-1000 logs, hive, ...): https://gerrit.wikimedia.org/r/#/c/156793/ This header will hold the used PHP implementation [1]. Planned deployment is between 2014-09-01 and 2014-09-02. Have fun, Christian [1] https://wikitech.wikimedia.org/wiki/X-Analytics#Keys -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

1 2

My transition to Hadoop streaming (thanks to gage & ottomata!)
by Aaron Halfaker 01 Dec '14

01 Dec '14

Hey folks, I just finished a blog post about how I'm incorporating hadoop streaming into my workflow. http://socio-technologist.blogspot.com/2014/11/fitting-hadoop-streaming-int… TL;DR: I have strong opinions about Good Ways(TM) to process large datafiles in interesting ways and hadoop streaming will support them nicely. :) Props to ottomata for spending a bunch of time helping me get up to speed with our cluster and to gage for making it easier to find hadoop's error messages. -Aaron

3 2

Fwd: [Ops] Network outage for rack C4 in eqiad
by Ori Livneh 01 Dec '14

01 Dec '14

See message below about a network outage currently affecting multiple servers in eqiad. The set of affected servers includes gadolinium and hafnium, so udp2log-based web request logging and EventLogging-based metric reporters (e.g., Navigation Timing stats) are affected. ---------- Forwarded message ---------- From: Brandon Black <bblack(a)wikimedia.org> Date: Sat, Nov 29, 2014 at 9:54 PM Subject: [Ops] Network outage for rack C4 in eqiad To: Operations Engineers <ops(a)lists.wikimedia.org> We've lost network access (but not console access) to all the machines in eqiad rack C4 as of ~ 03:50 UTC (about 2 hours back from this email). This is mostly machines in a supporting role; no direct traffic front ends or app servers, etc. Phabricator is down as a result, as are a few monitoring -related bits and pieces. In the logs of asw-c-eqiad, it looks like the virtual chassis member for C4 was "removed" (log paste below). I haven't found any useful remote way to try to make that virtual chassis member restart yet. I'm not sure if it's worth waking anyone up in the middle of the night or anything at this point. Most likely this is going to involve some physical presence (or remote hands) at eqiad. ------------------------------- Nov 30 03:50:14 asw-c-eqiad /kernel: peer_inputs:3690 VKS0 closing connection peer type 24 indx 4 err 5 Nov 30 03:50:15 asw-c-eqiad chassism[1093]: CM_CHANGE: Member 1->1, Mode M->M, 1M 8B, GID 0, Master Unchanged, Members Changed Nov 30 03:50:15 asw-c-eqiad chassism[1093]: CM_CHANGE: 1M 2L 3L 5L 6L 7L 8B Nov 30 03:50:15 asw-c-eqiad chassism[1093]: CM_CHANGE: Signaling license service Nov 30 03:50:15 asw-c-eqiad chassisd[1512]: CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 7, jnxFruL1Index 5, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: EX4200-48T, 8 POE @ 4/*/*, jnxFruType 3, jnxFruSlot 4) Nov 30 03:50:15 asw-c-eqiad chassisd[1512]: CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 4 offline: Removal Nov 30 03:50:15 asw-c-eqiad chassisd[1512]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 4 Nov 30 03:50:15 asw-c-eqiad chassisd[1512]: CHASSISD_IFDEV_DETACH_FPC: ifdev_detach_fpc(4) Nov 30 03:50:15 asw-c-eqiad chassism[1093]: mvlan_member_change_delete: member id 4 (my member id 1, my role 1) Nov 30 03:50:15 asw-c-eqiad chassism[1093]: mvlan_delete_ifl: IFL resources for bme0.32773 (ifl_index 10) deleted Nov 30 03:50:15 asw-c-eqiad init: can not access /usr/sbin/smihelperd: No such file or directory Nov 30 03:50:15 asw-c-eqiad init: subscriber-management-helper (PID 0) started Nov 30 03:50:16 asw-c-eqiad vccpd[1095]: Member 3, interface vcp-0.32768 came up Nov 30 03:50:16 asw-c-eqiad vccpd[1095]: Member 7, interface vcp-1.32768 came up Nov 30 03:50:16 asw-c-eqiad vccpd[1095]: Member 8, interface vcp-1.32768 came up Nov 30 03:50:16 asw-c-eqiad vccpd[1095]: Member 6, interface vcp-1.32768 went down Nov 30 03:50:16 asw-c-eqiad vccpd[1095]: Member 2, interface vcp-1.32768 came up Nov 30 03:50:17 asw-c-eqiad fpc5 MRVL-L2:mrvl_brg_port_stg_entry_unset(),410:l2ifl not found! ifl 350 Nov 30 03:50:17 asw-c-eqiad /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err 5 (Invalid) Nov 30 03:50:17 asw-c-eqiad last message repeated 5 times Nov 30 03:50:18 asw-c-eqiad fpc5 MRVL-L2:mrvl_brg_port_stg_delete(),652:Port-STG-UnSet failed(Invalid Params:-2) Nov 30 03:50:18 asw-c-eqiad /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err 5 (Invalid) Nov 30 03:50:18 asw-c-eqiad last message repeated 5 times Nov 30 03:50:19 asw-c-eqiad fpc5 RT-HAL,rt_entry_delete_msg_proc,3539: l2_halp_vectors->delete failed proto MSTI, len 48 prefix 00350:00254 ------------------------------ _______________________________________________ Ops mailing list Ops(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops

4 3

RFC for media file request counts, please chime in
by Erik Zachte 26 Nov '14

26 Nov '14

Since 2008 Wikimedia collects pageview counts for most pages on nearly all wikis. A longstanding request of stakeholders (editors, researchers, GLAM advocates) has been to publish similar counts for media files: images, sounds, videos. A major obstacle to effectuate this was the existing traffic data collecting software. Webstatscollector simply couldn't be scaled up further without incurring huge costs. In 2014 WMF engineers rolled out a new Hadoop based infrastructure, which makes it possible to collect raw request counts for media files. So a few months after releasing extended pageview counts (with mobile/zero added), the time has come to produce similar data dumps for media files. https://www.mediawiki.org/wiki/Requests_for_comment/Media_file_request_count s Please comment onwiki. Thanks Erik Zachte

1 0

Writing EventLogging events to database failed on 2014-11-25 between 03:09 and ~midnight
by Christian Aistleitner 26 Nov '14

26 Nov '14

Hi, I am sorry to announce yet another EventLogging outage :-( EventLogging's database writer service failed to write events to the database between ~2014-11-25T03:09 and 2014-11-26T00:03. The recent adhoc increase of EventLogging's database throughput capacity to address the EventLogging database writing bottle-neck came with the known issue of not too robust exit synchronization of threads within the EventLogging database writer. This exit synchronization issue got the database writer stuck and caused the outage. A fix for that known issue is sitting in gerrit since Sunday, but due to the many meetings and discussions around recent EventLogging issues, the fix did not yet get reviewed and deployed. The still rather empty Incident Report is at https://wikitech.wikimedia.org/wiki/Incident_documentation/20141125-EventLo… I'll fill it with more information tomorrow. Backfilling from the logs is already running, and should also finish tomorrow. Best regards, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

3 3

Round-up of recent EventLogging issues
by Christian Aistleitner 24 Nov '14

24 Nov '14

Hi, with the recent events around EventLogging, I think a high-level round-up of what happened is overdue. There have been four unrelated issues: * Failed test to have EventLogging access its database through a high-availability proxy * Event volume growing beyond whan EventLogging's database writer could handle * db1020 outage * Untested EventLogging code got accidentally deployed (Please find the details below) For the last three of the four issues, backfilling is still pending, as focus up to now was on getting EventLogging under control again. As it seems EventLogging is under control again (keeping fingers crossed for the second item in the above list), backfilling is next. Sorry for the inconveniences, Christian * Failed test to have EventLogging access its database through a high-availability proxy Production's firewall got in the way [1]. Data got backfilled in the database from the logs. So no data got lost. The switch to a high-availabily proxy happened in the meantime. * Event volume growing beyond whan EventLogging's database writer could handle Event volume increased by not quite 60% over-night and the database writer could not handle the increased volume [2]. The database writer got restructured and got deployed yesterday in the UTC evening. Since then the restructured database writer could easily handle the increased volume. But before declaring victory, we have to wait for a few days and see how it handles hours of increased activity. During the time that the database writer could not handle the event volume, logging to disk could keep up with the increased volume, so backfilling should work, but it is still pending. * db1020 outage The database process of the m2 cluster (the one which EventLogging's database writer writes to) died [1]. Ops handled the issue promptly and failed-over to a slave database. We have logs in plain files for the affected periods. So backfilling should work, but it is still pending. * Untested EventLogging code got accidentally deployed It seems around trying to fix EventLogging for beta, an untested version accidentally got deployed to production. This accidentally deployed version stopped writing to the database from time to time and then started working again [4]. A working version has been deployed again. We have logs in plain files for the affected periods. So backfilling should work, but is still pending. [1] https://wikitech.wikimedia.org/wiki/Incident_documentation/20141113-EventLo… [2] https://wikitech.wikimedia.org/wiki/Incident_documentation/20141114-EventLo… [3] https://lists.wikimedia.org/mailman/private/ops/2014-November/043964.html [4] https://wikitech.wikimedia.org/wiki/Incident_documentation/20141118-EventLo… -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

4 6

Writing EventLogging events to database failed on 2014-11-22 for between ~19:30 and ~21:00
by Christian Aistleitner 24 Nov '14

24 Nov '14

Hi, as users of gerrit & Co. probably noticed, the m2 master database had issues again on 2014-11-22, and caused EventLogging to not be able to write events to the database on 2014-11-22 between about 19:30 and 21:00. Data for that period has been backfilled from logs again, so the database should again hold good numbers for that period. Just wanted to let you know what happened, in case you notice drops in graphs / dashboards that got generated between the incident and now. Sorry for the inconveniences, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

3 2

Where to check ongoing month traffic stats?
by Gilles Dubuc 21 Nov '14

21 Nov '14

The report card seems to only show data for the past month. Is there something I could check to verify if there was an overall drop in reader traffic over the last week? The reason I ask is that Media Viewer actions seem to be dropping across the board since November 13th: http://multimedia-metrics.wmflabs.org/dashboards/mmv This trend is seen on all wikis. I'm trying to find the cause of this problem, because we're launching significant UI changes to all wikis today and we'll need to make sure that our EventLogging figures are accurate.

2 3

Writing EventLogging events to database failed on 2014-11-18 for ~50 minutes between 14:14 and 15:02
by Christian Aistleitner 21 Nov '14

21 Nov '14

Hi, the m2 master crashed today (investigation still ongoing), and caused EventLogging to not be able to write events to the database on 2014-11-18 between 14:14 and 15:02. The data for that period is not lost, but is available in backup files, waiting to get injected again into the database. Just wanted to let you know what happened, in case you notice drops in graphs / dashboards during that period. Sorry for the inconveniences, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

4 5

Analytics Dev Team Commitments 2014-10-30 -- 2014-11-11
by Kevin Leduc 19 Nov '14

19 Nov '14

Hello, We kicked off our next sprint this morning, with the help of some release planning executed during the last 2 weeks. The sprint status is here: http://sb.wmflabs.org/t/analytics-developers/2014-10-30/ The focus of this sprint is working on the backend in preparation to display new data in Vital Signs. Bug ID Component Summary Points 72740 Dashiki Story: Vital Signs User selects the Daily Pageviews metrics 34 72741 EventLogging List tables/schemas with data retention needs 0 72642 EventLogging Story: Identify and direct the purging of Event logging raw logs older than 90 days in stat1002 0 67450 EventLogging database consumer could batch inserts (sometimes) 34 72746 Wikimetrics Story: WikimetricsUser tags a cohort using a pre-defined tag 5 72635 Wikimetrics report table performance, cleanup, and number of items 13 That’s 86 points in 4 stories. The bugs with 0 points are tasks for the team to track and follow up on, and the work mostly falls on other teams. Regards, Kevin Leduc

1 1

Adventures in Clusterland 2014-11-03--2014-11-09
by Christian Aistleitner 18 Nov '14

18 Nov '14

Hi, in the week from 2014-11-03–2014-11-09 Andrew, Jeff, and I worked on the following items around the Analytics Cluster and Analytics related Ops: * More research around making xmldumps available in the Analytics cluster * 'research' database user * Alter data type of time_firstbyte, and adding Range header to webrequest table * Automatic cleanup of EventLogging logs on stat1002 and stat1003 * Per wiki CSVs with daily aggregates of webstatscollector numbers (details below) Have fun, Christian * More research around making xmldumps available in the Analytics cluster In order make the xmldumps accessible easily from within the cluster, more research around WikiHadoop as InputFormat and Avro as serialization format was done. The proof of concept allowed to stream the xmldumps through WikiHadoop and write it into AVRO files. This approach would chunk the xmldumps into records containing the text (and metadata) for a revision and it's parent revision. Those records could be consumed directly from the cluster's processing platforms. * 'research' database user The needed code around making password changes of the 'research' database user got merged. So future password changes should be more frictionless. And the password was finally changed :-) * Alter data type of time_firstbyte, and adding Range header to webrequest table To be able to (at least mostly) disambiguate “seeking within a video file” from “starting to watch a video file” in the logs, we needed to add the Range header to the webrequest table. Additionally, the data type of the time_firstbyte column needed to get changed. While both migrations should just work according to Hive's documentation, testing them beforehand in labs showed that they “sometimes” screw up the table. So we prepared scripts to resurrect the table, if the migration blows up the table in the Analytics cluster. We migrated the table. The table exploded (Bug 73095). And the prepared scripts helped to rebuild it within a few minutes. Now, webrequest has the needed range header, a more granular data type for time_firstbyte, and all the partitions re-added. * Automatic cleanup of EventLogging logs on stat1002 and stat1003 File logs of EventLogging data on stat1002 and stat1003 are now automatically cleaned up after 90 days as required by the the data retention guidelines. * Per wiki CSVs with daily aggregates of webstatscollector numbers In order to ease upcoming plotting of webstatscollector data in Dashiki, we wrote code to automatically aggregate webstatscollector's hourly projectcounts files into per wiki CSVs with daily numbers. And we backfilled with data back to 2008. The code and data is still under code-review. -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

2 1

Re: [Analytics] Analytics
by Joel Sahleen 18 Nov '14

18 Nov '14

(Moving this discussion to analytics@ and localization-team@ based on Nuria’s suggestion below.) Hi Leila, The output I posted in the message is the only output I am seeing. I do not see the URL-encoded section or the validation section. I think there may be something wrong with my testing setup. Niklas Laxstöm has checked what is happening with our event logging in beta and he confirmed that we are sending events and the events are valid. The issue seems to be that we are logging events to the beta event logging db while what we checked earlier was the production event logging db. Can you (or anyone who is available) check the event logging db in beta to see if the table has been created and has data? The schema name again is ContentTranslation. If you don’t find anything, let us know and we will do some more investigation. If there is data in the beta db the next step would be to follow with Dan’s instructions to get a dashboard set up on limn1. I believe that most of Dan’s instructions need to be handled by someone on the analytics team, but let me know if there is anything I can help with. Thanks again for your help! Joel Joel Sahleen, Software Engineer Language Engineering Wikimedia Foundation jsahleen(a)wikimedia.org On Nov 11, 2014, at 11:47 PM, Leila Zia <leila(a)wikimedia.org> wrote: > Hi Joel, > > When you log events, the output will be the URL-encoded JSON sent by the browser, the event record (similar to what you pasted in your email), and whether the event validates against the schema. For the sample output you pasted earlier, or another sample output, can you let us know if validation section shows Valid? > > Leila > > On Mon, Nov 10, 2014 at 3:24 PM, Nuria Ruiz <nuria(a)wikimedia.org> wrote: > Joel, > > For questions like these going forward you can contact analytics@ as you will be getting amore prompt response. Both Dan and Leila are OOTO the next couple of days. > > >There are configuration options for the dev server that need to be added. Do similar options need to be added when not using the dev server? > No, there is no need. > > You would need sample rates to determine at which sampling rate you are logging if you are not logging all events, that is. > > Thanks, > > Nuria > > On Mon, Nov 10, 2014 at 2:39 PM, Dan Andreescu <dandreescu(a)wikimedia.org> wrote: > Adding Nuria as she can probably help > > On Monday, November 10, 2014, Joel Sahleen <jsahleen(a)wikimedia.org> wrote: > Hi Leila, > > I have tested our EventLogging code and it seems to be working fine with the event logging dev server. I can see the events coming through and they are valid. Here is some sample output: > > {"wiki": "wiki", "uuid": "e9dde14cf18552269ae81a7897f45d0c", "webHost": "localhost", "timestamp": 1415651367, "clientValidated": true, "recvFrom": "1.0.0.127.in-addr.arpa", "seqId": 2, "clientIp": "80f7683f3565e3d365740a1c8d1771ba95caaaaa", "schema": "ContentTranslation", "event": {"action": "create-translated-page", "targetLanguage": "ca", "token": "Tester", "version": 1, "contentLanguage": "es"}, "revision": 7146627} > > Are there additional configuration options we need to add to get EL working aside from just requiring the main extension file. There are configuration options for the dev server that need to be added. Do similar options need to be added when not using the dev server? > > Any help on this would be much appreciated. > > Thanks, > > Joel > > On Nov 7, 2014, at 3:52 PM, Joel Sahleen <jsahleen(a)wikimedia.org> wrote: > >> No problem, Dan. Enjoy your vacation! >> >> I will read through the document at the link you sent. I still need to fix our event logging code so it may be a couple days before we are ready anyway. If I have any questions I will contact Leila or Nuria. >> >> Thanks, >> >> Joel >> >> Joel Sahleen, Software Engineer >> Language Engineering >> Wikimedia Foundation >> jsahleen(a)wikimedia.org >> >> >> >> >> On Nov 7, 2014, at 3:10 PM, Dan Andreescu <dandreescu(a)wikimedia.org> wrote: >> >>> Joel, re: visualization, >>> >>> I'm going on vacation tomorrow and will be back on November 19th. If that's not too late, I can set up a limn instance then. If it's too late, that's ok, I wrote up the steps needed. Someone with access to the limn1.eqiad.wmflabs instance can perform them: https://wikitech.wikimedia.org/wiki/Analytics/Dashboards >>> >>> If you have the data or are generating the data in some other way, then you don't need half of that setup, you just need the part that sets up the limn dashboard which is only an hour or so of work. Sorry I'm running out the door and can't take care of that for you. >>> >>> Dan >>> >>> On Fri, Nov 7, 2014 at 7:37 AM, Joel Sahleen <jsahleen(a)wikimedia.org> wrote: >>> Thank you for the information, Pau. Very helpful. As you say, this does not change our current plans or hold us up in any way. I was just wasn’t clear about the relationship between the "high priorities" and "other metrics” sections. Knowing these came from different people at different times clarifies things a lot. >>> Joel >>> >>> On Nov 7, 2014, at 3:44 AM, Pau Giner <pginer(a)wikimedia.org> wrote: >>> >>>> @Pau, @Amir There is a section called High priorities for product management on the Content translation analytics page. Did these priorities come from outside the team or does this just represent our own internal view of the high priorities? >>>> >>>> Here is the story of that page as I'm aware of it: >>>> >>>> In September 2013, I was in a meeting with the analytics team in SF presenting an initial proposal for metrics. On that meeting, Dario recommended to create hierarchy of metrics based on the project goals. I created such image and a description for those metrics (the image is on top of our analytics page and the metrics are described in what it now the "Other metrics for created articles" section. >>>> >>>> In a meeting between Amir and Howie, they captured which should be the most important metrics from the product perspective in the "High priorities for product management". If I recalled correctly, as an outcome of later meetings between Howie and Amir, Howie was happy focusing on articles published as a single (initial?) metric for success. Amir can provide more details since I was not on those meetings. >>>> >>>> In short: The analytics page has pieces contributed by different people during the last year, and although there are many ideas to organise and detail, measuring the number of published articles seems to be the solid candidate to get started with, learn from the value we get from it and polish the rest of our goal-to-signal process for detecting better metrics. >>>> >>>> >>>> Pau >>>> >>>> On Fri, Nov 7, 2014 at 1:57 AM, Joel Sahleen <jsahleen(a)wikimedia.org> wrote: >>>> Hi All, >>>> >>>> I have been reviewing our requirements for Content translation analytics and I have a few questions/requests. I am sending them to the language team list and Leila and Dan in the hopes of getting some more clarity. I will add the same content to the Trello card. >>>> >>>> In the weekly team meeting earlier today we agreed that the first metric we want to collect data for is the number of articles created in each language over time. This is something has Amir has already set up our current Event Logging to track. Now that Kartik has enabled EL in beta, that part should be done. Since we are only barely turning it on, there will be very little data until people create more articles using CX. However, we should be set up to collect any new data that comes in. >>>> >>>> @Leila, can you verify that the db table now exists for the ContentTranslation schema? If it doesn’t, can you point us to right people we need to work with to troubleshoot the issue? Also you mentioned in our meeting that personal data may soon be purged after 90 days as part of a new privacy policy. Could you explain that a bit more or point us to more information? If this is the case, it may affect some of the metrics we would like to collect in the future. >>>> >>>> @Dan, what do we need to do next in order to set up a very simple visualization that would show the number of articles created per week by language. Pau has an image of what he would like on the Trello card. You mentioned something about being able to host a dashboard for us on one of the Limn servers you already have set up. >>>> >>>> @Santhosh, I believe you said earlier you have a script you use to export the data for the ULS analytics. If so can you share that please in case we need a similar script for CX so I don’t have to write a new script from scratch? >>>> >>>> @Pau, @Amir There is a section called High priorities for product management on the Content translation analytics page. Did these priorities come from outside the team or does this just represent our own internal view of the high priorities? If the latter, have these priorities been reviewed by anyone outside the team? I think we are safe to proceed with our current plan, but it would be good to have product sign off on things more generally. >>>> >>>> Thanks, >>>> >>>> Joel >>>> >>>> Joel Sahleen, Software Engineer >>>> Language Engineering >>>> Wikimedia Foundation >>>> jsahleen(a)wikimedia.org >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Localisation-team mailing list >>>> Localisation-team(a)lists.wikimedia.org >>>> https://lists.wikimedia.org/mailman/listinfo/localisation-team >>>> >>>> >>>> >>>> >>>> -- >>>> Pau Giner >>>> Interaction Designer >>>> Wikimedia Foundation >>>> _______________________________________________ >>>> Localisation-team mailing list >>>> Localisation-team(a)lists.wikimedia.org >>>> https://lists.wikimedia.org/mailman/listinfo/localisation-team >>> >>> >> > > >

5 24

Solving Limn - Zero graph descrepancy
by Yuri Astrakhan 17 Nov '14

17 Nov '14

I am trying to migrate Limn graphs to our own handling. Currently Zero graphs are generated as Limn dashboards, and after I applied this filter (taken from the HQL for counting article pageviews), i got matching (about 10% discrepancy) between our graphs and limn. Yet, one partner has discrepancy of 10 times, and I would like to see where that mismatch comes from. I looked at https://github.com/wikimedia/analytics-wp-zero but it seems there is other code that's missing from that repo. Any suggestions are welcome. Thanks! WHERE webrequest_source IN ('text', 'mobile') AND year=${year} AND month=${month} AND day=${day} AND x_analytics LIKE '%zero=%' AND SUBSTR(uri_path, 1, 6) = '/wiki/' AND ( ( SUBSTR(ip, 1, 9) != '10.128.0.' AND SUBSTR(ip, 1, 11) NOT IN ( '208.80.152.', '208.80.153.', '208.80.154.', '208.80.155.', '91.198.174.' ) ) OR x_forwarded_for != '-' ) AND SUBSTR(uri_path, 1, 31) != '/wiki/Special:CentralAutoLogin/' AND http_status NOT IN ( '301', '302', '303' ) AND uri_host RLIKE '^[A-Za-z0-9-]+(\\.(zero|m))?\\.[a-z]*\\.org$' AND NOT (SPLIT(TRANSLATE(SUBSTR(uri_path, 7), ' ', '_'), '#')[0] RLIKE '^[Uu]ndefined$')

2 1

analytics01 machine on betalabs
by Yuvi Panda 17 Nov '14

17 Nov '14

Is it being used for anything at all? Puppet has been failing for a while because it's trying to run the kraken role which hasn't been around for a while. Can it be deleted? -- Yuvi Panda T http://yuvi.in/blog

2 1

Wikimedia Research Showcase (November 14th)
by Aaron Halfaker 14 Nov '14

14 Nov '14

Streaming for this event will be starting shortly at https://www.youtube.com/watch?v=-FQ-TtTCdJo On Thu, Nov 13, 2014 at 10:51 AM, Aaron Halfaker <ahalfaker(a)wikimedia.org> wrote: > Hey folks, > > This month we're holding a special edition of the Research and Data > showcase > <https://www.mediawiki.org/wiki/Analytics/Research_and_Data/Showcase>. > We've invited Dr. Yan Chen, Professor from the UMuch iSchool to present her > work studying community dynamics with Kiva (micro-lending platform) and > what her results might imply for Wikimedia's sites. To take advantage of > her travel schedule, we'll be holding the event on *Friday** November 14 > at 11.30 PST (UTC-8) *rather than the usually 3rd Wednesday. The event > will be live streamed and recorded as usual. You can join the conversation > via IRC on freenode.net in the the #wikimedia-research channel. > > We look forward to seeing you there, > > -Aaron > > *Does Team Competition Increase Pro-Social Lending? Evidence from Online > Microfinance.*By Yan Chen <http://yanchen.people.si.umich.edu/>In the > first half of the talk, I will present our empirical analysis of the > effects of team competition on pro-social lending activity on Kiva.org, the > first microlending website to match lenders with entrepreneurs in > developing countries. Using naturally occurring field data, we find that > lenders who join teams contribute 1.2 more loans per month than those who > do not. Furthermore, teams differ in activity levels. To investigate this > heterogeneity, we run a field experiment by posting forum messages. > Compared to the control, we find that lenders from inactive teams make > significantly more loans when exposed to a goal-setting message and that > team coordination increases the magnitude of this effect.In the second > part of the talk, I will discuss a randomized field experiment we did in > May 2014, when we recommend teams to lenders on Kiva. We find that lenders > are more likely to join teams in their local area. However, after joining > teams, those who join popular teams (on the leaderboard) are more active in > lending. >

1 0

Writing EventLogging events to database failed on 2014-11-13 for ~23 minutes between 00:59 and 01:21
by Christian Aistleitner 14 Nov '14

14 Nov '14

Hi, a test to move the m2 database cluster behind a proxy towards high availability (ah the irony!) failed for EventLogging, and caused EventLogging to not be able to write events to the database between 00:59 and 01:21. The data for that period is not lost, but is available in backup files, waiting to get injected again into the database. Just wanted to let you know what happened, in case you notice drops in graphs / dashboards during that period. Sorry for the inconveniences, Christian P.S.: Documentation are at the (currently still rather empty) Incident Report: https://wikitech.wikimedia.org/wiki/Incident_documentation/20141113-EventLo… -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

1 1

Wikimedia Research
by Aaron Halfaker 14 Nov '14

14 Nov '14

Hey folks, This month we're holding a special edition of the Research and Data showcase <https://www.mediawiki.org/wiki/Analytics/Research_and_Data/Showcase>. We've invited Dr. Yan Chen, Professor from the UMuch iSchool to present her work studying community dynamics with Kiva (micro-lending platform) and what her results might imply for Wikimedia's sites. To take advantage of her travel schedule, we'll be holding the event on *Friday** November 14 at 11.30 PST (UTC-8) *rather than the usually 3rd Wednesday. The event will be live streamed and recorded as usual. You can join the conversation via IRC on freenode.net in the the #wikimedia-research channel. We look forward to seeing you there, -Aaron *Does Team Competition Increase Pro-Social Lending? Evidence from Online Microfinance.*By Yan Chen <http://yanchen.people.si.umich.edu/>In the first half of the talk, I will present our empirical analysis of the effects of team competition on pro-social lending activity on Kiva.org, the first microlending website to match lenders with entrepreneurs in developing countries. Using naturally occurring field data, we find that lenders who join teams contribute 1.2 more loans per month than those who do not. Furthermore, teams differ in activity levels. To investigate this heterogeneity, we run a field experiment by posting forum messages. Compared to the control, we find that lenders from inactive teams make significantly more loans when exposed to a goal-setting message and that team coordination increases the magnitude of this effect.In the second part of the talk, I will discuss a randomized field experiment we did in May 2014, when we recommend teams to lenders on Kiva. We find that lenders are more likely to join teams in their local area. However, after joining teams, those who join popular teams (on the leaderboard) are more active in lending.

2 1

Adventures in Clusterland 2014-10-27--2014-11-02
by Christian Aistleitner 12 Nov '14

12 Nov '14

Hi, in the week from 2014-10-27–2014-11-02 Andrew, Jeff, and I worked on the following items around the Analytics Cluster and Analytics related Ops: * Hive UDF to parse user agents with ua_parser * More kafkatee issues * Database replication getting stuck on 'Duplicate entry' * Ganglia's Views broke * Fixing sync of “aggregate-datasets” rsync * Turning down logstash logging * 'research' database user (details below) Have fun, Christian * Hive UDF to parse user agents with ua_parser A Hive UDF to parse User-Agent strings with ua_parser was merged and got deployed to the Analytics cluster. So people with Hive access can now use this UDF to automatically extract browser, OS, and device information. * More kafkatee issues After previous week's deployment of the new kafkatee build, we took a closer look at the generated files. While up to now, no partitions got dropped, it turned out that kafkatee is loosing lines if other processes have more disk activity. While---even if other processes have more disk activity---the kafkatee output files are still better than what udp2log can produce, we're investigating if the kafkatee output is good enough for users that need to stream data. (For non-streaming needs, it currently looks like Hive would be the more reliable choice.) * Database replication getting stuck on 'Duplicate entry' This week we've been having two more replication lag issues. From the five lag issues of October, the last three times, replication stopped with 'Duplicate entries'. Since this seems to be an emergent pattern, it has been called out with Ops and while they are aware of it, there is currently no fix for this issue. * Ganglia's Views broke Ganglia allows to have custom predefined dashboards (see Ganglia's “View” tab), which we use to watch kafka's and varnishkafka's key metrics. It seems that some puppet refactorings broke the existing Ganglia dashboards. As it seems we're one of the few teams using Ganglia dashboards regularly, we fixed puppet's Ganglia View setup. * Fixing sync of “aggregate-datasets” rsync Some weeks back, work was started to have stat1002's “aggregate-datasets” directory automatically publish it's content the website at http://datasets.wikimedia.org . Now final tweaks there have been put into place, and automatic publishing now works as expected. * Turning down logstash logging It turned out that the combination of Analytics cluster and other new log producers amount for more traffic than the current logstash setup can handle nicely. So the log level from the Analytics cluster got turned down until logstash itself got scaled up. * 'research' database user Many researchers and other WMFers are using the 'research' credentials to access the analytics databases, and the time came to switch those credentials to a new password. Since the password was not properly puppetized, discussions were started on how disruptive a change would be and how to best change it. Also puppetization work around it started. -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

1 0

"Infrastructure for Data Streams" - via Nuzzel
by Jeff Gerard 12 Nov '14

12 Nov '14

"How I store 200k messages/sec... Using Kafka & librdkafka" :D Infrastructure for Data Streams Tadas Vilkeliskis – Nov 10 See the top news stories shared by Jeff Gage's friends Sent from my iPhone

2 1

s1-analytics-slave impressively slow queries
by Sean Pringle 12 Nov '14

12 Nov '14

Three identical queries from the 'research_prod' user have just passed one month execution time on s1-anlytics-slave: select count(*) from staging.ourvision r where exists ( select * from staging.ourvision r1 inner join staging.ourvision r2 on r2.sha1 = r1.sha1 where r1.page_id = r.page_id and r2.page_id = r.page_id and DATE_ADD(r.timestamp, INTERVAL 1 HOUR) and r2.timestamp between r.timestamp and DATE_SUB(r.timestamp , INTERVAL 1 HOUR) and r1.sha1!= r.sha1 ); I havn't checked to see if the queries are just that amazingly slow, or if they're part of a larger ongoing transaction. In any case, three month-long transactions is pushing the resource limits of the slave and will soon result in either mass replication lag or some other interesting lockup that may in turn take days to rollback :-) Can we kill these? Can we optimize and/or redesign the jobs? Happy to help...

5 5

Some tables to purge from the EventLogging database
by Dan Garry 10 Nov '14

10 Nov '14

Hey. I'm quickly checking through the EventLogging database, and there are a lot of redundant tables in there which hold test data from before the app launched that we're never going to use for anything. As a consequence, these tables are safe to delete. The tables don't have many records in them so we're not going to free up much space, but they'll at least declutter the database a bit. The tables are: 1. MobileWikiAppEdit_8188113 2. MobileWikiAppEdit_8197809 3. MobileWikiAppLogin_8134781 4. MobileWikiAppLogin_8234429 5. MobileWikiAppLogin_9122346 6. MobileWikiAppOperatorCode_8678522 7. MobileWikiAppReadingAction_8233710 Please don't delete anything other than these specific ones! Thanks, Dan -- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation

2 3

Welcome to Public Science Framework
by ScienceFramework 08 Nov '14

08 Nov '14

1 0

analytics-store credentials
by Gilles Dubuc 07 Nov '14

07 Nov '14

Hi, Ops changed the password for the "research" user on analytics-store (I don't know what the context is, I just saw the commit summary on the Ops ML), which our team was using to generate tsvs for limn. Maybe it wasn't the right user to be using, I'm not sure, but one way or another we're going to need new credentials in order to keep our dashboards working. If someone has the credentials we should be using, I would be grateful to receive them through whatever medium is appropriate :) Thanks!

3 2

Re: [Analytics] stat1002 log cleanup
by Nuria Ruiz 05 Nov '14

05 Nov '14

Ok, so we should be set here to setup the log cleanup on 1002 , Andrew. Toby: Can you reply to the RT ticket giving the green light to do the cleanup? (#8760) Thanks Nuria On Wed, Nov 5, 2014 at 7:53 AM, Aaron Halfaker <ahalfaker(a)wikimedia.org> wrote: > I didn't find that obvious. Thanks Dario. I agree that a 90 day data > retention window would be long enough to deal with any corrupt/invalid > data. > > > On Tue, Nov 4, 2014 at 11:12 PM, Dario Taraborelli < > dtaraborelli(a)wikimedia.org> wrote: > >> On Nov 4, 2014, at 2:08 PM, Nuria <nuria(a)wikimedia.org> wrote: >> >> >> No, database records are not affected. It should not impact your work >> with EL in any way as your findings come from the records in the database. >> The logs are used mainly for operational purposes by the dev team as >> maintainers of the system. >> >> >> this obviously means that any missing, corrupted or invalid data from the >> log DB can only be recovered within the data retention window, which seems >> reasonable to me. >> >> On Nov 4, 2014, at 11:55 AM, Aaron Halfaker <ahalfaker(a)wikimedia.org> >> wrote: >> >> Hey guys, >> >> Sorry for the late response, but I'm still not sure what lives in >> */a/eventlogging/archive/** >> >> Will deleting from there affect what logs we have stored in the DB? Is >> this an intermediate log storage place, a canonical one, etc.? >> >> What will we no longer be able to do after it is pruned? >> >> -Aaron >> >> On Thu, Oct 30, 2014 at 2:35 PM, Nuria Ruiz <nuria(a)wikimedia.org> wrote: >> >>> >Also, I'm not clear on the significance of the EL archive directory. >>> Can you remind me/direct me to documentation? >>> Well, the logs just record the incoming pipeline of events, we have used >>> them to troubleshoot operational issues in the past but the bulk of data >>> analysis in EL happens from data stored on database. >>> >>> Some info here: >>> https://wikitech.wikimedia.org/wiki/EventLogging#Data_storage >>> >>> >>> On Thu, Oct 30, 2014 at 12:22 PM, Aaron Halfaker < >>> ahalfaker(a)wikimedia.org> wrote: >>> >>>> Nuria, can you specify which logs will be trimmed. >>>> >>>> Also, I'm not clear on the significance of the EL archive directory. >>>> Can you remind me/direct me to documentation? >>>> >>>> On Thu, Oct 30, 2014 at 12:36 PM, Nuria Ruiz <nuria(a)wikimedia.org> >>>> wrote: >>>> >>>>> >>>>> Hello, >>>>> >>>>> To comply with our privacy policy we are going to purge logs in 1002 >>>>> that are older than 90 days. Please let us know whether this is an issue. >>>>> We hope to have these changes done by the end of next week. >>>>> >>>>> A concrete example: >>>>> >>>>> Logs in, for example, the eventlogging archiving directory: >>>>> >>>>> @stat1002:/a/eventlogging/archive$ >>>>> >>>>> >>>>> will be restricted to the last 90 days. >>>>> >>>>> Thanks, >>>>> >>>>> Nuria >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Analytics mailing list >>>>> Analytics(a)lists.wikimedia.org >>>>> https://lists.wikimedia.org/mailman/listinfo/analytics >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Analytics mailing list >>>> Analytics(a)lists.wikimedia.org >>>> https://lists.wikimedia.org/mailman/listinfo/analytics >>>> >>>> >>> >>> _______________________________________________ >>> Analytics mailing list >>> Analytics(a)lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/analytics >>> >>> >> _______________________________________________ >> Analytics mailing list >> Analytics(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/analytics >> >> _______________________________________________ >> Analytics mailing list >> Analytics(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/analytics >> >> >> >> _______________________________________________ >> Analytics mailing list >> Analytics(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/analytics >> >> > > _______________________________________________ > Analytics mailing list > Analytics(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/analytics > >

2 1

data in Vital Signs
by Kevin Leduc 05 Nov '14

05 Nov '14

On October 30, updates were made to how metrics are calculated for the Vital Signs dashboards [1]. The result is an apparent jump up or down on some of the metrics starting October 30th. This is because we did not update the existing historical data. We are planning on recalculating all the historical data after more work is done on the backend over the next month. Here are the changes on the metrics: - Namespace Edits and Pages Created now include pages in all namespaces and pages that have been deleted. The plots for these metrics generally show a step up. On wikis where most of activity occurs on pages other than in namespace '0' (like Meta and Commons), you can see a dramatic difference in the data [2]. - We exclude bots in Rolling Active Editor, Rolling Surviving New Active Editor and Rolling Recurring Old Active Editor. The plots for these metrics show a small step down. This change is not as conspicuous as the previous one. If you were relying on this data right now, and need it consistent across time, please speak up. [1] https://metrics.wmflabs.org/static/public/dash/ [2] https://metrics.wmflabs.org/static/public/dash/#projects=commonswiki,metawi…

7 8

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Analytics November 2014