January 2015 - Analytics - lists.wikimedia.org

Relevant Content Availability
by Abdel Samad, Rawia 16 Oct '15

16 Oct '15

Hello, I work for a consulting firm called Strategy&. We have been engaged by Facebook on behalf of Internet.org to conduct a study on assessing the state of connectivity globally. One key area of focus is the availability of relevant online content. We are using a the availability of encyclopedic knowledge in one's primary language as a proxy for relevant content. We define this as 100K+ Wikipedia articles in one's primary language. We have a few questions related to this analysis prior to publishing it: * We are currently using the article count by language based on Wikimedia's foundation public link: Source: http://meta.wikimedia.org/wiki/List_of_Wikipedias. Is this a reliable source for article count - does it include stubs? * Is it possible to get historic data for article count. It would be great to monitor the evolution of the metric we have defined over time? * What are the biggest drivers you've seen for step change in the number of articles (e.g., number of active admins, machine translation, etc.) * We had to map Wikipedia language codes to ISO 639-3 language codes in Ethnologue (source we are using for primary language data). The 2 language code for a wikipedia language in the "List of Wikipedias" sometimes matches but not always the ISO 639-1 code. Is there an easy way to do the mapping? Many Thanks, Rawia [Description: Strategy& Logo] Formerly Booz & Company Rawia Abdel Samad Direct: +9611985655 | Mobile: +97455153807 Email: Rawia.AbdelSamad(a)strategyand.pwc.com<mailto:Rawia.AbdelSamad@strategyand.pwc.com> www.strategyand.com

5 6

Contribute
by Ron Baasland 17 Jun '15

17 Jun '15

Hello, My username is rbaasland and I would like to contribute to the analytics project. I was wondering if I could have access to the project, or how I go about contributing to this project? Thank you very much, Ron Baasland

3 3

Geo-aggregation of Wikipedia page views: Maximizing geographic granularity while preserving privacy – a proposal
by Dario Taraborelli 14 May '15

14 May '15

I’m sharing a proposal that Reid Priedhorsky and his collaborators at Los Alamos National Laboratory recently submitted to the Wikimedia Analytics Team aimed at producing privacy-preserving geo-aggregates of Wikipedia pageview data dumps and making them available to the public and the research community. [1] Reid and his team spearheaded the use of the public Wikipedia pageview dumps to monitor and forecast the spread of influenza and other diseases, using language as a proxy for location. This proposal describes an aggregation strategy adding a geographical dimension to the existing dumps. Feedback on the proposal is welcome on the lists or the project talk page on Meta [3] Dario [1] https://meta.wikimedia.org/wiki/Research:Geo-aggregation_of_Wikipedia_pagev… [2] http://dx.doi.org/10.1371/journal.pcbi.1003892 [3] https://meta.wikimedia.org/wiki/Research_talk:Geo-aggregation_of_Wikipedia_…

16 47

Re: [Analytics] Calculating interlinks between Wikipedias (Christy Okpo)
by E.C Okpo 02 Feb '15

02 Feb '15

Amir and Neta, This is interesting research! What was the visualization decision process like? I've often seen large inter-connections visualized using Chord or Network diagrams, did you decide on a heat map because of some peculiarities of this dataset? Regards, Christy

2 1

Hive operator precedence
by Nuria Ruiz 31 Jan '15

31 Jan '15

Team, Christian just let me know about the operator precedence in hive. Everyone writing queries should read about this as precedence it's not what you might expect and you query might end up taking forrrr everrrr making other users unhappy. https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/Queries#Avoiding… Thanks, Nuria

4 3

webrequest_misc added to Kafka + Hive
by Andrew Otto 30 Jan '15

30 Jan '15

Hello! Today qchris and I deployed some changes[1][2] to bring logs from misc_web varnishes into HDFS and Hive via Kafka. This isn’t a huge deal, but it does mean that we are now collecting webrequest logs for things like phabricator, annual.wikimedia.org <http://annual.wikimedia.org/>, graphite, stats.wikimedia.org <http://stats.wikimedia.org/>, etc. That is all, :) -Ao [1] https://gerrit.wikimedia.org/r/#/c/184183 <https://gerrit.wikimedia.org/r/#/c/184183> [2] https://gerrit.wikimedia.org/r/#/c/184191 <https://gerrit.wikimedia.org/r/#/c/184191>

4 5

Something's up with EventLogging since Jan 7th
by Gilles Dubuc 30 Jan '15

30 Jan '15

Hi all, I've tracked down an unexplained EL phenomenon that surfaced in our stats as a false trend in our global stats. The data I'm looking at specifically is coming from Media Viewer's MultimediaViewerNetworkPerformance_* tables. Have a look at this graph: https://docs.google.com/spreadsheets/d/1PJsyzAyj74dctGCl4-09L7LS4AMZRh57G56… the big change is on Jan 7th/8th It shows how many EL events we've recorded, per client-reported country, over the last 90 days. The sampling factor we use has been constant for each wiki over that period. Thus, the distribution shouldn't evolve drastically, aside from seasonal/local trends. Besides the Ukraine spike on a particular date (probably related to world events), the graph before Jan 7th looks like what you would expect. Then, following the outage that happened on Jan 7th, not only the balance is completely changed, but it evolves over time (the US and China are keeping "higher than normal" levels, while the rest seems to slide down lower than pre-7th quantities), showing me that something strange is happening and is probably unresolved. This balance shifting over time is really problematic for tracking Media Viewer client-side network performance, because Chinese traffic suddenly accounting for a bigger or smaller share of the overall recorded events creates big swings in the global averages/percentiles (since network performance in China is bad).

3 7

LinkedIn + Kafka Ecosystem
by Andrew Otto 30 Jan '15

30 Jan '15

Nice concise article about Kafka usage and plans at LinkedIn: https://engineering.linkedin.com/kafka/kafka-linkedin-current-and-future <https://engineering.linkedin.com/kafka/kafka-linkedin-current-and-future>

1 0

Grafana production configs
by E.C Okpo 29 Jan '15

29 Jan '15

Hello, I'm working on adding performance instrumentation to the Parsoid codebase with statsd/node-txstatsd, and then visualizing the metrics via Grafana. I'm at the stage where I'm looking to add the metrics' namespaces and schema to the WMF Grafana configs. It looks like WMF has Grafana working with Graphite/Carbon as a metrics database and ElasticSearch as the db database, where can I find the production Carbon config files to input the settings for tmy metrics? Also, from my research WMF's carbon data retention schema is set to '1m:1y, 10m:10y', should I default to this as my retention schema? Note that the metrics are fired off anytime the Parsoid API is used so each datapoint doesn't necessarily represent a minute/second/etc of data. Thanks, Christy

3 2

pagecounts-raw news
by Andrew Otto 29 Jan '15

29 Jan '15

Hi all! Some of you are probably aware of the pagecounts-raw dataset hosted at http://dumps.wikimedia.org/other/pagecounts-raw/ <http://dumps.wikimedia.org/other/pagecounts-raw/>. This week, we are making a change to how this dataset is generated. This should be mostly transparent, but an announcement is needed just in case anyone notices any differences. pagecounts-raw has historically been generated by piping the udp2log webrequest logs into a C program called webstatscollector[1]. This code is fairly old, and the logic it uses to generate pagecounts is out of date. However, since this data has been public for so long, we made an effort to continue to support it as is. We are still in the process of backfilling, but eventually all pagecounts-raw data after January 1 2015 will be generated from webrequest data stored in HDFS. This data is collected using Kafka, and pagecounts-raw is now generated by Hive. You may see a slight increase in article counts. The webrequest data in HDFS is less lossy than the udp2log data. By the way, do you know about the pagecounts-all-sites[2] dataset? pagecounts-all-sites is in a similar format to pagecounts-raw, but comes with more up to date pagecount logic. Most importantly, it includes mobile site pagecounts. Perhaps you should use pagecounts-all-sites instead of pagecounts-raw, eh? :) -Andrew Otto [1] https://github.com/wikimedia/analytics-webstatscollector <https://github.com/wikimedia/analytics-webstatscollector> [2] http://dumps.wikimedia.org/other/pagecounts-all-sites/ <http://dumps.wikimedia.org/other/pagecounts-all-sites/>

1 0

Early registration for CSCW 2015 ends January 30th
by Dario Taraborelli 28 Jan '15

28 Jan '15

For those of you interested in attending, the early registration deadline is January 30. See also https://meta.wikimedia.org/wiki/Research:CSCW_2015 <https://meta.wikimedia.org/wiki/Research:CSCW_2015> — — — CSCW 2015 | March 14-18 | Vancouver, BC, Canada http://cscw.acm.org <http://cscw.acm.org/> * Early registration ends January 30th. * Advance program is available at http://cscw.acm.org/2015/program/ <http://cscw.acm.org/2015/program/> * Conference hotel rooms are already selling out at $135/night The 18th ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW 2015) will be held March 14-18 in Vancouver, BC, Canada and is co-located with ACM Learning at Scale. CSCW is the premier venue for presenting research in the design and use of technologies that affect groups, organizations, communities, and networks. Bringing together top researchers and practitioners from academia and industry in the area of social computing, CSCW encompasses both the technical and social challenges encountered when supporting collaboration. Jeff Hancock from Cornell University will give the opening keynote address discussing "The Facebook Study: A Personal Account of Data Science, Ethics and Change." The closing keynote speaker will be Zeynep Tufekci from University of North Carolina Chapel Hill, speaking on "Algorithms in our Midst: Information, Power and Choice when Software is Everywhere." We are also pleased to announce that Wanda Orlikowski will receive the CSCW 2015 Lasting Impact Award and present a retrospective on her groundbreaking 1992 paper, "Learning from Notes: organizational issues in groupware implementation." Registration is available at: http://cscw.acm.org/2015/attend/registration.php <http://cscw.acm.org/2015/attend/registration.php> Registration questions? Ask Yvonne Lopez Hotel reservations can be made at: http://cscw.acm.org/2015/attend/hotel.php <http://cscw.acm.org/2015/attend/hotel.php> Use #CSCW2015 and follow us at http://twitter.com/ACM_CSCW <http://twitter.com/ACM_CSCW> or http://www.facebook.com/acmCSCW <http://www.facebook.com/acmCSCW> for updates. Conference Co-chairs Dan Cosley, Cornell University Andrea Forte, Drexel University chairs2015(a)cscw.acm.org <mailto:chairs2015@cscw.acm.org>

1 0

MariaDB sql_mode
by Sean Pringle 28 Jan '15

28 Jan '15

Another topic at MW Summit was using sql_mode to avoid some of MySQL's odd legacy behaviour. They are: https://mariadb.com/kb/en/mariadb/sql_mode/ You can set sql_mode per client connection on analytics-store without affecting anyone else or replication.

1 0

Non-MariaDB options
by Sean Pringle 27 Jan '15

27 Jan '15

Yesterday at MW Summit there were some non-MariaDB options tossed aorund. They were: More Hadoop usage than current efforts. Druid http://druid.io/ (Dan) RethinkDB http://rethinkdb.com/ (Ori) Hyperdex http://hyperdex.org/ (Sean) TokuMX http://www.tokutek.com/tokumx-for-mongodb/ (Sean) No data yet on whether any or all are useful or not. Druid is probably the most established of the bunch. TokuMX is a MongoDB clone using TokuDB under the hood, the tech we already use to make analytics-store not explode replicating all shards. If anyone is keen to experiment I think they should speak up and ask for hardware. Ops has some misc boxes around, plus a number of ex-MariaDB Dell R510's that might suit experiementation. BR Sean

2 1

analytics-store disk space
by Sean Pringle 27 Jan '15

27 Jan '15

Yesterday at the MW Summit I mentioned to Dan/Nuria/Halfak that analytics-store disk was ~20% used. JFTR that was wrong; it's actually closer to 45% used. It's a 6T RAID10 10K array holding S1-7, Eventlogging, Staging, and the Data Warehouse test schema. Still enough space for a good while at current growth rate, but we'll want to review it periodically. Sean

1 0

Better way to identify client IP address
by Ananth RK 27 Jan '15

27 Jan '15

We currently have a method in Geo UDF that takes an IP address from the remote host header and the X-Forwarded-For value and attempts to identify the originating client IP address by following a simple algorithm as follows: GetClientIP(remote_host, X-Forwarded-For) IF X-Forwarded-For value is not valid: return remote_host ELSE: FOR EACH valid IP address, proxy_ip, in the comma-separated X-Forwarded-For value: IF proxy_ip does not start with "127.0" or "192.168" or "10." or "169.254": return proxy_ip What are the ways in which this naive algorithm can be improved? For example, is it better to maintain a separate list of IP address to ignore (currently only 4)? If yes, how do we ensure that the list is exhaustive? Any other improvements? Thanks, Ananth

6 8

Getting Access to Wikipedia Database
by Neta Livneh 26 Jan '15

26 Jan '15

Hi everyone, My name is Neta Livneh and I'm a FOSS OPW Intern, working with the language engineering team on estimating how many translated pages there are in different wikipedias. For my project, we will need to sql queries on current wikipedia data (mostly revision history table). I already have a Gerrit account. Can I get SSH access for running such queries? Thanks, Neta

7 14

Re: [Analytics] u2opiamobile + WMF tech contacts
by Adam Baso 24 Jan '15

24 Jan '15

(message contents deleted)

1 0

Wikimedia referrer policy
by Dario Taraborelli 24 Jan '15

24 Jan '15

I’ve been discussing with the folks at CrossRef (the largest registry of Digital Object Identifiers, think of it as the ICANN of science) how to accurately measure the impact of traffic driven from Wikipedia/Wikimedia to scholarly resources. While digging into their data, we realized that since Wikimedia started the HTTPS switchover and an increasing portion of inbound traffic happens over SSL, Wikimedia sites may have stopped advertising themselves as sources of referred traffic to external sites. While this is a literal implication of HTTPS, it means that Wikimedia's impact on traffic directed to other sites is becoming largely invisible and Wikimedia might be turning into a large source of dark traffic. I wrote a proposal reviewing the CrossRef use case and discussing how other top web properties deal with this issue by adopting a so-called "Referrer Policy”: https://meta.wikimedia.org/wiki/Research:Wikimedia_referrer_policy <https://meta.wikimedia.org/wiki/Research:Wikimedia_referrer_policy> Feedback is welcome on the talk page: https://meta.wikimedia.org/wiki/Research_talk:Wikimedia_referrer_policy <https://meta.wikimedia.org/wiki/Research_talk:Wikimedia_referrer_policy> Dario

2 1

Re: [Analytics] Better way to identify client IP address
by Ananth RK 23 Jan '15

23 Jan '15

Thanks for the detailed response. > That looks like (if X-Forwarded-For is “valid”) your jumping straight > to considering the X-Forwarded-For header. You might want to check > beforehand whether the client ip is a “trusted” proxy. And only if it > is, you should walk the X-Forwarded-For. > Also, it sounds a bit like you're going through X-Forwarded-For from > left to right. Make sure to walk it from right to left, as proxies are > expected to append (not prepend) the client IP they forward for. Okay. Assuming we append client_ip to X-Forwarded-For and process the list from right-to-left, do you still think we should explicitly check whether client_ip is a trusted proxy? Because, per the modified algorithm, the client_ip will be the first item to be processed and if it is not a trusted proxy, it will be returned as the result - which is what even the explicit initial check will result in this case as well. Correct? > The way you combine the looping and if looks like you might be > skipping across entries that you cannot parse. > But if you find invalid entries while backtracking the IPs, you're > probably in parts of the X-Forwarded-For value that you shouldn't trust. > (X-Forwarded-For can easily be spoofed by clients) We only check that each entry is a valid IPv4 or IPv6 address and not just a random string. If we find invalid entries, then the only option we have is to continue the search further for a valid IP. > list_of_ips = append client_ip to X-Forwarded-For > for ip in reverse( list_of_ips ) > do > if ip is not a trusted proxy or the iterator does not have more elements > then > return ip > fi > done > (That should give invalid IP addresses for some requests. That > seems to be the correct thing from my point of view. But if you rather > geolocate wrong than not geolocate at all, you can choose to pick the > last known good IP address instead. I guess that's more a matter of > taste.) What would picking the last known good IP address in this context? An IP that matches the IPv4/IPv6 pattern _AND_ is not a trusted proxy? > Different stakeholders at the WMF use different lists of what kind of > proxies they consider. So things are not really clear cut. Compiling a common list - may be union of "General", "MediaWiki" and "Wikipedia Zero" - would be very useful. As Oliver suggests, it would be good to have them in site-wide config files. Thanks, Ananth

4 4

Re: [Analytics] collections entry points
by Jon Katz 22 Jan '15

22 Jan '15

+ product, analytics as fyi On Thu, Jan 15, 2015 at 3:27 PM, Jon Katz <jkatz(a)wikimedia.org> wrote: > Hi Folks, > Just sharing some data gleaned from the logs that I pulled in relation to > the collections project. I looked at clicks on the watchlist star (see > image), because we are considering that to be the first entry point for > creating a collection from an article on mobile web. Nothing > earth-shattering, but the data suggests that the pilot will not generate > enough data without: > > - a tutorial element/signage > - improved account creation funnel > - additional entry points (such as search results) > > -J > > > [image: Inline image 1][image: Inline image 4] > > *Summary: * > > - Data suggests additional entry points might be necessary in order to > derive meaningful results from the pilot. > - Current expected user population reaching collections entry point > without further intervention 2.4M per month. Of those, 28.6k are logged in > and able to access the feature funnel. > - Another 8.5k users end up signing up as a result of the prompt > (.3% conversion rate). > - Baseline established to measure interventions against. > - More baselines to come. > > > *Results* > > - in a 30 day period in April, ~28.6k logged in users clicked on > watchlist star (2.4M anonymous clicked, prompted to create account) > - in a 30 day period in June*, ~8.5k of the previously logged-out > users successfully created accounts after clicking on watchlist star (.3% > conversion rate). I do not know if these should be counted in the 28.6k > logged in "watch" events. > - Logins that resulted from CTA not tracked, but assumed to be low and > might trigger the measured events we did track. > - Chart view, clicks on watchlist star (see methods>assumptions, > below, for anonymous user calc): > > [image: Inline image 2] > > *How this informs Collections:* > > - If nothing changes, ~2.4M people have potential to see watchlist > creation prompt, of whom 28.6k are immediately eligible to create (don't > need login/account?). I think this latter number is a little low for a > pilot. You? Should we explore other entry points for pilot, like search > results? > - We have a baseline of ~81.5k watchlist clicks per month by logged in > users and 2.8M from loggedout. How much can we increase these numbers? I > don't think this number *needs* to grow significantly for success, > because 1 collection could drive infinite views, but it would definitely > help > - Any overlay or product introduction will need to increase this > number to be successful > - Changing the icon from a star to a heart or some thing else more > collection-y would also have to increase this number to be succesful > - Account creation rate has room to grow > - The CTA has a .3% conversion rate, which is very very low. If we > can boost this, we might be able to have a large impact on the number of > people who are eligible to create. > - Can we drive this number up by offering collections as an > incentive? > > *Next Steps* > > - We need to explore alternate entry points - such as search, as we > continue to build out the MVP. > - I think we should measure the following baselines to establish what > we should measure against and set realistic expectations: > - Book creations over time (to compare to collection creations) > - Edits on mobile (to compare to collection creations/edits) > - Watchlist menu clicks (similar path to editing your collections) > - Referrals via social (to establish baseline for social/share > traffic) > - Any others? > > *Methods:* > > - Queries used: > - Queried MobileBetaWatchlist_5281061 for 30 days--tracks clicks on > star (April 2014). Link to query > <https://docs.google.com/a/wikimedia.org/document/d/1-bfW9H2kApvOOOTKL40TjYK…> > . > - Queried ServerSideAccountCreation_5487345 for 30 days in (June > 2014) Link to query > <https://docs.google.com/a/wikimedia.org/document/d/1-bfW9H2kApvOOOTKL40TjYK…> > . > - Assumptions > - Assume unique [IP+UserAgent] = unique anonymous user. Definitely > undercounts users this way. > - Assume number of users who login as a result of clicking on > watchlist is small (not currently measured) and/or succesful logins from > this prompt sucesfully add article to watchlist and trigger event that we > *did* measure. So the above count might be underestimating total > events. > - Assuming addition is appropriate between the logged in watchlist > clicks and the account creations. I ran a join to see if a succesful > account creation then triggered a successful version of the original > watchlist event (i.e. anon click to watch article-->account signup > prompt-->create account-->article watch click triggered), but it kept > stalling out. This is the query, in case you're curious > <https://docs.google.com/a/wikimedia.org/document/d/1-bfW9H2kApvOOOTKL40TjYK…> > . > - Data alert > - I had a rough time with some queries because the event logs had > events from Jan- Dec, but there were obvious dropoffs and spikes due to > changes that were only caught when I ran a daily report for events in each > table. Without this the data would have been very very skewed (my original > count for anon watchlist clicks was 57k per month...it is actually 2.4M). Here > is a link > <https://docs.google.com/a/wikimedia.org/document/d/1-bfW9H2kApvOOOTKL40TjYK…> > to a helpful daily event query. > > > *necessary to use different time periods because the logs did not cleanly > overlap > > >

3 3

Fwd: Calculating interlinks between Wikipedias
by Neta Livneh 20 Jan '15

20 Jan '15

Hi, Amir Aharoni and I thought that this might be interesting for people here. We wanted to answer the following question: for each language, how many of the articles in the main namespace that appear in one Wikipedia (e.g., FR) also appear in another (e.g., EN). We calculated this as the percentage of articles that exist in two languages from the total number of articles of one of the languages (taken from [1]). That is, we calculated the intersection(EN, FR)/Count(FR)). We did this for all of the languages (287^2) [2]. Results: 1. The co-exist matrix of counts can be found Google Spreadsheet <https://docs.google.com/spreadsheets/d/1wj3fPkU8v2-KcEjTNtFLabMTXWyhgvRhgyw…> - It was generated on 01/09/2015 using the langlinks table of every wiki. The underlining query is -based on this code: (%s is the wiki code) SELECT '%s' as source, ll_lang as target, COUNT(*) as count FROM %s_p.langlinks LEFT JOIN %s_p.page ON page_id = ll_from WHERE page_namespace = 0 GROUP BY ll_lang; - The links are not symmetrical. there is on average less than one percent difference between the links from lang A to B compared to lang B to A. - However, it wasn't perfect. Wikis with less than 3500 links (that means the has less than 100 articles) have on average more than 20% out links (that is, taken from that language langlink table) than in links (other wikis pointing at that language). - As the number of langlinks gets bigger (and for most cases, the side of the wiki), the difference and variance between the in and out links gets smaller. - Some out links pointed to mistakes (zh-cn, zh-tw, nn) - is fixed. - The raw data can be sent on request. 2. A heat map of the co-exist wikis with more than 50,000 articles. It is ordered by size. As I mentioned, the above triangles are not symmetrical because the counts (which are themselves not equal but are close enough) are divided by the number of articles in each wiki. The heat map is between Red - high level of congruence to Yellow - low level. [image: Inline image 1] Points to notice: 1. Most languages have strong connections with English. 2. There is a group of interconnected wikis that are based on Swedish (Dutch, Waray-Waray, Cebuano, Vietnamese, Indonesian, Minangkabau). 3. Piedmontese is highly interconnected with Latin languages, as do Latin itself. On the other hand, Chechen is mostly connected to Russian. 4. a. Arabic has 8% more in links than out.There isn't one Wiki that caused this difference, so it's not a bot. 5. Telugu doesn't have many interlinks, not to English, Hindi or Bengali. 6. There are other visible strong connections (as Serbian and Serbo-Croatian) but they are not as surprising. Thoughts? Cheers, Neta [1] meta.wikimedia.org/wiki/List_of_Wikipedias updated on 01/12/2015. [2] You might be wondering why did we calculated both EN-> FR and FR-> EN as there is a 1 to 1 connection between the interlanguage links in Wikidata? We used the data from the langlinks table for every Wikipedia and not from the wiki interlanguage link table. We did so for two reasons: 1) it was computationally easier 2) we wanted to see if there are any irregulars in the data.

3 5

Making EventLogging output to a log file instead of the DB
by Gilles Dubuc 19 Jan '15

19 Jan '15

This depends on [1] so we're not going to need that immediately, but in order to help Erik Zachte with his RfC [2] to track unique media views in Media Viewer, I'm going to need to use something almost exactly like EventLogging. The main difference being that it should skip writing to the database and write to a log file instead. That's because we'll be recording around 20-25M image views per day, which would needlessly overload EventLogging for little purpose since the data will be used for offline stats generation and doesn't need to be made available in a relational database. Of course if storage space and EventLogging capacity were no object, we could just use EL and keep the ever-growing table forever, but I have the impression that we want to be reasonable here and only write to a log, since that's what Erik needs. So here's the question: for a specific schema, can EventLogging work the way it does but only record hits to a log file (maybe it already does that before hitting the DB?) and not write to the DB? If not, how difficult would it be to make EL capable of doing that? [1] https://phabricator.wikimedia.org/T44815 [2] https://www.mediawiki.org/wiki/Requests_for_comment/Media_file_request_coun…

6 16

Hadoop Upgrade and Downtime
by Andrew Otto 16 Jan '15

16 Jan '15

Hi all, I’m in the middle of a (slow) upgrade process for the Hadoop cluster. Currently, we are running CDH 5.0.2, and would like to upgrade to CDH 5.3. There are several steps to this process, the first of which is upgrading our OS to Ubuntu Trusty. Along the way, I’m replacing our current NameNodes with different hardware. I am ready to do this now. I don’t see much opportunity to schedule this over the next couple of weeks, due to All-Hands travel, so I’d like to schedule this for tomorrow morning (Friday January 16th). I expect this to be relatively simple downtime, that will only take a few minutes. Just in case, I’d like to reserve 2 hours of time. So, unless there are serious objections, plan for Hadoop to be offline from 2015-01-16 15:45 - 17:45 UTC Also, please don’t start jobs before this time slot that you think will take a long time. If there are running jobs, I either can’t shut down the cluster, or I will have to kill the jobs. If I see running jobs, I’ll try to reach out to you before I kill anything. If anyone is interested in a rough migration plan, it is here: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration… <https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration…> Thanks all! -Ao

1 3

Udp2log TSVs missing 1 hour of data
by Marcel Ruiz Forns 16 Jan '15

16 Jan '15

Hi, On Jan 13th 2015 between 22:20 and 23:18 UTC (~1 hour) stat1002 ceased receiving TSV data from udp2log for the following data streams: - Mobile requests stream - Pagecounts-raw - Requests stream - Zero requests stream The reason for that were routing problems in the firewall introduced by a change to the way iptables rules are created for udp2log. The problem was quickly resolved. Here is Phab task: https://phabricator.wikimedia.org/T86973 Apologies for the late email, Marcel

2 1

Office Hours for EventLogging & Dashboarding
by Kevin Leduc 15 Jan '15

15 Jan '15

Please join the Analytics Engineering team for... Office Hours: EventLogging & Dashboarding Hosts: Dan and Nuria Date: January 14 Time: 20:00 UTC - Convert to Local Time <http://www.timeanddate.com/worldclock/fixedtime.html?msg=EventLogging+and+D…> Hangout: https://plus.google.com/hangouts/_/wikimedia.org/a-batcave IRC: #wikimedia-analytics Description: Teams need metrics on how their product or feature is performing, then they need to visualize those metrics. This is accomplished with instrumenting code with EventLogging, mashing data with some queries and setting up a Limn Dashboard. The Analytics Engineering team is open for office hours to answer questions about the process, help solve any issues and listen to feedback on the process. Feel free to drop in the Goolge Hangout linked above or ask questions on the IRC channel during our Office Hours.

2 4

How to get latest stats?
by రహ్మానుద్దీన్ షేక్ 14 Jan '15

14 Jan '15

Hello, I have always seen a difference of about 2000 article count of total articles on Kannada Wikipedia (knwiki or kn.wikipedia.org), which is the accurate count? Also, the stats at stats.wikimedia.org are always a month old or more than that, for Indic languages - Telugu (tewiki), Odia (orwiki), Kannada (knwiki), Marathi (mrwiki). How to get latest stats? Are there any API calls, for Total editors, New editors, active editors, very active editors, article count, new articles per day, edits per month and page views for Indic languages like Kannada, Telugu, Odia, Marathi, etc ? -- With thanks & regards *Rahimanuddin Shaik* నాని [image: http://upload.wikimedia.org/wikipedia/meta/0/08/Wikipedia-logo-v2_1x.png] reachout ఒక విశ్వాన్ని ఊహించండి, ఎక్కడయితే ప్రతి మనిషి ఒక సంపూర్ణ విజ్ఞానభాండారాన్ని అందరితో పంచుకోగలడో, ఆ విశ్వాన్ని ఊహించండి. *అటువంటి విశ్వాన్ని నెలకొల్పడమే మా సంకల్పం.* తెలుగు వికీపీడియా : http://te.wikipedia.org CIS-A2K Program : http://meta.wikimedia.org/wiki/India_Access_To_Knowledge CIS-INDIA : http://cis-india.org A new address for ebooks : http://kinige.com *తెలుగువారికి సాంకేతిక సహాయం - http://techsetu.com <http://techsetu.com/>*

2 1

Beta Labs EventLogging logs
by Ryan Kaldari 14 Jan '15

14 Jan '15

It seems the EventLogging logs have disappeared from /var/log/upstart/ on Beta Labs (deployment-bastion). Does anyone know where they are now? Kaldari

6 12

most clicked links in articles
by Amir E. Aharoni 14 Jan '15

14 Jan '15

Hi, Are there metrics about which links in each article are the most clicked? I can think there's a lot to be learned from it: * Data-driven suggestions for manual of style about linking (too much and too few links are a perennial topic of argument) * How do people traverse between topics. * Which terms in the article may need a short explanation in parentheses rather than just a link. * How far down into the article do people bother to read. Anyway, I can think that accessibility to such data can optimize both readership and editing. And maybe this can be just taken right from the logs, without any additional EventLogging. -- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com ‪“We're living in pieces, I want to live in peace.” – T. Moore‬

8 12

Adding “php” key to X-Analytics header
by Christian Aistleitner 13 Jan '15

13 Jan '15

Hi, just a quick heads up, that Ops are about to add a “php” key to the X-Analytics header (i.e.: for sampled-1000 logs, hive, ...): https://gerrit.wikimedia.org/r/#/c/156793/ This header will hold the used PHP implementation [1]. Planned deployment is between 2014-09-01 and 2014-09-02. Have fun, Christian [1] https://wikitech.wikimedia.org/wiki/X-Analytics#Keys -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

1 2

January 2015 Wikimedia Research Showcase: Felipe Ortega and Benjamin Mako Hill
by Dario Taraborelli 13 Jan '15

13 Jan '15

The upcoming Wikimedia Research showcase <https://www.mediawiki.org/wiki/Analytics/Research_and_Data/Showcase> (Wednesday January 14, 11.30 PT) will host two guest speakers: Felipe Ortega <https://en.wikipedia.org/wiki/User:GlimmerPhoenix> (University of Madrid) and Benjamin Mako Hill <https://en.wikipedia.org/wiki/User:Benjamin_Mako_Hill> (University of Washington). As usual, the showcase will be broadcast on YouTube (the livestream link will follow on the list) and we’ll host the QA on the #wikimedia-research IRC channel on freenode. We look forward to seeing you there. Dario Functional roles and career paths in Wikipedia By Felipe Ortega <https://www.mediawiki.org/wiki/User:GlimmerPhoenix> An understanding of participation dynamics within online production communities requires an examination of the roles assumed by participants. Recent studies have established that the organizational structure of such communities is not flat; rather, participants can take on a variety of well-defined functional roles. What is the nature of functional roles? How have they evolved? And how do participants assume these functions? Prior studies focused primarily on participants' activities, rather than functional roles. Further, extant conceptualizations of role transitions in production communities, such as the Reader to Leader framework, emphasize a single dimension: organizational power, overlooking distinctions between functions. In contrast, in this paper we empirically study the nature and structure of functional roles within Wikipedia, seeking to validate existing theoretical frameworks. The analysis sheds new light on the nature of functional roles, revealing the intricate “ areer paths" resulting from participants' role transitions. Free Knowledge Beyond Wikipedia A conversation facilitated by Benjamin Mako Hill <https://www.mediawiki.org/wiki/User:Benjamin_Mako_Hill> In some of my research with Leah Buechley <http://mako.cc/academic/buechley_hill_DIS_10.pdf>, I’ve explored the way that increasing engagement and diversity in technology communities often means not just attacking systematic barriers to participation but also designing for new genres and types of engagement. I hope to facilitate a conversation about how WMF might engage new readers by supporting more non-encyclopedic production. I'd like to call out some examples from the new Wikimedia project proposals list <https://meta.wikimedia.org/wiki/Proposals_for_new_projects>, encourage folks to share entirely new ideas, and ask for ideas about how we could dramatically better support Wikipedia's sister projects.

1 0

Re: [Analytics] Performance Visualization Frontend (Christy Okpo)
by E.C Okpo 12 Jan '15

12 Jan '15

Hi, In terms of where the data will be coming from, it will be coming from node-txstatsD, a port of statsD/node-statsD, a statsD client. I had previously been working on using Event Emitters in conjunction with statsD, but node-txstatsD seems to solve the task of obtaining and sending timing to a visualization front-end. The data to be visualized will be timing (in ms), counts, and maybe sizes(KB, MB). Based on this, what would be the recommended front-end to then visualize the information? Message: 1 > Date: Tue, 30 Dec 2014 07:37:35 -0800 > From: Nuria Ruiz <nuria(a)wikimedia.org> > To: "A mailing list for the Analytics Team at WMF and everybody who > has an interest in Wikipedia and analytics." > <analytics(a)lists.wikimedia.org> > Subject: Re: [Analytics] Performance Visualization Frontend > Message-ID: > < > CAMpYYkG4aVNLFJL3p6+a2mH3gTcntH7oxu+haGFsdaA_ig2Qzw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > The more important question is where will your data come from: event > logging? graphite? elsewhere? visualization comes secondary to this. > > EventLogging is a good solution for structured, somewhat complex, > application data, graphite is s good solution for plain counters, which is > well suited to perf data. Let us know if you already have data and we can > proceed from there. > > Thanks, > > Nuria > > >

2 1

Renaming wmf.webstats Hive table and /wmf/data/archive/webstats HDFS path
by Christian Aistleitner 10 Jan '15

10 Jan '15

Hi, within the Analytics cluster, the pagecounts-all-sites dataset is still referred to by the legacy name “webstats” in * the wmf.webstats Hive table, and * the /wmf/data/archive/webstats HDFS path. Both have no known external customers, so renaming them to "pagecounts-all-sites" should not affect anyone. But just in case ... if you use any of them, let us know by 2015-01-09 08:00 UTC. If no one speaks up, I'll remove the webstats Hive table, and the webstats path in HDFS. The new Hive table wmf.pagecounts_all_sites and the new HDFS path /wmf/data/archive/pagecounts-all-sites are already available and contain both the old and also the new data. Have fun, Christian P.S.: This only affects the data in the Analytics cluster. The public URL stays unaffected: http://dumps.wikimedia.org/other/pagecounts-all-sites/ No changes there. -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

1 1

Catching EventLogging schema violations
by Ori Livneh 10 Jan '15

10 Jan '15

One of the persistent pain-points for teams using EventLogging seems to be the ease with which schema violation errors can slip through to production and linger unnoticed. Validation errors are written to the browser console, where they are easy to miss. There are several things we could do to make it easier to spot problems in client-side analytics code. I'd like to have live graphs showing the volume of data (valid and invalid) currently being logged under a particular schema integrated into the schema page in MetaWiki, and I'd like us to have some automatic alerting for that. Nuria is also working on making the process of testing analytics code in labs clearer and more useful. One thing you can do right now (and that I hope you will do) is to add a small piece of custom JavaScript to your global.js that will flash validation errors in an error bar on the page you are viewing. To do that, edit both https://meta.wikimedia.org/wiki/User:<your username>/global.js (for production) and http://meta.wikimedia.beta.wmflabs.org/wiki/User:<your username>/global.js (for labs), and add the following: // Show EventLogging validation errors in a dismissible bar at the top of page. var $el = $( '<pre style="background: yellow; margin: 0; padding: 8px; position: fixed; top: 0; width: 100%; z-index: 99"></pre>' ); $el.click( function () { $el.empty().detach(); } ); mw.trackSubscribe( 'eventlogging.error', function ( topic, err ) { $el.text( function ( idx, text ) { return ( text && text + '\n' ) + err; } ).appendTo( 'body' ); } ); This code won't do anything to change your user experience unless there are schema violations in the JavaScript code that is loaded by the page. If there are any errors, they will looks like this: http://i.imgur.com/ReVnfbn.png

3 2

Analytics Engineering Team Commitments 2014-12-11 -- 2014-12-25
by Kevin Leduc 10 Jan '15

10 Jan '15

Hello, It has been a while since the last email of this kind. The team continued it’s bi-weekly sprints around Columbus day, US Thanksgiving and through the switch from bugzilla to Phabricator. We have now re-organized our processes around phabricator and are excited to see how this tool will display our point burndown during the sprint. Start 2014-12-11 End 2014-12-23 Theme Wikimetrics Theme song All Along the Watchtower <https://en.wikipedia.org/wiki/All_Along_the_Watchtower> Point Commitment 62 # of Tasks 7 Burndown Chart https://phabricator.wikimedia.org/sprint/view/935/ Sprint Board https://phabricator.wikimedia.org/sprint/board/935/query/all/ Note: emails have not gone out for the last few sprints, but you can see the slideshows of their showcases below. Sprint ending 2014-12-09: https://docs.google.com/presentation/d/1LmkWEpcJD0-AtQMRmLEFSM-T_9hNCWVGvCv… Sprint ending 2014-11-25: https://docs.google.com/presentation/d/1siaxV4CVzx-Rqbs9zrEC9lmfhijweaNuRz1… Sprint ending 2014-11-12 https://docs.google.com/presentation/d/1XTy0yLCCKFk-CFKXiiUAZd1uVYL61vwAlOo… Cheers, Kevin (Analytics Product Manager) PS >From now on, the team will vote on a theme song at the sprint planning session. The song must start with the next letter in the alphabet of the previous song and will loosely reflect the mood of the team.

3 3

Image serving performance discoveries
by Gilles Dubuc 10 Jan '15

10 Jan '15

Hi everyone, I recently looked very closely at client-gathered statistics about image serving performance from within Media Viewer. Looking more specifically at the effect of thumbnail pre-rendering at upload time (which has been live for a few months) and thumbnail chaining (which was live for a few weeks and has now been turned off). The main question I was looking to answer is whether either of those techniques improved performance as experienced by users. Chaining, when combined with pre-rendering, had no noticeable effect on performance experienced by viewers. This is logical because pre-rendering means that the thumbnail generating gains only happen at upload time, therefore clients requesting the image later won't be affected. As for the effect on image scalers load, it was so insignificant that it couldn't be measured. Chaining is probably still useful for people requesting non-standard thumbnail sizes, which I'm not measuring since I've only been looking at Media Viewer, but the priority of addressing the community concerns over JPG sharpening in order to redeploy chaining seems much lower to me now if that's the only use case chaining will be useful for. The big discovery in my research is that we set out to do pre-rendering based on a wrong assumption. When looking at performance statistics earlier last year, we clearly saw that Varnish misses performed a lot worse than Varnish hits (well, duh) and so we set out to deploy pre-rendering the thumbnail sizes Media Viewer needs in order to get drasticaly reduce the amount of the Varnish misses. The reduction didn't happen. The wrong assumption was that each varnish miss is a case where the thumbnail requested has to be generated on the fly by the backend. The data I've just discovered shows that this is very rare for the thumbnail sizes Media Viewer currently uses. The vast majority of Varnish misses merely pulls from Swift a thumbnail that has already been rendered at some earlier point in time and just happens to not have been requested for a while. And that Swift pull + Varnish re-add is what's making the majority of Varnish misses perform worse than hits, not the need to generate the thumbnail with ImageMagick. The bottom line is that the thumbnail prerendering provided insignificant performance gains for this set of sizes. Infrequently requested thumbnails is the main problem, not the fact that they are rendered on the fly the first time they are requested. It seems like the only way to increase image serving performance in our current setup is to increase the expiry value in Varnish and/or increase Varnish capacity. Right now 17% of image requests in Media Viewer are Varnish misses, and 99.5% of those are pulling an existing thumbnail from Swift. Varnish misses are twice as slow as hits on average. I plan to disable pre-rendering next week in order to confirm these findings and determine for certain what percentage of image requests pre-rendering is useful for on the set of sizes Media Viewer currently uses. If you want to dig into the data, the relevant tables on the analytics DB are MultimediaViewerNetworkPerformance* and more specifically the event_varnish*, event_timestamp and event_lastModified columns.

8 13

WikiGrok and EventLogging
by Leila Zia 09 Jan '15

09 Jan '15

Hi, The mobile team is planning to switch WikiGrok on for non-logged in users next week (2014-01-12). The widget will be on on 166,029 article pages in enwiki. There are two EventLogging schema that may collect data heavily and we want to make sure EL can handle the influx of data. The two schema collecting data are: https://meta.wikimedia.org/wiki/Schema:MobileWebWikiGrok https://meta.wikimedia.org/wiki/Schema:MobileWebWikiGrokError and the list of pages affected is in: wgq_page in enwiki.wikigrok_questions. It would be great if someone from the dev side let us know whether we will need sampling. Thanks, Leila

6 16

Per-namespace daily edit numbers
by Gergo Tisza 09 Jan '15

09 Jan '15

I want to check what effect MediaViewer had on file namespace edits. Aggregating the standard MediaWiki dumps over all wikis seems like a pain; is there a more convenient source for that data? Even better if it can be filtered by the editcount of the user at the time of the edit. I looked at the Edit* EventLogging schemas, but those are either fairly recent or not used. Is there any other source where this information could be retrieved from? thanks Gergő

6 8

Per-namespace pageview data from half a year ago
by Gergo Tisza 08 Jan '15

08 Jan '15

I would like to graph the correlation between file namespace page views and MediaViewer image views. Back when MediaViewer was launched, I added a namespace parameter to NavigationTiming to be able to track per-namespace pageviews, but I messed up and it only got deployed around the time MediaViewer was enabled on Commons, so we have no data for the early steps of the deploy process. Do you know of any other source for per-namespace pageview data that is still available for the 2014 April-June period? Technically the raw pagecount files contain the information but aggregating those would be a horribly complicated way of getting this information. Does Hadoop pageview data go back that far? thanks Gergő

4 9

autopromote values - do we know how they affect participation?
by Amir E. Aharoni 08 Jan '15

08 Jan '15

Hi, Is there any research about the influence of autopromote variables on participation, editor retention, and such things? I am mainly talking about $wgAutoConfirmAge and $wgAutoConfirmCount. See the current values at http://noc.wikimedia.org/conf/highlight.php?file=InitialiseSettings.php These are different in various languages, and they seem rather random to me. The idea is supposed to be to prevent vandalism without hurting wikiness and editor retention, but are the current values based on any analytics? I went over all the bugs that are mentioned in the file to which I linked above. All of them say "we had a discussion and we reached a consensus". I cannot read all these languages, but my wild guess is that people just threw some numbers around without basing it on analytics, and voted to accept them. The discussion in the English Wikipedia[1] is, non-surprisingly, the longest (47 A4 pages); I didn't read it all, but it doesn't seem to be based on any metrics either. Anecdotally, I can recall many more times when I, as a Wikipedian, had to explain people that they need to do a few more edits to get a permission to move pages, than I had to revert bad page moves by new editors, so there is a possibility that the autopromote values are not actually very good. [1] https://en.wikipedia.org/wiki/Wikipedia:Autoconfirmed_Proposal/Poll -- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com ‪“We're living in pieces, I want to live in peace.” – T. Moore‬

3 3

Only parts of EventLogging events getting written to the database since 2015-01-07 ~1:55
by Christian Aistleitner 08 Jan '15

08 Jan '15

Hi, just a quick heads up that since 2015-01-07 ~1:55 only <30% of the EventLogging events are getting written to the database. It seems a deployment went wrong and validation is no longer working as expected. Somewhere >70% of the messages no longer validate and hence do not get written to the database. The raw log files (pre-validation) are still getting written, so data is not lost, and backfilling is possible. Best regards, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

6 10

Pageviews update
by Oliver Keyes 08 Jan '15

08 Jan '15

I'm pleased to say we now have the prototype pageviews definition as a UDF! For those with cluster access: CREATE TEMPORARY FUNCTION pageview as 'org.wikimedia.analytics.refinery.hive.isPageviewUDF'; ...and then just apply it. It outputs a boolean, so you can easily go WHERE is.Pageview(fields) and treat it as a conditional. Great success! What this means for the definition is twofold; it means it's a lot easier to tests it accuracy, and it means that it's a lot easier to make sure we're all using the same definition going forward. Once we have the legacy definition as a UDF, refining and testing will proceed at great speed, although I encourage anyone with time on their hands who wants to help out to do some testing of their own :) -- Oliver Keyes Research Analyst Wikimedia Foundation

4 6

A good Samza article
by Andrew Otto 08 Jan '15

08 Jan '15

http://thenewstack.io/apache-samza-linkedins-framework-for-stream-processin… <http://thenewstack.io/apache-samza-linkedins-framework-for-stream-processin…>

2 1

Mail list in subscription
by masssly 08 Jan '15

08 Jan '15

Please unsubscribe me from this mailing list. Thank you. -Masssly Sent from Samsung Mobile

1 0

Setting up eventlogging-devserver
by Roxana Necula 07 Jan '15

07 Jan '15

Hi, When trying to run 'eventlogging-devserver' command, like suggested in the EventLogging guide [1], I get an error [2]. This happens in Vagrant and also on my local environment. In README.rst from EventLogging/server/ I found some small dependencies and I installed them. I also tried this after a fresh install of MediaWiki, and after deleting the egg files from /vagrant/mediawiki/extensions/EventLogging/server/ but with the same result. This is what I have added in my LocalSettings.php [3]. I mention that I have enabled the eventlogging role in Vagrant. And when verifying the setup indicated here [4] the output is "ready" and it gives me an object. I was wondering if I have a bad setup or if I missed something. [1] https://www.mediawiki.org/wiki/Extension:EventLogging/Guide#Installing_the_… [2] http://pastebin.com/C6bkVpHU [3] http://pastebin.com/tyRt7mk8 [4] https://www.mediawiki.org/wiki/Extension:EventLogging#Developer_setup Thank you, Roxana Necula

2 1

Troubleshooting
by Jon Katz 07 Jan '15

07 Jan '15

Hi, This is not urgetn, but if anyone is interested in troubleshooting a (probably basic) problem I am having with mysql, I would appreciate it! I am trying to query the MobileWebClickTracking table, which is ginormous and keeps timing out or boiling my RAM So Dario suggested I use screen to dump a section of the table into a more workable table. Here is his test query that seemed to work: *screen mysql -h analytics-store.eqiad.wmnet -B -e "CREATE TABLE staging.jkatz_foo SELECT * FROM enwiki.user LIMIT 300;"* I tried to do this on my own (I use analytics-slave instead, as my credentials don't seem to work on analytics-store), and it doesn't seem to do anything. Can you try on your end and let me know if you're having any luck? Here is the query: jkatz@bast1001:~$ screen jkatz@bast1001:~$ mysql -h analytics-slave.eqiad.wmnet -u research -pJoFjnA90Ajyp -B -e "Insert into staging.jkatz_clicktracking1 Select * from log.MobileWebClickTracking_5929948 WHERE ('timestamp' between 20141101000000 and 20141130000000) and wiki like 'enwiki';" When I look in processes in stat1003, I only see a sleep command--no query: [image: Inline image 2] Is the query within that sleep? Anyway, I seem to be creating tables but they do not have any rows in them. I have double checked that the data between those dates exists in that table. Any thoughts? Best, J

4 6

webrequest bits kafka traffic
by Andrew Otto 07 Jan '15

07 Jan '15

Hi yalls, Christian and I were talking a bit today about how to figure out why high traffic (bits and upload) esams varnishes occasionally have latency issues[1] which cause buffers to fill up which causes a small amount of message loss[2]. We aren’t totally sure if this is an overall system throughput issue (network and/or Kafka brokers), or just something that might be fixable by tweaking more configs on the individual varnishkafkas. We don’t think that anyone is using the webrequest bits data. It isn’t included in udp2log, and therefore doesn’t affect any legacy analytics. Nor are we using it for any productionized analytics in Hadoop. We aren’t sure if others are relying on this data for adhoc analysis though. If no one objects, we’d like to temporarily disable the bits varnishkafka instances. If we do and the uploads esams varnishes then stop having problems, we will know that this is a system throughput issue. If we continue to have problems with esams uploads, then we will know that it is more likely a local varnishkafka issue. So, are there any objections to removing webrequest bits from Kafka webrequest logs for a little while? -Ao [1] varnishkafka rtt average: http://grafana.wikimedia.org/#/dashboard/db/kafka?from=1420484972895&to=142… <http://grafana.wikimedia.org/#/dashboard/db/kafka?from=1420484972895&to=142…> or http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&hreg[]=%28amss… <http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&hreg%5B%5D=%28…> [2] varnishkafka delivery errors: http://grafana.wikimedia.org/#/dashboard/db/kafka?from=1420484972895&to=142… <http://grafana.wikimedia.org/#/dashboard/db/kafka?from=1420484972895&to=142…> or http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&hreg[]=%28amss… <http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&hreg%5B%5D=%28…>

1 0

Discussion about target URL for varnish monitoring requests (Angelsberg/Main_Page -> Undefined ?)
by Christian Aistleitner 03 Jan '15

03 Jan '15

Hi, TL;DR: If “Angelsberg” does not ring a bell, you're not affected :-) Otherwise, you probably run machinery against request logs that cares about filtering requests made by WMF monitoring infrastructure. Just a heads up that it seems at https://gerrit.wikimedia.org/r/#/c/182558/ a discussion is starting on how to detect such monitoring requests and how they might change. Chime in there, or at least watch it's outcome to make sure your scripts keep on detecting/filtering monitoring requests. Have fun, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Analytics January 2015