Analytics August 2016

analytics@lists.wikimedia.org

24 participants
23 discussions

by Nuria Ruiz

Maria, You can ask questions such as this one in analytics@ public e-mail list (cc-ed). Downloadable files with pageview counts are available here: https://dumps.wikimedia.org/other/pageviews/ Format of those files is described here: https://wikitech.wikimedia.org/wiki/Analytics/Data/Pageviews#Availability Thanks, Nuria ---------- Forwarded message ---------- From: MARIA ELENA ALONSO MENCIA <100304201(a)alumnos.uc3m.es> Date: Wed, Aug 24, 2016 at 4:41 AM Subject: Vital Signs To: nuria(a)wikimedia.org Dear Nuria Ruiz, My name is Maria Elena Alonso and I am an Spanish student doing my thesis in Stuttgart now. I have found 'Vital Signs' dashboard ( https://analytics.wikimedia.org/dashboards/vital-signs/# projects=enwiki/metrics=Pageviews) and the data is just exactly the one I need for my thesis. I tried to find any contact that could help me and through github I found your account (I hope it is okey for you, I do not try to cause any problem), but please forward my message to any contact if you are not able to help me. I just need to know if there is anyway that I can download the displayed values by day (writing the values per day by looking at the plot couls take too long). Thank you in advance for your help and sorry again for any inconvenience I can cause you. Yours sincerely, Elena

7 years, 8 months

Weigh in on whether to finalize "Diversity" and “Conflict of interest” sections of Code of Conduct

by Matthew Flaschen

Please comment on whether to approve the "Diversity" and "Conflict of interest" section of the draft Code of conduct for technical spaces: * Diversity - https://www.mediawiki.org/wiki/Code_of_Conduct/Draft#Diversity * Conflict of interest - https://www.mediawiki.org/wiki/Code_of_Conduct/Draft#Conflict_of_interest You can comment at https://www.mediawiki.org/wiki/Talk:Code_of_Conduct/Draft#Finalize_.22Diver… . A position and brief comment is fine. You can also send private feedback to conduct-discussion(a)wikimedia.org . Thanks again, Matt Flaschen

7 years, 8 months

Upcoming Research Showcase - August 17, 2016

by Sarah R

Hi Everyone, The next Research Showcase will be live-streamed this Wednesday, Aug 17, 2016 at 11:30 AM (PST) 18:30 (UTC). YouTube stream: http://youtu.be/rsFmqYxtt9w As usual, you can join the conversation on IRC at #wikimedia-research. And, you can watch our past research showcases here <https://www.mediawiki.org/wiki/Wikimedia_Research/Showcase#Archive>. This month's showcase includes. Computational Fact Checking from Knowledge NetworksBy *Giovanni Luca Ciampaglia <https://www.mediawiki.org/wiki/User:Junkie.dolphin>*Traditional fact checking by expert journalists cannot keep up with the enormous volume of information that is now generated online. Fact checking is often a tedious and repetitive task and even simple automation opportunities may result in significant improvements to human fact checkers. In this talk I will describe how we are trying to approximate the complexities of human fact checking by exploring a knowledge graph under a properly defined proximity measure. Framed as a network traversal problem, this approach is feasible with efficient computational techniques. We evaluate this approach by examining tens of thousands of claims related to history, entertainment, geography, and biographical information using the public knowledge graph extracted from Wikipedia by the DBPedia project, showing that the method does indeed assign higher confidence to true statements than to false ones. One advantage of this approach is that, together with a numerical evaluation, it also provides a sequence of statements that can be easily inspected by a human fact checker. Deploying and maintaining AI in a socio-technical system. Lessons learnedBy *Aaron Halfaker <https://www.mediawiki.org/wiki/User:Halfak_(WMF)>*We should exercise great caution when deploying AI into our social spaces. The algorithms that make counter-vandalism in Wikipedia orders of magnitude more efficient also have the potential to perpetuate biases and silence whole classes of contributors. This presentation will describe the system efficiency characteristics that make AI so attractive for supporting quality control activities in Wikipedia. Then, Aaron will tell two stories of how the algorithms brought new, problematic biases to quality control processes in Wikipedia and how the Revision Scoring team <https://meta.wikimedia.org/wiki/R:Revision_scoring_as_a_service> learned about and addressed these issues in ORES <https://meta.wikimedia.org/wiki/ORES>, a production-level AI service for Wikimedia Wikis. He'll also make an overdue call to action toward leveraging human-review of AIs biases in the practice of AI development. We look forward to seeing you! -- Sarah R. Rodlund Senior Project Coordinator-Engineering, Wikimedia Foundation srodlund(a)wikimedia.org

7 years, 8 months

Re: [Analytics] Urgent Data Issue

by Nuria Ruiz

Dylan, (cc-ing analytics@ public list) Please see announcement about deprecation of datasets: https://lists.wikimedia.org/pipermail/analytics/2016-August/005339.html Thanks, Nuria On Tue, Aug 16, 2016 at 12:53 PM, Dylan Wenzlau <dylan(a)graphiq.com> wrote: > It seems the pagecounts-all-sites dumps have completely stopped updating, > and I don't see any warning or message about why this is the case or > whether it's currently being resolved. Our company relies pretty heavily on > this data, as I imagine other projects & companies do as well, so I think > it would be useful to at least display a big warning message on the > documentation pages explaining why these are no longer updating. > > Thanks, > > -- > *Dylan Wenzlau* | Director of Engineering | >

7 years, 8 months

pageview types from the new pageviews data set

by Aubrey Rembert

our team is trying to distiguish mobile pageviews from non-mobile pageviews in the new data feed https://dumps.wikimedia.org/other/pageviews/. we are under the impression that the .m and .zero page view type extensions are page views of the mobile site. is this correct? what does the extension .q mean? also, can we safely assume that if a page view has a language but no extension (.m, .zero, .q, etc…), then it is a view of the desktop site? thanks!

7 years, 8 months

WikiConference North America 2016

by Sydney Poore

WikiConference North America 2016 7-10 October 2016, San Diego, CA, USA WikiConference North America (formerly WikiConference USA) is the third annual conference on the North American continent devoted to Wikipedia and other Wikimedia projects. The weekend will feature both academic and casual presentations on Wikimedia-related outreach activities, workshops to improve the skills of grassroots organizers, and discussions on the past, present, and future of the Wikimedia projects. The conference features offerings about community outreach, online activity, partnerships with institutions of knowledge, and technology. Keynote speakers are scheduled to include Katherine Maher, Executive Director of the Wikimedia Foundation, and Merrilee Proffitt, Senior Program Officer of OCLC Research. The last day of the conference will feature programming coinciding with Indigenous Peoples' Day. Registration for the conference is now open. You can register at https://wikiconference.org. Scholarships partially covering costs of travel and attendance are available for active contributors to Wikimedia projects. Apply by August 23rd for scholarships at https://wikiconference.org/wiki/2016/Scholarships. This is a volunteer run conference and volunteers are needed for any number of tasks. If you are attending, please consider volunteering for at https://wikiconference.org/wiki/Volunteers. We seek presentations addressing topics related to Wikipedia or open access and culture. Presentations may be from any discipline regarding any relevant topic. Please submit a description of your proposed presentation using our online submission process at https://wikiconference.org/ wiki/Submissions. If you are interested in participating in the peer-reviewed academic track, see our call for academic submissions at https://wikiconference.org/wiki/Call_for_Academic_Presentations. - Sydney Poore (User:FloNight) and Rosie Stephenson-Goodknight (User:Rosiestep), conference organizers

7 years, 8 months

Wiki pagecounts-raw is missing singe 08/05

by Kushal Tayal

Hi, I noticed that there are no wikipage counts that have been published since 08/05 - https://dumps.wikimedia.org/other/pagecounts-raw/2016/2016-08. Is this a known bug? Will this get fixed soon? Thanks, Kushal

7 years, 8 months

pagecounts-all-sites

by Dylan Wenzlau

Hello analytics team! It seems that your amazingly useful pageview dumps <https://dumps.wikimedia.org/other/pagecounts-all-sites/2016/2016-08/> stopped updating on August 5th, a week ago. Was there a planned migration or retirement that happened to these dump files, or is there a technical difficulty? Forgive me if I missed the announcement, I read through the various pagecounts wikis and didn't find anything. We love being able to use the wiki pageview dumps, they're tremendously useful. -- *Dylan Wenzlau* | Director of Engineering |

7 years, 8 months

Re: [Analytics] Analytics dashboards by country

by Nuria Ruiz

Stephen: Please direct questions such as this one to analytics@ as you will get a faster response. >Back when stats.wikimedia.org was being actively updated, we relied on the pageviews per country reports as evidence during some of our trademark applications. We still use stats.wikimedia.org and we will not replace it completely for quite some months. The new dashboards are "flashier" but *data source is the same for both places when it comes to pageviews.* Seems to me that this report has the most up-to-date data and you can use it to prioritize trademark registrations: https://stats.wikimedia.org/wikimedia/squids/SquidReportPageViewsPerCountry… See note " *Feb 2016: This report has been upgraded" * Thanks, Nuria On Thu, Aug 11, 2016 at 1:46 PM, Stephen LaPorte <slaporte(a)wikimedia.org> wrote: > Hi Nuria, > > I have some questions about the views/unique devices per country stats in > the analytics dashboards here: https://analytics.wikimedia.org > > Back when stats.wikimedia.org was being actively updated, we relied on > the pageviews per country reports as evidence during some of our trademark > applications. Local trademark authorities usually ask us to provide > evidence that we have readership in their country over a certain time > period. We used to be able to print these stats from the stats portal -- > will we be able to do something similar in the analytics dashboards? We > also used the total views per country to choose how we prioritize trademark > registrations in the places where we have more readership. Is there a > dashboard to see countries ranked by views? > > I really like the new graphs and data available in the dashboards. I'm > happy to see that we're updating the place where we track this sort of > important data. > > Best, > Stephen > > -- > Stephen LaPorte > Senior Legal Counsel > Wikimedia Foundation > > *NOTICE: This message may be confidential or legally privileged. If you > have received it by accident, please delete it and let us know about the > mistake. As an attorney for the Wikimedia Foundation, for legal and ethical > reasons, I cannot give legal advice to, or serve as a lawyer for, community > members, volunteers, or staff members in their personal capacity. For more > on what this means, please see our legal disclaimer > <https://meta.wikimedia.org/wiki/Wikimedia_Legal_Disclaimer>.* >

7 years, 8 months

Readership metrics for the timespan until July 31, 2016

by Tilman Bayer

Hi all, this resumes the usual look <https://www.mediawiki.org/wiki/Reading/Readership_metrics_reports> at our most important readership metrics. This edition of the report introduces a new year-over-year comparison chart to better understand which traffic changes are seasonal, and highlights a recent browser-related issue that has, suddenly and probably artificially, increased our global usage by 11%. We also take a first look of how the ratio of pageviews per device has been developing over time, and point out some other events that had an impact on core metrics. As laid out earlier <https://lists.wikimedia.org/pipermail/mobile-l/2015-September/009773.html>, the main purpose is to raise awareness about how these are developing, call out the impact of any unusual events, and facilitate thinking about core metrics in general. As always; feedback and discussion welcome. Week-over-week and month-over-month changes are now being recorded on the Product page <https://www.mediawiki.org/wiki/Wikimedia_Product#Reading> at MediaWiki.org. This edition of the report covers a timespan of 14 weeks. Some other recent items of interest, in case they didn’t already catch your attention: - The WMF Reading team published its quarterly review presentation for Q4 2015-16 (April-June), which includes lots of traffic and usage data <https://commons.wikimedia.org/w/index.php?title=File:Reading_Quarterly_Revi…> . - I gave an update on long-term traffic trends at the Foundation’s metrics meeting in May, see slides <https://commons.wikimedia.org/w/index.php?title=File%3AWMF_Metrics_%26_Acti…> (and will again later this month). Now to the usual data. (All numbers below are averages for April 25-July 31, 2016 unless otherwise noted.) Pageviews Total: 511 million/day (-5.0% from the previous report) Context (April 2015-July 2016): <https://commons.wikimedia.org/wiki/File:Wikimedia_daily_pageviews,_all_vs._…> See also the Vital Signs dashboard <https://analytics.wikimedia.org/dashboards/vital-signs/#projects=all/metric…> (Small caveat: Android app pageviews in April and early May were undercounted by up to 1-2 million/dayl due to a bug <https://phabricator.wikimedia.org/T132965>.) Overall pageviews developed relatively uneventfully during the timespan of this report until around July 20, when the desktop traffic to the main pages of several large Wikipedias (including English, Russian and Dutch) suddenly increased drastically. The investigation <https://phabricator.wikimedia.org/T141506> is still ongoing. At this point though it seems likely that this is not due an undetected spider or such, but caused by browsers of real users behaving in an unusual way, namely an old version of Chrome on Windows. Overall pageviews for all projects appear to have increased around 11% due to this. To improve our understanding of which traffic movements are seasonal and which may indicate lasting changes, I have pieced together a chart overlaying the total pageview numbers back to May 2013 (the earliest time for which we have data according to the current pageview definition): <https://commons.wikimedia.org/wiki/File:Wikimedia_pageviews_year-over-year_…> For example, we talked before about how the large year-over-year decrease seen in earlier months (e.g. -11.9% from April 2015 to April 2016) could probably be attributed to two one-time events in May/June 2015: the switch to HTTPS-only connections and the block of the Chinese Wikipedia in China. At the time this was based on more indirect evidence, in particular changes in the ratio of pageviews from the Global North. Now that a year has passed, we can directly compare data from after these events year-over-year. It confirms that conclusion (in the chart, compare the red line against the other three on the left hand side). In July 2016 there actually was a 1.1% increase compared to July 2015. Of course this is partly due to the aforementioned main pages anomaly, which is also visible clearly in this chart. Desktop: 54.1% (previous report: 55.2%) Mobile web: 44.6% (previous report: 43.6%) Apps: 1.3% (previous report: 1.2%) (small caveat: app percentage is a bit too low due to the aforementioned Android-related bug in April/May) Context (April 2015-July 2016): <https://commons.wikimedia.org/wiki/File:Wikimedia_daily_pageviews,_mobile_p…> After largely stagnating earlier this year following the christmas bump, the mobile percentage started to rise again towards parity around May (reaching 48.6% in the week until July 11). However, the recent anomalously high main pages traffic described above was entirely on desktop and has thus, perhaps temporarily, brought this percentage down again (to 42.7% in the week until July 31). As a reminder, mobile already has a solid majority in terms of unique devices, cf. below. Global North ratio: 75.5% of total pageviews (previous report: 76.5%) Context (last seven months): After a notable decrease (or, increase in the Global South ratio) at the beginning of the year, this number was relatively steady over the timespan of this report. NB: We are currently rethinking this metric and might replace it with a different country selection constructed as part of the work on the New Readers project. Unique devices See the announcement blog post <https://blog.wikimedia.org/2016/03/30/unique-devices-dataset/> from March for background and details on this recently introduced metric. These estimated numbers are provided for all Wikimedia language projects (separately for the desktop and mobile web version). Because of the instrumentation method, there is no global metric for all projects and all languages, but following some recent discussions it is now planned <https://phabricator.wikimedia.org/T138027> to extend it to a cross-language global metric per project at least. For now, we track the daily numbers of the English Wikipedia in this report. Daily unique devices estimate for English Wikipedia: - Mobile web: 32.2 million (previous report: 33.2 million) - Desktop: 22.3 million (previous report: 25.7 million) - Total: 54.5 million (previous report: 58.9 million) - Average mobile web ratio: 59.3% (previous report: 56.5%) Context (January-August 2016): See also the new Vital Signs dashboard <https://analytics.wikimedia.org/dashboards/vital-signs/#projects=enwiki/met…> As mentioned in the previous report, there was a drop in early to mid March which hasn’t seen an explanation so far. This metric hasn’t existed long enough yet to get a good sense of what yearly seasonalities may exist, but it looks a bit like a decreasing trend in desktop devices during these seven months. Interestingly though, pageviews did not decrease as much. Or to put it differently, pageviews per device rose during this time (even if one disregards the aforementioned main page increase since the end of July): <https://commons.wikimedia.org/wiki/File:Daily_views_per_unique_device_for_E…> New app installations Android:19.7k/day (-37.7% from the previous report) Daily installs per device, from Google Play Context (last six months): As described in the previous report, in March a Google bug had accidentally increased visitor number to the app’s Play Store listing enormously (spike cut off in this chart). So this it’s not surprising that the average install rate over the timespan of this report is much lower. Even disregarding this factor though, the install rates have been decreasing over the last few months and are matched by uninstalls on most days (except for a bump during two weeks in July whose reason is unknown to me). Combined with devices going inactive etc. this has meant that the app’s install base has been shrinking, from from 15.5 million on March 31 to 15.1 million today. On July 28, an updated version of the app with several exciting new features was launched, with a blog post <https://blog.wikimedia.org/2016/07/28/wikipedia-android-redesign/> that had very high pageview numbers and was picked up by media outlets in various countries. Unfortunately this public attention has not translated into a lot of downloads - we estimate around 3000 (a few hours’ worth at the normal rate). In comparison, the relaunch of the OS app in March brought roughly 50k additional installs (see previous report). iOS: 3.18 k/day (-37.2% from the previous report) Download numbers from App Annie Context (last six months): As mentioned, the iOS app’s huge relaunch fell into the timespan of the previous report, so the decrease here is not surprising. App user retention Android: 16.7% (previous report: 15.4%) (Ratio of app installs opened again 7 days after installation, among all installed one week before a date that falls within this report. 1:100 sample) Context (last six months): As remarked in earlier reports, this data is a bit too noisy for drawing conclusions about whether retention changed significantly between different releases. But we can at least rule out the existence of major shifts during this timespan. iOS: N/A <https://phabricator.wikimedia.org/T126693> Unique app users Android: 1.145 million / day (-5.0% from the previous report timespan, reconstructed <https://phabricator.wikimedia.org/T132965>) Context (last six months): Continuing the slight but steady decline that had been interrupted by the aforementioned Google bug windfall in March. iOS: N/A <https://phabricator.wikimedia.org/T130432> While we don’t have total active user numbers available for the iOS app any more, let’s mention an interesting increase that happened in the app’s pageviews back in May, by around 20%. (Because of the app’s small share in overall traffic, it did not affect the global pageviews discussed above in a notable way.) After some investigation, it turned out this coincided with the release of version 5.0.3 of the app, see the yellow slice on the right: Josh and I are planning to look further into possible causes; at the moment it’s still possible that this is either an artefact or due to real improvements in the app. ---- For reference, the queries and source links used are listed below (access is needed for each). Unless otherwise noted, all content of this report is © Wikimedia Foundation and released under the CC BY-SA 3.0 <https://creativecommons.org/licenses/by-sa/3.0/> license. Most of the above charts are available on Commons, too <https://commons.wikimedia.org/w/index.php?title=Special:ListFiles&offset=20…>; together with PDF versions of this report. SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, sum(IF(access_method <> 'desktop', view_count, null)) AS mobileviews, SUM(view_count) AS allviews FROM wmf.projectview_hourly WHERE year>0 AND agent_type = 'user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT LEFT(timestamp, 10) AS date, sum(IF(access_method <> 'desktop', pageviews, null)) AS mobileviews, SUM(pageviews) AS allviews FROM staging.pageviews05 WHERE is_spider = FALSE AND is_automata = FALSE GROUP BY date; SELECT access_method, SUM(view_count)/(7*14) FROM wmf.projectview_hourly WHERE agent_type = 'user' AND CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) BETWEEN "2016-04-25" AND "2016-07-31" GROUP BY access_method; SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")), SUM(view_count) AS all, SUM(IF (FIND_IN_SET(country_code, 'AD,AL,AT,AX,BA,BE,BG,CH,CY,CZ,DE,DK,EE,ES,FI,FO,FR,FX,GB,GG,GI,GL,GR,HR,HU,IE,IL,IM,IS,IT,JE,LI,LU,LV,MC,MD,ME,MK,MT,NL,NO,PL,PT,RO,RS,RU,SE,SI,SJ,SK,SM,TR,VA,AU,CA,HK,MO,NZ,JP,SG,KR,TW,US') > 0, view_count, 0)) AS Global_North_views FROM wmf.projectview_hourly WHERE year > 0 AND agent_type='user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; https://console.developers.google.com/storage/browser/pubsite_prod_rev_0281… (“overview”) https://www.appannie.com/dashboard/252257/item/324715238/downloads/ (select “Total”) SELECT LEFT(timestamp, 8) AS date, SUM(IF(event_appInstallAgeDays = 0, 1, 0)) AS day0_active, SUM(IF(event_appInstallAgeDays = 7, 1, 0)) AS day7_active FROM log.MobileWikiAppDailyStats_12637385 WHERE userAgent LIKE '%-r-%' AND userAgent NOT LIKE '%Googlebot%' GROUP BY date ORDER BY DATE; (with the retention rate calculated as day7_active divided by day0_active from seven days earlier, of course) SELECT CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, unique_count AS Android_DAU FROM wmf.mobile_apps_uniques_daily WHERE platform = 'Android'; SELECT year, month, day, CONCAT(year,'-',LPAD(month,2,'0'),'-',LPAD(day,2,'0')) AS date, CONCAT(SPLIT(user_agent_map['wmf_app_version'], '\\.')[0], COALESCE( CONCAT('.', SPLIT(user_agent_map['wmf_app_version'], '\\.')[1]), ''), COALESCE( CONCAT('.', SPLIT(user_agent_map['wmf_app_version'], '\\.')[2]), '')) AS app_version, SUM(view_count) AS views FROM wmf.pageview_hourly WHERE year >0 AND access_method = 'mobile app' AND user_agent_map['os_family'] = 'iOS' AND agent_type = 'user' GROUP BY year, month, day, user_agent_map['wmf_app_version'] ORDER BY year, month, day, app_version LIMIT 20000; -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB

7 years, 8 months

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Analytics August 2016