Wikitech-l August 2016

wikitech-l@lists.wikimedia.org

78 participants
101 discussions

Readership metrics for the timespan until July 31, 2016
by Tilman Bayer 08 Aug '16

08 Aug '16

Hi all, this resumes the usual look <https://www.mediawiki.org/wiki/Reading/Readership_metrics_reports> at our most important readership metrics. This edition of the report introduces a new year-over-year comparison chart to better understand which traffic changes are seasonal, and highlights a recent browser-related issue that has, suddenly and probably artificially, increased our global usage by 11%. We also take a first look of how the ratio of pageviews per device has been developing over time, and point out some other events that had an impact on core metrics. As laid out earlier <https://lists.wikimedia.org/pipermail/mobile-l/2015-September/009773.html>, the main purpose is to raise awareness about how these are developing, call out the impact of any unusual events, and facilitate thinking about core metrics in general. As always; feedback and discussion welcome. Week-over-week and month-over-month changes are now being recorded on the Product page <https://www.mediawiki.org/wiki/Wikimedia_Product#Reading> at MediaWiki.org. This edition of the report covers a timespan of 14 weeks. Some other recent items of interest, in case they didn’t already catch your attention: - The WMF Reading team published its quarterly review presentation for Q4 2015-16 (April-June), which includes lots of traffic and usage data <https://commons.wikimedia.org/w/index.php?title=File:Reading_Quarterly_Revi…> . - I gave an update on long-term traffic trends at the Foundation’s metrics meeting in May, see slides <https://commons.wikimedia.org/w/index.php?title=File%3AWMF_Metrics_%26_Acti…> (and will again later this month). Now to the usual data. (All numbers below are averages for April 25-July 31, 2016 unless otherwise noted.) Pageviews Total: 511 million/day (-5.0% from the previous report) Context (April 2015-July 2016): <https://commons.wikimedia.org/wiki/File:Wikimedia_daily_pageviews,_all_vs._…> See also the Vital Signs dashboard <https://analytics.wikimedia.org/dashboards/vital-signs/#projects=all/metric…> (Small caveat: Android app pageviews in April and early May were undercounted by up to 1-2 million/dayl due to a bug <https://phabricator.wikimedia.org/T132965>.) Overall pageviews developed relatively uneventfully during the timespan of this report until around July 20, when the desktop traffic to the main pages of several large Wikipedias (including English, Russian and Dutch) suddenly increased drastically. The investigation <https://phabricator.wikimedia.org/T141506> is still ongoing. At this point though it seems likely that this is not due an undetected spider or such, but caused by browsers of real users behaving in an unusual way, namely an old version of Chrome on Windows. Overall pageviews for all projects appear to have increased around 11% due to this. To improve our understanding of which traffic movements are seasonal and which may indicate lasting changes, I have pieced together a chart overlaying the total pageview numbers back to May 2013 (the earliest time for which we have data according to the current pageview definition): <https://commons.wikimedia.org/wiki/File:Wikimedia_pageviews_year-over-year_…> For example, we talked before about how the large year-over-year decrease seen in earlier months (e.g. -11.9% from April 2015 to April 2016) could probably be attributed to two one-time events in May/June 2015: the switch to HTTPS-only connections and the block of the Chinese Wikipedia in China. At the time this was based on more indirect evidence, in particular changes in the ratio of pageviews from the Global North. Now that a year has passed, we can directly compare data from after these events year-over-year. It confirms that conclusion (in the chart, compare the red line against the other three on the left hand side). In July 2016 there actually was a 1.1% increase compared to July 2015. Of course this is partly due to the aforementioned main pages anomaly, which is also visible clearly in this chart. Desktop: 54.1% (previous report: 55.2%) Mobile web: 44.6% (previous report: 43.6%) Apps: 1.3% (previous report: 1.2%) (small caveat: app percentage is a bit too low due to the aforementioned Android-related bug in April/May) Context (April 2015-July 2016): <https://commons.wikimedia.org/wiki/File:Wikimedia_daily_pageviews,_mobile_p…> After largely stagnating earlier this year following the christmas bump, the mobile percentage started to rise again towards parity around May (reaching 48.6% in the week until July 11). However, the recent anomalously high main pages traffic described above was entirely on desktop and has thus, perhaps temporarily, brought this percentage down again (to 42.7% in the week until July 31). As a reminder, mobile already has a solid majority in terms of unique devices, cf. below. Global North ratio: 75.5% of total pageviews (previous report: 76.5%) Context (last seven months): After a notable decrease (or, increase in the Global South ratio) at the beginning of the year, this number was relatively steady over the timespan of this report. NB: We are currently rethinking this metric and might replace it with a different country selection constructed as part of the work on the New Readers project. Unique devices See the announcement blog post <https://blog.wikimedia.org/2016/03/30/unique-devices-dataset/> from March for background and details on this recently introduced metric. These estimated numbers are provided for all Wikimedia language projects (separately for the desktop and mobile web version). Because of the instrumentation method, there is no global metric for all projects and all languages, but following some recent discussions it is now planned <https://phabricator.wikimedia.org/T138027> to extend it to a cross-language global metric per project at least. For now, we track the daily numbers of the English Wikipedia in this report. Daily unique devices estimate for English Wikipedia: - Mobile web: 32.2 million (previous report: 33.2 million) - Desktop: 22.3 million (previous report: 25.7 million) - Total: 54.5 million (previous report: 58.9 million) - Average mobile web ratio: 59.3% (previous report: 56.5%) Context (January-August 2016): See also the new Vital Signs dashboard <https://analytics.wikimedia.org/dashboards/vital-signs/#projects=enwiki/met…> As mentioned in the previous report, there was a drop in early to mid March which hasn’t seen an explanation so far. This metric hasn’t existed long enough yet to get a good sense of what yearly seasonalities may exist, but it looks a bit like a decreasing trend in desktop devices during these seven months. Interestingly though, pageviews did not decrease as much. Or to put it differently, pageviews per device rose during this time (even if one disregards the aforementioned main page increase since the end of July): <https://commons.wikimedia.org/wiki/File:Daily_views_per_unique_device_for_E…> New app installations Android:19.7k/day (-37.7% from the previous report) Daily installs per device, from Google Play Context (last six months): As described in the previous report, in March a Google bug had accidentally increased visitor number to the app’s Play Store listing enormously (spike cut off in this chart). So this it’s not surprising that the average install rate over the timespan of this report is much lower. Even disregarding this factor though, the install rates have been decreasing over the last few months and are matched by uninstalls on most days (except for a bump during two weeks in July whose reason is unknown to me). Combined with devices going inactive etc. this has meant that the app’s install base has been shrinking, from from 15.5 million on March 31 to 15.1 million today. On July 28, an updated version of the app with several exciting new features was launched, with a blog post <https://blog.wikimedia.org/2016/07/28/wikipedia-android-redesign/> that had very high pageview numbers and was picked up by media outlets in various countries. Unfortunately this public attention has not translated into a lot of downloads - we estimate around 3000 (a few hours’ worth at the normal rate). In comparison, the relaunch of the OS app in March brought roughly 50k additional installs (see previous report). iOS: 3.18 k/day (-37.2% from the previous report) Download numbers from App Annie Context (last six months): As mentioned, the iOS app’s huge relaunch fell into the timespan of the previous report, so the decrease here is not surprising. App user retention Android: 16.7% (previous report: 15.4%) (Ratio of app installs opened again 7 days after installation, among all installed one week before a date that falls within this report. 1:100 sample) Context (last six months): As remarked in earlier reports, this data is a bit too noisy for drawing conclusions about whether retention changed significantly between different releases. But we can at least rule out the existence of major shifts during this timespan. iOS: N/A <https://phabricator.wikimedia.org/T126693> Unique app users Android: 1.145 million / day (-5.0% from the previous report timespan, reconstructed <https://phabricator.wikimedia.org/T132965>) Context (last six months): Continuing the slight but steady decline that had been interrupted by the aforementioned Google bug windfall in March. iOS: N/A <https://phabricator.wikimedia.org/T130432> While we don’t have total active user numbers available for the iOS app any more, let’s mention an interesting increase that happened in the app’s pageviews back in May, by around 20%. (Because of the app’s small share in overall traffic, it did not affect the global pageviews discussed above in a notable way.) After some investigation, it turned out this coincided with the release of version 5.0.3 of the app, see the yellow slice on the right: Josh and I are planning to look further into possible causes; at the moment it’s still possible that this is either an artefact or due to real improvements in the app. ---- For reference, the queries and source links used are listed below (access is needed for each). Unless otherwise noted, all content of this report is © Wikimedia Foundation and released under the CC BY-SA 3.0 <https://creativecommons.org/licenses/by-sa/3.0/> license. Most of the above charts are available on Commons, too <https://commons.wikimedia.org/w/index.php?title=Special:ListFiles&offset=20…>; together with PDF versions of this report. SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, sum(IF(access_method <> 'desktop', view_count, null)) AS mobileviews, SUM(view_count) AS allviews FROM wmf.projectview_hourly WHERE year>0 AND agent_type = 'user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT LEFT(timestamp, 10) AS date, sum(IF(access_method <> 'desktop', pageviews, null)) AS mobileviews, SUM(pageviews) AS allviews FROM staging.pageviews05 WHERE is_spider = FALSE AND is_automata = FALSE GROUP BY date; SELECT access_method, SUM(view_count)/(7*14) FROM wmf.projectview_hourly WHERE agent_type = 'user' AND CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) BETWEEN "2016-04-25" AND "2016-07-31" GROUP BY access_method; SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")), SUM(view_count) AS all, SUM(IF (FIND_IN_SET(country_code, 'AD,AL,AT,AX,BA,BE,BG,CH,CY,CZ,DE,DK,EE,ES,FI,FO,FR,FX,GB,GG,GI,GL,GR,HR,HU,IE,IL,IM,IS,IT,JE,LI,LU,LV,MC,MD,ME,MK,MT,NL,NO,PL,PT,RO,RS,RU,SE,SI,SJ,SK,SM,TR,VA,AU,CA,HK,MO,NZ,JP,SG,KR,TW,US') > 0, view_count, 0)) AS Global_North_views FROM wmf.projectview_hourly WHERE year > 0 AND agent_type='user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; https://console.developers.google.com/storage/browser/pubsite_prod_rev_0281… (“overview”) https://www.appannie.com/dashboard/252257/item/324715238/downloads/ (select “Total”) SELECT LEFT(timestamp, 8) AS date, SUM(IF(event_appInstallAgeDays = 0, 1, 0)) AS day0_active, SUM(IF(event_appInstallAgeDays = 7, 1, 0)) AS day7_active FROM log.MobileWikiAppDailyStats_12637385 WHERE userAgent LIKE '%-r-%' AND userAgent NOT LIKE '%Googlebot%' GROUP BY date ORDER BY DATE; (with the retention rate calculated as day7_active divided by day0_active from seven days earlier, of course) SELECT CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, unique_count AS Android_DAU FROM wmf.mobile_apps_uniques_daily WHERE platform = 'Android'; SELECT year, month, day, CONCAT(year,'-',LPAD(month,2,'0'),'-',LPAD(day,2,'0')) AS date, CONCAT(SPLIT(user_agent_map['wmf_app_version'], '\\.')[0], COALESCE( CONCAT('.', SPLIT(user_agent_map['wmf_app_version'], '\\.')[1]), ''), COALESCE( CONCAT('.', SPLIT(user_agent_map['wmf_app_version'], '\\.')[2]), '')) AS app_version, SUM(view_count) AS views FROM wmf.pageview_hourly WHERE year >0 AND access_method = 'mobile app' AND user_agent_map['os_family'] = 'iOS' AND agent_type = 'user' GROUP BY year, month, day, user_agent_map['wmf_app_version'] ORDER BY year, month, day, app_version LIMIT 20000; -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB

1 0

Wikidata descriptions to help disambiguate article topic on mobile web
by Moushira Elamrawy 08 Aug '16

08 Aug '16

Greetings, The Reading team <https://www.mediawiki.org/wiki/Reading> [0] would like to help readers learn faster about the topics they are browsing on our mobile site, by displaying Wikidata descriptions underneath article title. This has been the case on apps for a while now, and the team would like to move with the same practice to mobile web, and help our mobile readers, by using content from a Wikipedia sister project. There is a page the describes the idea and the details here <https://www.mediawiki.org/wiki/Reading/web/Projects/Wikidata_Descriptions> [1] .Please check to learn more about the rationale and the details of the suggested change. Thank you, Moushira [0] https://www.mediawiki.org/wiki/Reading [1] https://www.mediawiki.org/wiki/Reading/web/Projects/Wikidata_Descriptions

1 0

The Revision Scoring weekly update
by Aaron Halfaker 08 Aug '16

08 Aug '16

Hey folks, This is the 16th weekly update from revision scoring team that we have sent to this mailing list. New developments: - We created dashboards for the ORES service in the Beta cluster[1] and created panes for tracking failed jobs[2]. - We extended the documentation for the ORES review tool[3,4] Maintenance: - We did some work to make the Beta cluster look more like production so that we can do better testing before the next deployment - We set up a password on the Beta redis server[5] - We configured the Beta ORES extension to actually use the Beta ORES service[6] - We also prepared a set of puppet changes for the deployment of a refactored version of ORES to production[7] Issues in WMFLabs - We investigated a series of timeout errors that were appearing in the logs[8] - We investigated a periodic redis-related error that shower up when scoring edits[9] - We fixed our "05" web node that was periodically running out of memory[10] Estimating future resource needs - In preparation for buying new hardware, we measured our past memory usage and extrapolated forward two years to estimate what hardware requirements we'll have[11] 1. https://phabricator.wikimedia.org/T142294 - Dashboard or pane for ORES service in beta 1. https://phabricator.wikimedia.org/T142119 - Dashboard or pane for ORES failed jobs on beta 1. https://phabricator.wikimedia.org/T140150 - Make user-centered documentation for review tool 1. https://www.mediawiki.org/wiki/ORES_review_tool 1. https://phabricator.wikimedia.org/T141823 - Set up password on ORES Beta redis server 1. https://phabricator.wikimedia.org/T141825 - Config beta ORES extension to use the beta ORES service 1. https://phabricator.wikimedia.org/T141575 - Puppet config changes for ORES refactor 1. https://phabricator.wikimedia.org/T141368 - [Investigate] ORES time out errors in logs 1. https://phabricator.wikimedia.org/T141946 - [Investigate] Periodic redis related errors in wmflabs 1. https://phabricator.wikimedia.org/T141523 - [Investigate] web-05 downtime 1. https://phabricator.wikimedia.org/T142046 - Extrapolate memory usage per worker forward 2 years Sincerely, Aaron from the Revision Scoring team

1 0

New scap version is live
by Tyler Cipriani 08 Aug '16

08 Aug '16

tl;dr: The new scap version is live in production. It has canary deploys. Scap v.3.2.2-1 was deployed to production today. There are some new internal improvements as well as some that are user-facing. The improvements you'll probably notice are: * Tab completion works for scap subcommands(!) * Canary checks for MediaWiki deployments Canary deployments: 1. Sync your change(s) to the api and appserver canary hosts 2. Wait (20 seconds) for traffic to hit those host 3. If there isn't a large increase in the error rate on those hosts (10x), release your changes to the remainder of the fleet. If, for whatever reason, you find yourself in a position where you don't care about the error-rate change on canary nodes, use the --force flag, i.e.: scap sync-file --force README 'Important README update' Please report any problems in a phab ticket tagged with "Scap3" or in #wikimedia-releng in IRC on freenode. <3, Your Hometown Release Engineers

2 2

Discovery Weekly Update for the week starting 2016-08-01
by Chris Koerner 08 Aug '16

08 Aug '16

Hello, Apologies for the late email. Here are the updates from the Discovery department for this past week. * Search: stripping of '?' in search queries released to production [1] * Search: namespace missing when search results from different language are shown released to production [2] * Portal: updated stats on wikipedia.org [3] * Portal: fixed language links that are less than 1,000 that were showing as zero [4] * Portal: community member fix pushed to production (dark background in search box on Linux OS) [5] * Analysis: updated geographical breakdown to filter out larger countries on portal dashboard [1] https://lists.wikimedia.org/pipermail/discovery/2016-August/001231.html [2] https://phabricator.wikimedia.org/T141503 [3] https://phabricator.wikimedia.org/T128546 [4] https://phabricator.wikimedia.org/T139350 [5] https://phabricator.wikimedia.org/T139349 [6] https://phabricator.wikimedia.org/T139548 ---- The full update, and archive of past updates, can be found on Mediawiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as Easy or volunteer needed in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

CirrusSearch index not update after edits
by Aran 08 Aug '16

08 Aug '16

Hi, I've installed the current CirrusSearch on a MediaWiki 1.27 with local ElasticSearch 1.7.5 and everything went smoothly, except that the index doesn't update by itself after I edit pages. Only if I re-run the maintenance script do new edits get included in search results. I don't have $wgDisableSearchUpdate=true in LocalSettings.php and have tried settings it to false too. Any ideas what might be the trouble? Thanks, Aran

2 2

RESTBase multiple pages with one request
by Toni Hermoso Pulido 07 Aug '16

07 Aug '16

Hello, is it already possible to retrieve data from different pages just by using one request? E.g by combining: https://en.wikipedia.org/api/rest_v1/page/summary/Electron and https://en.wikipedia.org/api/rest_v1/page/summary/Dog Thanks! -- Toni Hermoso Pulido http://www.cau.cat http://www.similis.cc

3 5

Acquiring list of templates including external links
by Takashi OTA 06 Aug '16

06 Aug '16

Hoi, This is an inquiry from my friend in academia, researching about Wikipedia. He would like to know whether there's a way to acquire a list of templates including external links. Here are some examples including external links. https://ja.wikipedia.org/wiki/Template:JOI/doc https://ja.wikipedia.org/wiki/Template:Twitter/doc Such links are stored in externallinks.sql.gz, in an expanded form. When you want to check increase/decrease of linked domains in chronological order through edit history, you have to check pages-meta-history1.xml etc. In a such case, traditional links and links by templates are mixed, Therefore, the latter ones (links by templates) should be expanded to traditional link forms. Sorry if what I am saying does not make sense. Thanks in advance, --Takashi Ota [[U:Takot]]

12 23

Re: [Wikitech-l] JetBrains IDE licenses for MW developers - php, js/node, python, C/C++, puppet, ruby, ...
by Florian Schmidt 05 Aug '16

05 Aug '16

We've an all products Plan, so all IDEA Desktop products are included, so I'm pretty sure, you can use our existing subscriptions. Best, Florian

1 0

Re: [Wikitech-l] JetBrains IDE licenses for MW developers - php, js/node, python, C/C++, puppet, ruby, ...
by Ilya Korniyko 05 Aug '16

05 Aug '16

Did not get the answer, so repeating in the mailing list. We might need Idea license to support https://commons.wikimedia.org/wiki/Commons:WLX_Jury_Tool for Wiki loves monuments as it is the best IDE for Scala and we want to onboard someone except me to support the jury tool. Regards, Ilya On Mon, Nov 2, 2015 at 8:31 PM, Ilya Korniyko <intracer(a)gmail.com> wrote: > Hi Yuri, > Does the license work for Idea 15 that is released today? > > It says: License key is in legacy format. Press Convert button, and after > pressing Convert asks for First name, Last Name and Email > When I enter my name and email, it says that Your license key cannot be > found in the JetBrains database. > >>>>> >>>>> Date: Mon, 19 Oct 2015 23:02:02 +0300 >>>>>> From: Yuri Astrakhan <yastrakhan(a)wikimedia.org> >>>>>> To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org> >>>>>> Subject: [Wikitech-l] JetBrains IDE licenses for MW developers - php, >>>>>> js/node, python, C/C++, puppet, ruby, ... >>>>>> Message-ID: >>>>>> <CALOOOkiP-vsX6he4i451Dg5TcH2vtDvdYkF4cr2 >>>>>> K3c9nFZCmWQ(a)mail.gmail.com> >>>>>> Content-Type: text/plain; charset=UTF-8 >>>>>> >>>>>> Developers, we now have licenses for JetBrains' IDEA Ultimate >>>>>> <https://www.jetbrains.com/idea/> (JavaScript, PHP, Python, ...) >>>>>> and CLion >>>>>> <https://www.jetbrains.com/clion/> (C/C++). The IDE supports >>>>>> debugging in >>>>>> Vagrant and on-the-fly static code analysis. >>>>>> >>>>>> If you are a volunteer developer, and want to use an IDE, contact me >>>>>> for a >>>>>> license. Those with access to the office wiki may register directly >>>>>> at the >>>>>> licenses <https://office.wikimedia.org/wiki/JetBrains> page. >>>>>> >>>>>> Please try it for a bit before getting a license -- just in case you >>>>>> don't >>>>>> like it. >>>>>> >>>>>> P.S. This is only for those who want JetBrains' IDE. If you are happy >>>>>> with >>>>>> Vim, good for you. If you like Atom or Sublime, great. If notepad is >>>>>> the >>>>>> best text editor of all times, all the power to you. Bikeshedding >>>>>> should >>>>>> go into a separate thread )) >>>>>> >>>>>> >>>>>> >>>> >>> >> >

1 0

← Newer
1
...
5
6
7
8
9
10
11
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l August 2016