Wikitech-l December 2016

wikitech-l@lists.wikimedia.org

83 participants
77 discussions

phan (new static analysis tool) is now voting on MediaWiki core

by Legoktm

Hi! Thanks to some awesome work by Erik Bernhardson, phan[1], a new static analysis tool, is now voting[2] on MediaWiki core patches. It's significantly more advanced than any of our other current tools, and should help identify some types of errors. It uses PHP 7's AST to process code, but is capable of analyzing PHP5 code. There's documentation on mediawiki.org[3] about how it is currently configured, and how to set it up locally. You'll need PHP 7 with the ast extension to actually run phan. If that's not possible for your system, you can rely on jenkins to run it for you. [1] https://github.com/etsy/phan [2] https://phabricator.wikimedia.org/T132636 [3] https://www.mediawiki.org/wiki/Continuous_integration/Phan -- Legoktm

7 years, 4 months

Wikimedia Developer Summit DRAFT program

by Quim Gil

Hi, the Wikimedia Developer Summit Program committee is happy to announce a draft schedule: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Program#Sche… This schedule includes two plenary sessions: * Has our success made it hard to see your own contribution? <https://phabricator.wikimedia.org/T153024> by Ward Cunningham * Wikimedia Foundation Technology and Product Q&A, with Victoria Coleman (CTO) and Wes Moran (VP of Product). You can propose and vote questions for Victoria and Wes between now and the beginning of the Summit: http://www.allourideas.org/wikidev17-product-technology-questions We welcome feedback about possible improvements to the schedule. Please share your suggestions at the related discussion page <https://www.mediawiki.org/wiki/Talk:Wikimedia_Developer_Summit/2017/Program>. We plan to confirm the final schedule on Friday, December 23rd. We are working on the Unconference, the third day of the Summit, and the social events. We will share more details soon. Stay tuned! -- Quim Gil Engineering Community Manager @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil

7 years, 4 months

No Tech News the weeks of December 26 and January 2

by Johan Jönsson

Hi everyone, As there are no deployments scheduled for the weeks of December 19 and December 26, Tech News[1] will take a hiatus for two weeks, too. The next issue will be distributed to the wikis on Monday; the issue after that one on January 9. If you have something you need included in Tech News before the end of the year, please add the user-notice tag in Phabricator, write me or add it to the current draft[2] before Friday this week. Deadlines and inclusion critera: https://meta.wikimedia.org/wiki/Tech/News/For_contributors [1] https://meta.wikimedia.org/wiki/Tech/News [2] https://meta.wikimedia.org/wiki/Tech/News/2016/51 //Johan Jönsson --

7 years, 4 months

Discussion Platform

by MAYANK JINDAL

Hi, We are using IRC for discussion purpose. How will it be if we change our discussion platform? Many organizations have switched to gitter that have very user-friendly UI and very easy to use. Please give a view on my proposal. -- Kind Regards, Mayank Jindal, Third year undergraduate student, Indian Institute of Technology Kharagpur Mobile : +91- 7076670299 || 8875432718

7 years, 4 months

Fwd: Readership metrics for the timespan until December 4, 2016

by Zareen Farooqui

Hi all, Please see below for the Readership metrics report for the timespan until December 4, 2016. Zareen Farooqui ---------- Forwarded message ---------- From: Zareen Farooqui <zareenf(a)gmail.com> Date: Mon, Dec 12, 2016 at 8:45 PM Subject: Readership metrics for the timespan until December 4, 2016 To: mobile-l(a)lists.wikimedia.org Cc: schang(a)wikimedia.org, slien(a)wikimedia.org, wikitech-l(a)lists.wikimedia.org, analytics(a)lists.wikimedia.org, fr-online(a)wikimedia.org Hi all, This resumes the usual look <https://www.mediawiki.org/wiki/Reading/Readership_metrics_reports> at our most important readership metrics. This time we can report that daily pageviews are up 4.8% (since the last report), with an interesting recent peak which meant that November’s pageviews surpassed those of November 2015 (coincidentally also) by 4.8%, after October had already seen a 2.1% year-over-year increase. The iOS Wikipedia app saw increased downloads, while the Android app’s install base has stopped its previous downward trend. As laid out earlier <https://lists.wikimedia.org/pipermail/mobile-l/2015-September/009773.html>, the main purpose is to raise awareness about how these metrics are developing, call out the impact of any unusual events, and facilitate thinking about core metrics in general. As always; feedback and discussion welcome. Week-over-week and month-over-month changes are now being recorded on the Product page <https://www.mediawiki.org/wiki/Wikimedia_Product#Reading> at MediaWiki.org. This edition of the report covers a timespan of eighteen weeks. Some other recent items of interest, in case they didn’t already catch your attention: - The WMF Reading team published its quarterly review presentation <https://commons.wikimedia.org/wiki/File:Reading_and_Comm_Tech_Quarterly_Rev…> for Q1 2016-17 (July-September), which includes lots of traffic and usage data. - At the Foundation’s August metrics meeting <https://commons.wikimedia.org/w/index.php?title=File%3AAugust_2016_Monthly_…>, the Reading team gave an update on longer-term traffic trends since 2013. (TL;DR: Overall pageviews have been flat to slightly declining, mobile has been steadily rising but recently slowed down, desktop has declining during these three years. However, total pageviews have been slightly increasing year-over-year in the last few months.) See the chart below, updated with data until November: [image: Wikimedia monthly pageviews (desktop+mobile), 2013-2016 (version December 2016).png] In particular, as mentioned, the number of total pageview saw year-over-year increases of +2.1% for October and +4.8% in November, in contrast to e.g. the -10.5% we had for May 2015-May 2016. Now to the usual data. (All numbers below are averages for August 1-December 4, 2016 unless otherwise noted.) Pageviews Total: 529 million/day (+4.76% from the previous report timeframe, with corrected numbers for anomalously high traffic on some main pages <https://phabricator.wikimedia.org/T141506>) Context (April 2015-December 2016): See also the Vital Signs dashboard <https://analytics.wikimedia.org/dashboards/vital-signs/#projects=all/metric…> (Small caveats: iOS app’s pageviews were undercounted by about 1.6 million/day from mid September to early November due to a bug <https://phabricator.wikimedia.org/T148663>.) Overall pageviews increased steadily during the timespan of this report aside from the week ending October 30th (right before Halloween) and the end of November. There appears to be a peak in pageviews in November. To facilitate our understanding of which traffic movements are seasonal and which may indicate lasting changes, here is a chart overlaying the total pageview numbers back to May 2013 (the earliest time for which we have data according to the current pageview definition): The blue line indicates a non-seasonal rise peaking around November 12. We checked whether this peak came from a particular country and were able to exclude that possibility. Wikimedia Daily Pageviews from US Pageviews in US do not show any drastic changes (even around the time of the US elections). Wikimedia Daily Pageviews from Mexico In Mexico, there seems to been a huge drop starting October 27th (perhaps some sort of local outage). Wikimedia Daily Pageviews from Ecuador Ecuador shows a huge spike on October 30th, followed by a drop for several days. Desktop: 54.1% (previous report: 54.1%) Mobile web: 44.8% (previous report: 44.6%) Apps: 1.1% (previous report: 1.3%) (missing some iOS pageviews, cf. above) Context (December 2015 - December 2016): Overall mobile percentage is similar to the last report, but we did see a small increase (besides one week in August) until late September. As a reminder, mobile already has a solid majority in terms of unique devices, cf. below. Global North ratio: 75.3% of total pageviews (previous report: 75.5%) Context (January - December 2016): There is a slight increase in Global North percentage in late October and November, but overall this number is relatively steady. NB: We are currently rethinking this metric and might replace it with a different country selection constructed as part of the work on the New Readers project. Unique devices See the announcement blog post <https://blog.wikimedia.org/2016/03/30/unique-devices-dataset/> from March for background and details on this recently introduced metric. These estimated numbers are provided for all Wikimedia language projects (separately for the desktop and mobile web version). Because of the instrumentation method, there is no global metric for all projects and all languages, but following some recent discussions it is now planned <https://phabricator.wikimedia.org/T138027> to extend it to a cross-language global metric per project at least. For now, we track the daily numbers of the English Wikipedia in this report. Daily unique devices estimate for English Wikipedia: - Mobile web: 33.9 million (previous report: 32.2 million) - Desktop: 24.1 million (previous report: 22.3 million) - Total: 57.9 million (previous report: 54.5 million) - Average mobile web ratio: 58.7% (previous report: 59.3%) Context (January- December 2016): This metric hasn’t existed long enough yet to get a good sense of what yearly seasonalities may exist, but since the last report time frame daily unique desktop devices show an increasing upward trend until early November, where there is a peak and then an apparent drop (which parallels the peak in pageviews). To understand the peak in daily unique devices, we plotted the ratio of daily pageviews to daily unique devices. There is a small bump in the mobile ratio in early November, but it is too small to fully explain the rise in pageviews around this time. New app installations Android: 19.9k/day (+1.21% from the previous report) Daily installs per device, from Google Play Context (last seven months): Download numbers increased in September and have sustained previous levels so far, possibly helped by the app’s inclusion on the Education category page <https://play.google.com/store/apps/category/EDUCATION> in Google’s Play Store. There was an additional bump in mid September (reason unknown). The number of daily uninstalls decreased slightly. During the timespan of the previous report, installs had markedly decreased (except for a so far unexplained bump during two weeks in July), which combined with the high uninstall numbers had caused the app’s install base to shrink from 15.5 million on March 31 to 15.1 million at the beginning of August. This trend has now been stopped, although the number was still at 15.1 million on November. (Google has since discontinued <https://support.google.com/googleplay/android-developer/answer/7003402#acti…> this “current installs” metric in favor of a new “installs on active devices" metric.) iOS: 6.35k/day (+99.9% from the previous report) Download numbers from App Annie Daily downloads (with the largest countries broken out): [image: Wikipedia iOS app daily downloads by country, Aug 1 ~ Dec 7, 2016 (App Annie).png] The iOS Wikipedia app was promoted on iTunes in September, with huge impact (twice as many downloads than resulted from a comparable promotion in September 2015). Total daily downloads remained on a higher level afterwards (4.8k/day during the last week of this report, Nov 28-Dec 4). Around October 24, downloads from the US spiked, possibly helped by some organic media coverage such as this <http://thetechbeard.com/best-free-ios-10-widgets/>. On November 11, we registered an extra 9000 downloads from Sweden, for unknown reasons (no app store promotions or media coverage that we are aware of). App user retention Android: 16.4% (previous report: 16.7%) (Ratio of app installs opened again 7 days after installation, among all installed one week before a date that falls within this report. 1:100 sample) Context (last seven months): As remarked in earlier reports, this data is a bit too noisy for drawing conclusions about whether retention changed significantly between different releases. Retention appears to have dropped somewhat in September after having gradually grown earlier during the timespan of the last report and after. iOS: N/A <https://phabricator.wikimedia.org/T126693> (We recently began measuring this on the iOS app - using the same method as on Android, restricted to users who opt-in to data collection - but there is not enough data yet.) Unique app users Android: 1.146 million / day (+0.0% from the previous report) Context (last six months): In September/October, there seems to be a very small growth in daily average users. iOS: N/A <https://phabricator.wikimedia.org/T130432> Zareen Farooqui, Data Analyst Intern, Wikimedia Foundation Tilman Bayer, Senior Analyst, Wikimedia Foundation Data sources For reference, the queries and source links used are listed below (access is needed for each). Unless otherwise noted, all content of this report is © Wikimedia Foundation and released under the CC BY-SA 3.0 <https://creativecommons.org/licenses/by-sa/3.0/> license. Most of the above charts are available on Commons, too. <https://commons.wikimedia.org/wiki/Category:Wikimedia_readership_metrics_re…> SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, sum(IF(access_method <> 'desktop', view_count, null)) AS mobileviews, SUM(view_count) AS allviews FROM wmf.projectview_hourly WHERE year>0 AND agent_type = 'user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT LEFT(timestamp, 10) AS date, sum(IF(access_method <> 'desktop', pageviews, null)) AS mobileviews, SUM(pageviews) AS allviews FROM staging.pageviews05 WHERE is_spider = FALSE AND is_automata = FALSE GROUP BY date; Pivot URLs: United States <https://pivot.wikimedia.org/#pageviews-daily/line-chart/2/EQUQLgxg9AqgKgYWA…> Mexico <https://pivot.wikimedia.org/#pageviews-daily/line-chart/2/EQUQLgxg9AqgKgYWA…> Ecuador <https://pivot.wikimedia.org/#pageviews-daily/line-chart/2/EQUQLgxg9AqgKgYWA…> SELECT access_method, SUM(view_count)/(7*18) FROM wmf.projectview_hourly WHERE agent_type = 'user' AND CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) BETWEEN "2016-08-01" AND "2016-12-04" GROUP BY access_method; SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")), SUM(view_count) AS all, SUM(IF (FIND_IN_SET(country_code, 'AD,AL,AT,AX,BA,BE,BG,CH,CY,CZ,DE,DK,EE,ES,FI,FO,FR,FX,GB, GG,GI,GL,GR,HR,HU,IE,IL,IM,IS,IT,JE,LI,LU,LV,MC,MD,ME,MK,MT, NL,NO,PL,PT,RO,RS,RU,SE,SI,SJ,SK,SM,TR,VA,AU,CA,HK,MO,NZ,JP,SG,KR,TW,US') > 0, view_count, 0)) AS Global_North_views FROM wmf.projectview_hourly WHERE year > 0 AND agent_type='user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, SUM(IF(uri_host LIKE 'en.m.wikipedia%', uniques_estimate, 0)) AS enwiki_mobile_web FROM wmf.last_access_uniques_daily WHERE year=2016 GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, SUM(IF(uri_host LIKE 'en.wikipedia%', uniques_estimate, 0)) AS enwiki_desktop FROM wmf.last_access_uniques_daily WHERE year=2016 GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) AS date, SUM(IF(access_method = 'mobile web', view_count, null)) AS mobilewebviews, SUM(IF(access_method = 'desktop', view_count, null)) AS desktopviews FROM wmf.projectview_hourly WHERE year=2016 AND agent_type = 'user' AND project = 'en.wikipedia' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT LEFT(timestamp, 8) AS date, SUM(IF(event_appInstallAgeDays = 0, 1, 0)) AS day0_active, SUM(IF(event_appInstallAgeDays = 7, 1, 0)) AS day7_active FROM log.MobileWikiAppDailyStats_12637385 WHERE timestamp LIKE '201510%' AND userAgent LIKE '%-r-%' AND userAgent NOT LIKE '%Googlebot%' GROUP BY date ORDER BY DATE; (with the retention rate calculated as day7_active divided by day0_active from seven days earlier, of course) https://analytics.itunes.apple.com/#/retention?app=324715238 SELECT CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, unique_count AS Android_DAU FROM wmf.mobile_apps_uniques_daily WHERE platform = 'Android'; https://console.developers.google.com/storage/browser/pubsite_prod_rev_ 02812522755211381933/stats/installs/ (“overview”) https://www.appannie.com/dashboard/252257/item/324715238/downloads/ (select “Total”) Zareen Farooqui

7 years, 4 months

Discovery Weekly Update for the week starting 2016-12-05

by Chris Koerner

This is the weekly update for the week starting 05 December. As always, feedback and questions are welcome. ==Highlights== * Load tests for cross-project searching were completed successfully. [0] We are now testing how many results are produced for typical queries as a first step towards figuring out how relevant the results are. [1] * Released tabular and map data namespaces to Commons, after reviewing implementation, consulting legal, enabling shared geojson data storage, and adding metrics tracking. [2] [3] [4] [5] [6] [7] [8] [9] === Search === * We've put together a draft proposal for how to deal with the interaction of all the possible additional search options—suggestions, quote-stripping, "wrong keyboard", language ID, etc.—to have a more co-ordinated conversation about all this. Comments on the talk page there are welcome. [10] * Load tests for cross-project searching were completed successfully. [11] We are now testing how many results are produced for typical queries as a first step towards figuring out how relevant the results are. [12] * Upgraded to Java 8 in preparation for upgrading ElasticSearch next quarter. [13] * Performed some refactoring to improve interwiki searching. [14] * Improved handling of characters containing diacritics or accents. [15] [16] * Added support for minimum query length to detect the language of the query. When a minimum length is eventually chosen, this will solve edge cases where some very short queries and queries consisting just of numbers could produce unexpected results. [17] * Fixed an edge case where third-party users of CirrusSearch could get an exception when building a completion suggester index. [18] === Analysis === * Fixed an error with installing R package Boom (& bsts) on stat1002 (but can on stat1003) [19] === Portal === * Add node 6 compatibility to Portal build tasks [20] === Interactive === * Enhance Report Updater to be able to send data to graphite [21] * Increase user sampling rate [22] * Investigate ClearTables [23] * Bug fixed: Closing maps by going back doesn't always work [24] * Updated and adapted Tabular Data to match published formats [25] * Updated Legal wording update for tabular and map data release [26] * Added information on how to change error message [27] * Updated the logging for Kartotherian from info to warn [28] == Other Noteworthy Stuff == * Published Wikimedia Discovery's narrative and roadmap for FY 2016/17 (July 2016 - June 2017) [29] [0] https://phabricator.wikimedia.org/T149740 [1] https://phabricator.wikimedia.org/T151935 [2] https://phabricator.wikimedia.org/T148745 [3] https://www.mediawiki.org/wiki/Help:Tabular_Data [4] https://www.mediawiki.org/wiki/Help:Map_Data [5] https://www.mediawiki.org/wiki/Extension:JsonConfig/Tabular [6] https://phabricator.wikimedia.org/T134426 [7] https://phabricator.wikimedia.org/T152553 [8] https://phabricator.wikimedia.org/T137930 [9] https://phabricator.wikimedia.org/T152661 [10] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/So_Many_Search_Optio… [11] https://phabricator.wikimedia.org/T149740 [12] https://phabricator.wikimedia.org/T151935 [13] https://phabricator.wikimedia.org/T151325 [14] https://phabricator.wikimedia.org/T141033 [15] https://phabricator.wikimedia.org/T137830 [16] https://phabricator.wikimedia.org/T146402 [17] https://phabricator.wikimedia.org/T149318 [18] https://phabricator.wikimedia.org/T150799 [19] https://phabricator.wikimedia.org/T147682: [20] https://phabricator.wikimedia.org/T150190 [21] https://phabricator.wikimedia.org/T150187 [22] https://phabricator.wikimedia.org/T152174 [23] https://phabricator.wikimedia.org/T141964 [24] https://phabricator.wikimedia.org/T151915 [25] https://phabricator.wikimedia.org/T152184 [26] https://phabricator.wikimedia.org/T152553 [27] https://phabricator.wikimedia.org/T152554 [28] https://phabricator.wikimedia.org/T148116 [29] https://www.mediawiki.org/wiki/File:Discovery_narrative_FY_2016-17.pdf ---- The full update, and archive of past updates, can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as Easy or Volunteer needed in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

7 years, 4 months

ArchCom RFC Meeting W51: Deprecation policy for PHP code in MediaWiki (2016-12-14, #wikimedia-office)

by Daniel Kinzler

Hi all! At the ArchCom RFC meeting on Wednesday [E412], we will talk about defining an official deprecation policy [T146965][DP]. I think it would be quite useful to discuss the different modes and methods of deprecation we have, and what they mean for MediaWiki core, and for extension developers. To me, one of the interesting questions is what happens if a deprecated function is still used in an extension that is not actively maintained. We can provide a patch - but who can review it? And if nobody reviews it, can we still remove the deprecated method? The meeting will be at the usual time (Wednesday 21 UTC, 14 PDT, 23 CET) and place (#wikimedia-office). For an overview of ArchCom activity, see [ArchComStatus]. -- Daniel [E412]: <https://phabricator.wikimedia.org/E412> [T146965]: <https://phabricator.wikimedia.org/T146965> [DP]: <https://www.mediawiki.org/wiki/Requests_for_comment/Deprecation_policy> [ArchComStatus]: <https://www.mediawiki.org/wiki/ArchComStatus> -- Daniel Kinzler Senior Software Developer Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.

7 years, 4 months

Deploying the Linter extension to Wikimedia wikis

by Legoktm

Hi, Arlo and myself have been working on a new MediaWiki extension to expose Parsoid's "lint errors" to users. A little bit of background, Parsoid has a linter that identifies some issues in wikitext that while may not result in user-facing errors, are still not wanted in wikitext. An example might be [[File:Example.png|foo|bar|baz]]. In this case, "foo" and "bar" are ignored, and "baz" is the actual caption - but the bogus image options should be removed. Other errors include missing end tags, obsolete HTML tags, fostered content, etc. The main advantage of this over tracking categories is that we know the location in the wikitext so it should be easier to identify the error and fix it, as well as knowing whether the issue was caused via a template or not. The main ticket tracking deployment is <https://phabricator.wikimedia.org/T148609>. -- Legoktm

7 years, 4 months

Introduction for Gsoc 2017

by MAYANK JINDAL

Hi, My name is Mayank Jindal. I am third year undergraduate student currently studying at Indian Institute of Technology, Kharagpur. I want to take part in Gsoc-2017 from Wikimedia. I have knowledge of C, C++, JAVA, Python, Android app development and Web development and beginner in *Machine Learning, Artificial Intelligence.* I am very enthusiastic to learn new skills which would be required. Kindly guide me to proceed further. -- Kind Regards, Mayank Jindal, Third year undergraduate student, Indian Institute of Technology Kharagpur Mobile : +91- 7076670299 || 8875432718

7 years, 4 months

Cancellation of January 2017 CREDIT showcase

by Adam Baso

Hi all - just a quick note we'll cancel the January 2017 CREDIT showcase <https://www.mediawiki.org/wiki/CREDIT_showcase> session. January is practically speaking really hard for ensuring that it works for most schedules. Meanwhile, be on the lookout for a survey for reflection on 2016's CREDIT. Thanks! -Adam

7 years, 4 months

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l December 2016