Wikitech-l January 2017

wikitech-l@lists.wikimedia.org

85 participants
79 discussions

More useful Puppet docs for MediaWiki-Vagrant
by Bryan Davis 16 Jan '17

16 Jan '17

Hashar did some magic and replaced our use of `puppet doc` with yard which is now generating prettier and more useful documentation of the roles and other Puppet components in the MediaWiki-Vagrant project. Go check them out if you are interested: <https://doc.wikimedia.org/mediawiki-vagrant/>. Try searching for "role::" to get the list of all roles. Bryan -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855

3 4

Discovery Weekly Update for the week starting 2017-01-02
by Chris Koerner 10 Jan '17

10 Jan '17

Hello! Here is another weekly update on the work of the Discovery department. Appologies for the late summary. If you have questions, please ask. Due to the Developer Summit and WMF All-hands event this coming week, the next weekly summary will be the week of 16 January. == Highlights == * Secondary search results are now possible over the API! This is currently used to suggest pages from other language projects that might be relevant for users. Please review the examples for more information. [0] [1] * Added cool new to the Metrics dashboard that highlight the selected date range [2] * Added a full geographic breakdown to the Portal dashboard to see traffic and clickthrough rates on Wikipedia.org portal across all countries [3] * Added final results of the successful long term testing of the addition of app and app page links to wikipedia.org portal page [4] * OSM now allows users who have a Wikimedia account to log into OSM and edit the map data, also new guide added. [5] * Extensions now can define search keywords using CirrusSearchAddQueryFeatures hook. [6] == Search == * Secondary search results are now possible over the API! This is currently used to suggest pages from other language projects that might be relevant for users. Please review the examples for more information. [0] [1] * Fixed a typo in the search preferences page [7] * Added support for extensions to register keywords with the search system [8] * Migrated some keywords to Extension:GeoData to test keyword registration support Analysis [9] [10] * Fixed a data bug where there was a spike in user engagement [11] == Portal == * Updated the stats on Wikipedia.org to reflect that a couple wiki's that have recently achieved big goals in stats: Maithili Wikipedia is now over 10,000 articles and Hebrew Wikipedia now has over 200,000 articles. [12] * Closed out a long term test for adding app and app page links to the wikipedia.org portal page [13] == Interactive == * Released mapframe to French and Finnish Wikipedias [14] [15] * Updated attribution for static and dynamic maps [16] [17] [0] https://phabricator.wikimedia.org/T142795#2923977 [1] https://phabricator.wikimedia.org/T142795 [2] http://discovery.wmflabs.org/metrics/ [3] http://discovery.wmflabs.org/portal/#all_country [4] https://phabricator.wikimedia.org/T146807 [5] https://www.mediawiki.org/wiki/Help:Extension:Kartographer/OSM%7Chere [6] https://www.mediawiki.org/wiki/Manual:Hooks/CirrusSearchAddQueryFeatures [7] https://phabricator.wikimedia.org/T154532 [8] https://phabricator.wikimedia.org/T152517 [9] https://www.mediawiki.org/wiki/Extension:GeoData [10] https://phabricator.wikimedia.org/T152730 [11] https://phabricator.wikimedia.org/T153887 [12] https://phabricator.wikimedia.org/T128546 [13] https://phabricator.wikimedia.org/T137495 [14] https://phabricator.wikimedia.org/T154524 [15] https://phabricator.wikimedia.org/T151591 [16 https://phabricator.wikimedia.org/T151906 [17] https://phabricator.wikimedia.org/T151900 ---- The full update, and archive of all past updates, can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

New version of wikilabels
by Amir Tafreshi 08 Jan '17

08 Jan '17

Hello, Wikilabels [1] is the system to label edits for ORES. Until now, users would have to visit a page in Wikipedia, for example WP:Labels [2] and install a gadget and then label edits for ORES. With the new version (0.4.0) deployed today, you can directly go to Wikilabels home page, for example https://labels.wmflabs.org/ui/enwiki and label edits from there. If you installed the gadget, you can remove it now. We also provided some sort of minification and bundling to improve its performance. Labeling edits would help ORES work more accurately and in case ORES review tool is not enabled in your wiki, you can provide these data for us using wikilabels so can enable it for your wiki as well! [1] https://meta.wikimedia.org/wiki/Wiki_labels [2] https://en.wikipedia.org/wiki/Wikipedia:Labels Best -- Amir Sarabadani Tafreshi Software Engineer (contractor) ------------------------------------- Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin http://wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

2 3

I8n
by Megadev 08 Jan '17

08 Jan '17

Question, is there a gerrit repo that has "global" localisation/lang l8n files im interested in providing localisation and gammar/spelling fixes for English for many extentsions and such for MW. Thanks, Zppix Userpage: www.enwp.org/User:Zppix

5 5

Secondary search results (e.g. language detection) now available over API
by Dan Garry 07 Jan '17

07 Jan '17

Hello! *tl;dr: add srenablerewrites=yes to your API search queries to enable search results from different language projects* The Search Team <https://www.mediawiki.org/wiki/Wikimedia_Discovery#Search:_Backend> is thrilled to announce that secondary search results are now available over the API. This means that automated language detection (provided by TextCat <https://www.mediawiki.org/wiki/TextCat>) and query forwarding can now be used by API consumers. Here's the explanation. The Search Team's analysis of common search queries <https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Survey_of_Zero-Resul…> showed that there are quite a few search queries that aren't in the language of the wiki the user is on. To help alleviate this problem, and give users useful results, we added language detection and query forwarding; for example, Луковичная глава <https://en.wikipedia.org/w/index.php?title=Special:Search&profile=default&s…> now gives the user results from the Russian Wikipedia. This is the functionality that's now available over the API, as you can see if you perform the same search over the API <https://en.wikipedia.org/w/api.php?action=query&list=search&srsearch=%D0%9B…> with the srenablerewrites parameter enabled. The secondary results functionality was added to MediaWiki core <https://gerrit.wikimedia.org/r/#/c/324652/> and is extendable so that, in the future, if we (or someone else!) provide secondary results from other sources, then this functionality can be used for that. For backwards compatibility, don't add the srenablerewrites parameter and you'll continue getting the same results in the same format as before this change. Happy querying! Thanks, Dan -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation

2 1

Re: [Wikitech-l] Help me to fix a bug in my extension
by Victor Porton 06 Jan '17

06 Jan '17

On Sat, 2017-01-07 at 01:06 +0200, Victor Porton wrote: > Please help me to get my updates into Gerrit's master. > > I am confused how this works: > > https://gerrit.wikimedia.org/r/331086 > https://gerrit.wikimedia.org/r/331088 > > I will make other updates later.

1 0

Help me to fix a bug in my extension
by Victor Porton 06 Jan '17

06 Jan '17

My extension does add additional links (which it is created to render in addition to the normal [[...]] links) to the pagelinks table for the edited page every even edit and erroneously removes them back every odd edit. Please help to debug this silly behavior: https://gerrit.wikimedia.org/r/#/c/330816/1 is the patch which half- works. My extension: https://www.mediawiki.org/wiki/Extension:NamespacePopups It uses my another extension: https://www.mediawiki.org/wiki/Extension:PagePopups This is very important for me. Well, maybe some day we'll use these extensions in Wikipedia, who knows.

2 3

Readership metrics for the timespan until January 1, 2017
by Zareen Farooqui 06 Jan '17

06 Jan '17

Hi all, Here is the usual look <https://www.mediawiki.org/wiki/Reading/Readership_metrics_reports> at our most important readership metrics. This time we see a seasonal decline in desktop engagement and rise in mobile engagement over the winter holidays. As laid out earlier <https://lists.wikimedia.org/pipermail/mobile-l/2015-September/009773.html>, the main purpose is to raise awareness about how these are developing, call out the impact of any unusual events, and facilitate thinking about core metrics in general. As always; feedback and discussion welcome. Week-over-week and month-over-month changes are now being recorded on the Product page <https://www.mediawiki.org/wiki/Wikimedia_Product#Reading> at MediaWiki.org. This edition of the report covers a timespan of four weeks. The chart below looks at long-term traffic trends from May 2013 - December 2016. This shows that over the last three and a half years, the annual change in overall pageviews is down 3%, desktop is down 16%, and mobile (web + apps) is trending upwards at a rate of 23%. However, the past few months have seen total pageviews increasing year-over-year (chart further below). Now to the usual data. (All numbers below are averages for December 5, 2016 - January 1, 2017 unless otherwise noted.) Pageviews Total: 528 million/day (-0.2% from the previous report) Context (April 2015-January 2017): See also the Vital Signs dashboard <https://analytics.wikimedia.org/dashboards/vital-signs/#projects=all/metric…> Similar to last year <https://upload.wikimedia.org/wikipedia/commons/c/c6/Readership_metrics_for_…>, overall daily pageviews declined for the 3 weeks leading up to Christmas. However, mobile pageviews spike around Christmas and New Years. To facilitate our understanding of which traffic movements are seasonal and which may indicate lasting changes, here is a chart overlaying the total pageview numbers back to May 2013 (the earliest time for which we have data according to the current pageview definition): The blue lines shows a seasonal dip in pageviews around Christmas time. Since then, total pageviews have been rising, as expected. The year-over-year increase in total pageviews has continued (December 2016 is up 5.9% from December 2015). Desktop: 51.5% (previous report: 54.1%) Mobile web: 46.8% (previous report: 44.8%) Apps: 1.7% (previous report: 1.1%) Since the last report timeframe, overall mobile percentage rose, mainly due to the large spike around Christmas (58% mobile pageviews) and New Years (59% mobile pageviews). This resembles the Christmas bump from 2015, however at a level that is about 3-4% higher now. While December 20, 2015 was the first ever day where the majority of pageviews were mobile, the week from December 26, 2016 to January 1, 2017 marks the first full week where this was the case (53.6% mobile wep + app)! Presumably though this will drop again below parity in January. Also, a caveat that part of this increase was caused by the iOS Wikipedia app’s pageviews almost quadrupling from December 20 on, which might <https://phabricator.wikimedia.org/T154735> be an anomaly. However, this does not have a large effect overall, as it increases mobile % by only about 1.3%. Global North ratio: 76.8% of total pageviews (previous report: 75.3%) Context (January 2016-January 2017): There is a noticeable decrease in Global North pageviews percentage for a couple days around December 21, but a growth to almost 80% immediately follows. NB: We are currently rethinking this metric and might replace it with a different country selection constructed as part of the work on the New Readers project. Unique devices See the announcement blog post <https://blog.wikimedia.org/2016/03/30/unique-devices-dataset/> from March 2016 for background and details on this metric. These estimated numbers are provided for all Wikimedia language projects (separately for the desktop and mobile web version). Because of the instrumentation method, there is no global metric for all projects and all languages, but following some recent discussions it is now planned <https://phabricator.wikimedia.org/T138027> to extend it to a cross-language global metric per project at least. For now, we track the daily numbers of the English Wikipedia in this report. Daily unique devices estimate for English Wikipedia: - Mobile web: 33.6 million (previous report: 33.9 million) - Desktop: 21.6 million (previous report: 24.1 million) - Total: 55.2 million (previous report: 57.9 million) - Average mobile web ratio: 61.2% (previous report: 58.7%) Context (January 2016- January 2017): This metric hasn’t existed long enough yet to get a good sense of what yearly seasonalities may exist, but since the last report time frame daily unique desktop devices continued their decline until a small rise for a couple days following Christmas. Daily unique mobile devices also followed this pattern, but the decline leading up to Christmas was less pronounced. Interestingly, even though the number of unique desktop devices was declining in early December, the number of daily views per desktop device was rising. The small increase in mobile pageviews mentioned in the last report has continued. New app installations Android: 19.1k/day (-3.9% from the previous report) Daily installs per device, from Google Play Context (last eight months): At the beginning of December, the previously discussed increase in installs stopped for an unknown reason. However, there is a large spike of 28,452 installs on December 25. We continue to see above average install numbers for a few days, until December 31st. The number of uninstalls have been dropping slowly for the last few weeks in December. iOS: 5.33 k/day (-0.2% from the previous report) Download numbers from App Annie Context (last two months): [image: Wikipedia iOS app daily downloads by country, Nov 1, 2016 - Jan 1, 2017 (App Annie).png] Downloads of the iOS app saw a considerable increase around December 25, remaining at higher levels for the following week - clearly the result of Apple providing a lot of christmas presents again this year. There was also a spike on December 18 in downloads from the Netherlands, which because of its form and the fact that the increase was by a suspiciously round number (from around 50 to 10,050 downloads per day) is likely to have been an anomaly. App user retention Android: 16.8% (previous report: 16.4%) (Ratio of app installs opened again 7 days after installation, among all installed one week before a date that falls within this report. 1:100 sample) Context (last seven months): As remarked in earlier reports, this data is a bit too noisy for drawing conclusions about whether retention changed significantly between different releases. But we can at least rule out the existence of major shifts during this timespan. iOS: N/A <https://phabricator.wikimedia.org/T126693> Unique app users Android: 1.179 million / day (+0.3% from the previous report) Context (last seven months): Android daily active users remain steady until a spike around Christmas and New Year which matches the spike in unique mobile devices and mobile pageviews. iOS: N/A <https://phabricator.wikimedia.org/T130432> Zareen Farooqui, Data Analyst Intern, Wikimedia Foundation Tilman Bayer, Senior Analyst, Wikimedia Foundation Data sources For reference, the queries and source links used are listed below (access is needed for each). Unless otherwise noted, all content of this report is © Wikimedia Foundation and released under the CC BY-SA 3.0 <https://creativecommons.org/licenses/by-sa/3.0/> license. Most of the above charts are available on Commons, too. <https://commons.wikimedia.org/wiki/Category:Wikimedia_readership_metrics_re…> SELECT year, month, CONCAT(year,"-",LPAD(month,2,"0")) as yearmonth, SUM(view_count) AS allhuman, sum(IF(access_method = 'desktop', view_count, null)) AS desktophuman FROM wmf.projectview_hourly WHERE year = 2016 AND month > 10 AND agent_type = 'user' GROUP BY year, month ORDER BY year, month LIMIT 1000; SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, sum(IF(access_method <> 'desktop', view_count, null)) AS mobileviews, SUM(view_count) AS allviews FROM wmf.projectview_hourly WHERE year>0 AND agent_type = 'user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT access_method, SUM(view_count)/(7*4) FROM wmf.projectview_hourly WHERE agent_type = 'user' AND CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) BETWEEN "2016-12-05" AND "2017-01-01" GROUP BY access_method; SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")), SUM(view_count) AS all, SUM(IF (FIND_IN_SET(country_code, 'AD,AL,AT,AX,BA,BE,BG,CH,CY,CZ,DE,DK,EE,ES,FI,FO,FR,FX,GB,GG,GI,GL,GR,HR,HU,IE,IL,IM,I S,IT,JE,LI,LU,LV,MC,MD,ME,MK,MT,NL,NO,PL,PT,RO,RS,RU,SE,SI,SJ,SK,SM,TR,VA,AU,CA,HK,MO, NZ,JP,SG,KR,TW,US') > 0, view_count, 0)) AS Global_North_views FROM wmf.projectview_hourly WHERE year > 0 AND agent_type='user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, SUM(IF(uri_host LIKE 'en.m.wikipedia%', uniques_estimate, 0)) AS enwiki_mobile_web FROM wmf.last_access_uniques_daily WHERE year=2016 OR year=2017 GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, SUM(IF(uri_host LIKE 'en.wikipedia%', uniques_estimate, 0)) AS enwiki_desktop FROM wmf.last_access_uniques_daily WHERE year=2016 OR year=2017 GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) AS date, SUM(IF(access_method = 'mobile web', view_count, null)) AS mobilewebviews, SUM(IF(access_method = 'desktop', view_count, null)) AS desktopviews FROM wmf.projectview_hourly WHERE year=2016 OR year=2017 AND agent_type = 'user' AND project = 'en.wikipedia' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; SELECT LEFT(timestamp, 8) AS date, SUM(IF(event_appInstallAgeDays = 0, 1, 0)) AS day0_active, SUM(IF(event_appInstallAgeDays = 7, 1, 0)) AS day7_active FROM log.MobileWikiAppDailyStats_12637385 WHERE userAgent LIKE '%-r-%' AND userAgent NOT LIKE '%Googlebot%' GROUP BY date ORDER BY DATE; (with the retention rate calculated as day7_active divided by day0_active from seven days earlier, of course) https://console.developers.google.com/storage/browser/pubsite_prod_rev_0281… (“overview”) https://www.appannie.com/dashboard/252257/item/324715238/downloads/ (select “Total”) https://analytics.itunes.apple.com/#/retention?app=324715238 SELECT CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, unique_count AS Android_DAU FROM wmf.mobile_apps_uniques_daily WHERE platform = 'Android'; Zareen Farooqui

1 0

Your questions and votes for our CTO and VP of Product Q&A
by Quim Gil 06 Jan '17

06 Jan '17

Hi, On the second day of the Wikimedia Developer Summit (January 10) there will be a Q&A session with Victoria Coleman (Wikimedia Foundation CTO) and Wes Moran (VP of Product). It is a plenary session and it will be video-streamed. The questions for this session are being crowdsourced at http://www.allourideas.org/wikidev17-product-technology-questions. Anyone can propose questions and vote, anonymously, as many times as you want. At the moment, we have 25 questions and 451 votes. An important technical detail: questions posted later have also good chances to make it to the top of the list as long as new voters select them. The ranking is made out of comparisons between questions, not accumulation of votes. For instance, the current top question is in fact one of the last that has been submitted so far. Why posting or voting a good question? One obvious reason is to encourage the Foundation's Technology and Product top managers to bring a good answer in a public session with minutes taken and video recording. :) Beyond that, if the ranking of questions makes sense and is backed by participation numbers, it has a serious chance to influence plans and discussions beyond the Summit. The current ranking does make sense, but maybe you could help covering more areas, other perspectives? 1. How do we deal with the lack of maintainers for all Wikimedia deployed code? 2. Do we have a plan to bring our developer documentation to the level of a top Internet website, a major free software project? 3. For WMF dev teams, what is the right balance between pushing own work versus seeking and supporting volunteer contributors? 4. During the next year or so, what balance do you think we should strike between new projects and technical debt? 5. When are we going to work on a modern talk pages system for good? 6. Whose responsibility is to assure that all MediaWiki core components and the extensions deployed in Wikimedia have active maintainers? 7. How important is to have a well maintained and well promoted catalog of tools, apps, gadgets, bots, templates, extensions...? 8. Will MediaWiki ever become easier to install and manage? (e.g. plugin manager à la Wordpress). How much do we care about enterprise users? 9. What should be the role of the Architecture Committee in WMF planning (priorities, goals, resources...) and are we there yet? 10. In addition to Community Tech, should the other WMF Product teams prioritize their work taking into account the Community Wishlist results? The full list: http://www.allourideas.org/wikidev17-product-technology-questions/results -- Quim Gil Engineering Community Manager @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil

2 3

Localizable cross-wiki templates
by Yuri Astrakhan 06 Jan '17

06 Jan '17

TLDR: Templates can now use datasets on Commons for all their localizable messages, making it possible to copy/paste templates between wiki without any changes. If you look at the Graph:Lines template [1], there is a small link under each graph that points to the data source. That link is automatically localized on enwiki and ruwiki because the actual messages are stored in a Commons' Dataset, while the template itself is identical on both wikis. The magic happens in the Module:TNT (hoping it won't blow up): *{{#invoke:TNT | msg | DatasetDict.tab | message-key | param1 | param2 | ... }}* I would love to support template doc pages too, but I'm not sure how well VE and other tools will work with the <templatedata> This is an experimental thing, just hacking on it in my free time, and would love to get your feedback on the viability of this approach. Thanks! Links: enwiki: https://en.wikipedia.org/wiki/Template:Graph:Lines ruwiki: https://ru.wikipedia.org/wiki/Template:Graph:Lines TNT: https://commons.wikimedia.org/wiki/Module:TNT Dictionary: https://commons.wikimedia.org/wiki/Data:Original/Template:Graph:Lines.tab

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l January 2017