Wikitech-l May 2016

wikitech-l@lists.wikimedia.org

113 participants
123 discussions

Last call: on the idea of "last call"
by Rob Lanphier 05 May '16

05 May '16

Hi folks, One thing we've implicitly adopted is "last calls" for ArchCom-RFCs. I filed it as an RFC to get it on our workboard: https://phabricator.wikimedia.org/T120164 However, what that means is that we need to have a last call on the "last call" RFC. Obligatory xkcd reference: https://xkcd.com/244/ Barring objection in Phab, we'll consider T120164 approved then. Rob

1 0

How to click custom html option button via api?
by Shrinivasan T 04 May '16

04 May '16

I am trying to work on https://ta.wikisource.org with python API. This tamil wikisource has custom html option buttons to mention the edit status. See a screenshot here. https://snag.gy/f3aTBn.jpg How to enable these custom option buttons via API? Thanks. -- Regards, T.Shrinivasan My Life with GNU/Linux : http://goinggnu.wordpress.com Free E-Magazine on Free Open Source Software in Tamil : http://kaniyam.com Get Free Tamil Ebooks for Android, iOS, Kindle, Computer : http://FreeTamilEbooks.com

3 5

Automatic image colorization
by Ori Livneh 04 May '16

04 May '16

Colorization <https://en.wikipedia.org/wiki/Film_colorization#Digital_colorization> refers to the process of adding color to black-and-white photographs. This work was historically done by hand. These days, colorization is usually done digitally, with the support of specialized tooling. But it is still quite labor-intensive. A forthcoming paper <http://hi.cs.waseda.ac.jp/~iizuka/projects/colorization/en/> from researchers at Waseda University of Japan have developed a method for automatic image colorization using deep learning neural network. The results are both impressive and easy to reproduce, as the authors have published their code <https://github.com/satoshiiizuka/siggraph2016_colorization> to GitHub with a permissive license. Someone has already taken this code and packaged it as a simple webapp, available at http://colorizr.io/ (NSFW). The webapp lets you upload black-and-white pictures and colorizes them for you. (The site is currently not safe for work because it displays a gallery of recent uploads. We know how that goes.) In this thread, let's discuss how this technology could be integrated with the projects. Should we have a bot that can perform colorization on demand, the way Rotatebot <https://commons.wikimedia.org/wiki/User:Rotatebot> can rotate images?

4 6

ImageMagick reported security vulnerability
by Brion Vibber 04 May '16

04 May '16

There's a reported ImageMagick security vulnerability making the rounds: http://arstechnica.com/security/2016/05/easily-exploited-bug-exposes-huge-n… Many MediaWiki sites are configured to use ImageMagick's 'convert' command to perform image rescaling/thumbnailing, so it's worth double-checking that everything is secure... MediaWiki already performs file type validation checks on uploads, which *should* prevent exploitation of the vulnerability for the standard image types (JPEG, PNG, GIF, SVG etc) -- this is one of the recommended mitigation strategies. But I am not confident enough to pronounce us immune without actually checking! Some folks are also recommending tweaking the ImageMagick policy.xml to disable the most dangerous code modules; see the article linked above for further links. -- brion

1 0

upgrade of irc.wikimedia.org
by Daniel Zahn 03 May '16

03 May '16

Hello, today the server that runs irc.wikimedia.org has been upgraded. Or actually, we have installed a new, second server running on Debian jessie and ported the puppet manifests and the IRC bot to work on that, while the old server still exists. MW appservers are sending RC data to both of them. If you are making a new connection to irc.wikimedia.org you should now be served by the new host, kraz.wikimedia.org but old connections have not been broken. The old server is still running unchanged and is reachable as argon.wikimedia.org (as it was before too). We have also made the change "URLs in the recent changes IRC feed will no longer be rewritten to unencrypted HTTP." which was announced and scheduled for today, May 2nd. No action is required if your bot automatically reconnects, but bot owners should ensure no IP addresses are hardcoded (see T123729 for details.) DNS caches slowly roll over and new connections will use kraz. Existing sessions on argon and clients that hardcoded the argon IP won't be affected yet, but after a grace period we are going to shut down the old server argon. refs: https://phabricator.wikimedia.org/T123729#2216681 https://meta.wikimedia.org/wiki/Tech/News/2016/02 Best regards, Daniel -- Daniel Zahn <dzahn(a)wikimedia.org> Operations Engineer

1 0

Readership metrics for the timespan until April 24, 2016
by Tilman Bayer 03 May '16

03 May '16

Hi all, here is the usual look <https://www.mediawiki.org/wiki/Reading/Readership_metrics_reports> at our most important readership metrics. This time, highlights include the impact of the iOS app’s big release of a revamped version, a Google bug that brought the Android app a windfall of about half a million new users, and a first examination of the new unique devices dataset. As laid out earlier <https://lists.wikimedia.org/pipermail/mobile-l/2015-September/009773.html>, the main purpose is to raise awareness about how these are developing, call out the impact of any unusual events, and facilitate thinking about core metrics in general. As always, feedback and discussion are welcome. Week-over-week and month-over-month changes are now being recorded on the Product page <https://www.mediawiki.org/wiki/Wikimedia_Product#Reading> at MediaWiki.org. This edition of the report covers a timespan of eleven weeks. Some other recent items of interest, in case they didn’t already catch your attention: - At the Foundation’s February metrics meeting, I gave an update on longer-term traffic trends since 2013. (TL;DR: Overall pageviews have been flat to slightly declining, mobile has been steadily rising but recently slowed down, desktop is declining.) See the slide deck here or the chart below (updated with March data): [image: Wikimedia monthly pageviews (worldwide, mobile vs desktop), 2013-2016.png] <https://commons.wikimedia.org/wiki/File:Wikimedia_monthly_pageviews_(worldw…> - The WMF Reading team published its quarterly review presentation <https://commons.wikimedia.org/wiki/File:Reading_Quarterly_Review_Q3_2015-16…> for Q3 2015-16 (January-March), with lots of traffic and usage data. - I recently gave a Tech Talk about “new readership data” <https://commons.wikimedia.org/wiki/File:New_readership_data_(Wikimedia_Foun…> - i.e. some data that, separately from the core metrics discussed here, gives insights on how Wikipedia is read. Now to the usual data. (All numbers below are averages for February 8-April 24, 2016 unless otherwise noted.) Our family of core metrics who are populating this report mourns a deplorable casualty (iOS daily active users are no longer available due to the app’s switch to opt-in data collection, T130432 <https://phabricator.wikimedia.org/T130432>), rejoices about the speedy convalescence of an accident victim thanks to expert therapy administered by the Analytics and iOS teams (iOS pageviews, T131824 <https://phabricator.wikimedia.org/T131824>), sends well-wishes for recovery to two other patients (Android app DAUs and pageviews, T132965 <https://phabricator.wikimedia.org/T132965>), and lastly welcomes a new member: unique devices. Pageviews Total: 538 million/day (+1.5% from the previous report) (caveat: likely undercounting by 1-2 million/day since April due to a bug <https://phabricator.wikimedia.org/T132965> related to the Android app’s gradual switch to RESTBase) Context (April 2015-April 2016): [image: Wikimedia daily pageviews, all vs. mobile (April 2015-2016-04-24).png] Overall, pageviews were up compared to the previous report (which had still covered the slump around Christmas). There was a drop in early/mid March though, which probably can’t be fully explained by a change in the pageview definition <https://meta.wikimedia.org/wiki/Research:Page_view#Change_log> on March 9 that improved the exclusion of bot pageviews, nor does it look seasonal compared to March 2014 and March 2015. See also the Vital Signs dashboard <https://vital-signs.wmflabs.org/#projects=all/metrics=Pageviews>. Desktop: 55.2% (previous report: 54.3%) Mobile web: 43.6% (previous report: 44.4%) Apps: 1.2% (previous report: 1.3%) (caveat: app percentage likely too low due to the aforementioned Android-related bug) Context (April 2015-April 2016): [image: Wikimedia daily pageviews, mobile percentage (April 2015..2016-04-24).png] Mobile pageviews now regularly reach parity on weekends. Global North ratio: 76.5% of total pageviews (previous report: 78.3%) Context (April 2015-April 2016): [image: Percentage of Wikimedia pageviews from the Global North (April 2015..2016-04-24).png] <https://commons.wikimedia.org/wiki/File:Percentage_of_Wikimedia_pageviews_f…> In the previous report, we could already witness the ratio of Global North pageviews falling (or conversely, the Global South ratio rising) from a peak at the beginning of January. Afterwards, it saw a somewhat conspicuous drop in mid-February, went back up a bit but is now back to the levels from a year ago, before the HTTPS-only rollout in June. Unique devices Recently, the Analytics team made a new metric available: Daily and monthly unique devices (see their announcement blog post <https://blog.wikimedia.org/2016/03/30/unique-devices-dataset/> for background and details). These estimated numbers are provided for all Wikimedia projects (separately for the desktop and mobile web version), but because of the instrumentation method, there is no global metric for all projects. For now, we’ll pick the daily numbers of the English Wikipedia for inclusion in this report. As for other parts of this reports, the selection and presentation are always being optimized; feedback is welcome. Daily unique devices estimate for English Wikipedia: - Mobile web: 33.2 million - Desktop: 25.7 million - Total: 58.9 million - Average mobile web ratio: 56.5% Like on many of our projects, mobile(-site using) devices outnumber desktop devices, but there are fewer pageviews per device on mobile. (For comparison, the English Wikipedia’s pageviews were at 55.9% desktop, 42.9% mobile web and 1.2% apps; i.e. slightly less mobile than our sites overall.) Context (January-April 2016): [image: Daily unique devices for English Wikipedia (estimate), mobile web vs. desktop, 2016-01-01..2016-04-24.png] The early/mid March drop observed above for pageviews is visible here, too. New app installations Android: 31.6k/day (-17.8% from the previous report) Daily installs per device, from Google Play Context (last three months, log scale): <https://commons.wikimedia.org/wiki/File:Wikipedia_Android_app_-_Daily_insta…> Around March 11/12, Google accidentally redirected Wikipedia search results directly to the app’s listing in Google Play, instead of offering the user to read the article on the mobile web version of Wikipedia. (This was a general bug affecting other apps/websites too.) While resulting spike of ca. 600k additional installs was partially negated by rising uninstalls, a month later the app’s install base still remained around half a million - ca. 3% - larger than before the incident. Overall though, due to a general decrease, downloads per day are down from the previous report <https://commons.wikimedia.org/w/index.php?title=File%3AReadership_metrics_f…>, which had included the “Best Apps of 2015” promotion that brought ca. 350k additional installs. (See also the summary <https://commons.wikimedia.org/w/index.php?title=File%3AReading_Quarterly_Re…> in our quarterly review and the discussion of retention rates below) iOS: 5.06k/day (+2.5% from the previous report) Download numbers from App Annie Context (last three months): <https://commons.wikimedia.org/wiki/File:Wikipedia_iOS_app_daily_downloads,_…> In March, the launch of the revamped 5.0 version of the app, with accompanying media attention, brought roughly 50k additional installs. There also was a smaller spike in mid-February whose reason we don’t know. App user retention Android: 15.4% (previous report: 16.6%) (Ratio of app installs opened again 7 days after installation, among all installed one week before a date that falls within this report. 1:100 sample.) Context (last three months): [image: Day-7 retention of Wikipedia Android app users (install dates 2015-10-01..2016-04-17).png] <https://commons.wikimedia.org/wiki/File:Day-7_retention_of_Wikipedia_Androi…> As remarked in earlier reports, this data is a bit too noisy for drawing conclusions about whether retention changed significantly between different releases. But we can discern an effect of the aforementioned Google bug that inadvertently sent many surfers from the search results page directly to the app’s store page. While retention dropped a lot on the first day of the incident, it actually rose far above average later before the spike ended. Overall, the retention of users who came in during those three days (March 11-13) was 14.1%, not far below the average of 15.4%. iOS: N/A A new instrumentation has been built to measure retention in the same way as for the Android app, but unfortunately it turned out to be a bit buggy, yielding rosy retention rates above 100% ;) We’ll resume reporting after this is sorted out (T126693 <https://phabricator.wikimedia.org/T126693>). Since improving retention was a central goal of the new 5.0 version (and the new instrumentation would not have been available for comparison in older versions of the app, anyway), the iOS team resorted to Apple’s metric once again in order to assess the effect of the rollout. Disregarding the imperfections that had made us stop relying on it earlier, it indicated that day-7 retention had risen from about 10% to 15% in the new version (see quarterly review <https://commons.wikimedia.org/w/index.php?title=File%3AReading_Quarterly_Re…> ). Unique app users Android: N/A Unfortunately, there is currently no valid data for this metric because of issues having to do with the app’s recent switch to the RESTBase-based Content Service, see T132965 <https://phabricator.wikimedia.org/T132965>. (I might still have included the usual 3-month chart showing the development up to that point, but as I’m writing this, Google Sheets doesn’t let you <https://productforums.google.com/forum/#!msg/docs/yB_xNk6mn8U/QjRDR4WtNAAJ> create timeline charts that include the month of February. True story; this report covers not one but two severe bugs in Google products ;) The linked bug contains a chart with a different timespan though - keep in mind that the decline visible from the beginning April is artificial.) iOS: N/A With the new 5.0 version that launched in March, only a small minority of devices continue to send the data that this metric is based on (T130432 <https://phabricator.wikimedia.org/T130432>), so it is no longer valid as a measure of the apps overall usage volume. Data sources For reference, the queries and source links used are listed below (access is needed for each). Unless otherwise noted, all content of this report is © Wikimedia Foundation and released under the CC BY-SA 3.0 <https://creativecommons.org/licenses/by-sa/3.0/> license. Most of the above charts are available on Commons, too <https://commons.wikimedia.org/w/index.php?title=Special:ListFiles&offset=20…> . hive (wmf)> SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, sum(IF(access_method <> 'desktop', view_count, null)) AS mobileviews, SUM(view_count) AS allviews FROM wmf.projectview_hourly WHERE year>0 AND agent_type = 'user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; hive (wmf)> SELECT access_method, SUM(view_count)/77 FROM wmf.projectview_hourly WHERE agent_type = 'user' AND CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) BETWEEN "2016-02-08" AND "2016-04-24" GROUP BY access_method; hive (wmf)> SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")), SUM(view_count) AS all, SUM(IF (FIND_IN_SET(country_code, 'AD,AL,AT,AX,BA,BE,BG,CH,CY,CZ,DE,DK,EE,ES,FI,FO,FR,FX,GB,GG,GI,GL,GR,HR,HU,IE,IL,IM,IS,IT,JE,LI,LU,LV,MC,MD,ME,MK,MT,NL,NO,PL,PT,RO,RS,RU,SE,SI,SJ,SK,SM,TR,VA,AU,CA,HK,MO,NZ,JP,SG,KR,TW,US') > 0, view_count, 0)) AS Global_North_views FROM wmf.projectview_hourly WHERE year > 0 AND agent_type='user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; https://console.developers.google.com/storage/browser/pubsite_prod_rev_0281… (“overview”) https://www.appannie.com/dashboard/252257/item/324715238/downloads/ (select “Total”) SELECT LEFT(timestamp, 8) AS date, SUM(IF(event_appInstallAgeDays = 0, 1, 0)) AS day0_active, SUM(IF(event_appInstallAgeDays = 7, 1, 0)) AS day7_active FROM log.MobileWikiAppDailyStats_12637385 WHERE timestamp LIKE '2016%' AND userAgent LIKE '%-r-%' AND userAgent NOT LIKE '%Googlebot%' GROUP BY date ORDER BY DATE; (with the retention rate calculated as day7_active divided by day0_active from seven days earlier, of course) https://analytics.itunes.apple.com/#/retention?app=324715238 hive (wmf)> SELECT CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, unique_count AS Android_DAU FROM wmf.mobile_apps_uniques_daily WHERE platform = 'Android'; hive (wmf)> SELECT CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, unique_count AS iOS_DAU FROM wmf.mobile_apps_uniques_daily WHERE platform = 'iOS'; -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB

1 0

How might interactive maps be used on Wikipedia?
by Chris Koerner 02 May '16

02 May '16

Hello, The Maps team at the Wikimedia Foundation is getting closer to make it possible to add interactive maps <https://www.mediawiki.org/wiki/Maps> to Wikipedia. If you've ever used services like Google Maps or Mapquest you may be familiar with interactive maps. We’d like to invite editors to have a conversation on how these maps might be used within articles. We've put together information on how these maps and their style works from a technical perspective <https://www.mediawiki.org/wiki/Maps/Conversation_about_interactive_map_use> – where the data comes from, how maps are styled, how to add an interactive map, and a few example use cases. In particular we would like to focus the discussion around three key questions (open discussion outside these questions is welcome too). * What types of articles would use interactive maps? * How do these articles differ in their requirements? * Are there any classes of articles whose map styling requirement is fundamentally in conflict with other article classes, thus requiring multiple styles? If you are interested, please visit https://www.mediawiki.org/wiki/Maps/Conversation_about_interactive_map_use to learn more and get involved. -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Migration of browsertests* Jenkins jobs to selenium* jobs
by Željko Filipin 02 May '16

02 May '16

Hi, We have a set of Jenkins jobs that run daily and execute Ruby+Selenium tests. Recently, the way we run those jobs reached a point where we had to do some serious refactoring[0]. The old jobs were named browsertests*[1], the new are named selenium*[2]. Changes: #1 The creation and deletion of jobs have been made simpler. Each repository now has only a single job defined in Jenkins. It is a multi configuration job that spawns one or more child job based on a configuration in each repository: `tests/browser/ci.yml`[3]. The main job will then spawn child jobs based on its content. #2 Jobs execute `selenium` Rake target (`bundle exec rake selenium`). It is defined in the Rakefile of each repository and load the Rake task from mediawiki_selenium Ruby gem version 1.7.0[4]. What does it mean for you? At this point, no action in needed. All required changes have already been made. When the selenium* job passed for a repository, I have already deleted the browsertests* legacy one. There is still a few repositories (Flow, MobileFrontend, MultimediaViewer, Wikidata) that need to be moved, but we are working on that. If you have any questions, let me know. Željko -- 0: https://phabricator.wikimedia.org/T128190 1: https://integration.wikimedia.org/ci/view/BrowserTests/view/-Dashboard/ 2: https://integration.wikimedia.org/ci/view/Selenium/ 3: https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#ci.yml 4: https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#Rake

1 0

Weekly update #2
by Amir Ladsgroup 02 May '16

02 May '16

The second weekly update on the Revision Scoring project. *New developments* - ORES has graphite dashboard now [1,2] - Deploying new campaigns and testing Wikilabels got easier [3,4] - Revscoring feature extraction got about 13% faster [5] - We deployed new versions of ORES and Wikilabels [6,7] - Wikidata ScoredRevision gadget had a serious issue, it got fixed [8] *Progress in supporting new languages* - Wikidata damaging and goodfaith models are built and deployed [9,10,11] - Dutch damaging and goodfaith models are built and deployed [12] - We are working on langauge utilities of Tamil [13] *Active Labeling campaigns* - Edit quality (damaging and good faith) - Wikipedias: Arabic, Azerbaijani, German, French, Hebrew, Hungarian, Indonesian, Italian, Japanese, Norweigian, Persian (v2), Polish, Spanish, Ukranian, Urdu, Vietnamese - Edit type - English Wikipedia 1. 1. https://grafana.wikimedia.org/dashboard/db/ores 1. 2. https://phabricator.wikimedia.org/T127594 1. 3. https://phabricator.wikimedia.org/T133557 1. 4. https://phabricator.wikimedia.org/T102336 1. 5. https://github.com/wiki-ai/revscoring/pull/268 1. 6. https://phabricator.wikimedia.org/T134032 1. 7. https://phabricator.wikimedia.org/T134174 2. 8. https://phabricator.wikimedia.org/T133903 1. 9. https://phabricator.wikimedia.org/T130274 1. 10. https://phabricator.wikimedia.org/T130301 1. 11. https://lists.wikimedia.org/pipermail/wikidata/2016-May/008641.html 1. 12. https://phabricator.wikimedia.org/T133563 13. https://phabricator.wikimedia.org/T134105 Sincerely, The Revision Scoring team.

1 0

[GSoC 2016] Community Bonding
by Alangi Derick 02 May '16

02 May '16

Hi everyone I am Alangi Derick and also d3r1ck on the IRC. I was selected in the GSoC 2016 program and was opportuned to work on the project titled "Integration of IFTTT support for Wikidata" and I am very happy to be the first African GSoCer in Wikimedia Foundation. This is indeed a priviledge and I wish to thank everyone that guided me in this movement to attain this level. I specially want to thank my mentors (Stephen Laporte, Marius Hoch, Lydia Pintscher, Benedikt Seidl and Sam Tarling) for helping me before and within this program. Also I won't forget to thank some very key org admins for their great help when I just joined the movement (Quim Gil, Brian Wolf, Andre Klapper), I really thank you all for your help. I will love to keep working with you all so that I can better shape my future in Wikimedia and also hope to see you all in person when the time comes :) and also to all Wikimedia Developers, I shall keep in touch and we all shall make the movement better in the future. Cheers!!!! Regards Alangi Derick Ndimnain

2 1

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l May 2016