Wikitech-l November 2016

wikitech-l@lists.wikimedia.org

94 participants
104 discussions

productivity of mediawiki developers
by rupert THURNER 30 May '17

30 May '17

hi, is there a statistics about mediawiki developer productivity? i just fell over a couple of pages and i am quite impressed i must say: * gabriel, https://github.com/gwicke, 2'300 commits last year * jeroen, https://github.com/JeroenDeDauw, 3'700 commits a year * ori, https://github.com/atdt, 1'700 * james, https://github.com/jdforrester, 1'200 * yuri, https://github.com/nyurik, 900 * matt: https://github.com/mattflaschen, 400 commits rupert

11 14

Parsoid: node 0.1x deprecated now; node 0.1x support will end March 31st, 2017
by Subramanya Sastry 07 Apr '17

07 Apr '17

The Parsing team at the Wikimedia Foundation that develops the Parsoid service is deprecating support for node 0.1x. Parsoid is the service that powers VisualEditor, Content Translation, and Flow. If you don't run a MediaWiki install that uses VisualEditor, then this announcement does not affect you. Node 0.10 has reached end of life on October 31st, 2016 [1] and node 0.12 is scheduled to reach end of life December 31st, 2016 [1]. Yesterday, we released a 0.6.1 debian package [2] and a 0.6.1 npm version of Parsoid [3]. This will be the last release that will have node 0.1x support. We'll continue to provide any necessary critical bug fixes and security fixes for the 0.6.1 release till March 31st 2017 and will be completely dropping support for all node versions before node v4.x starting April 2017. If you are running a Parsoid service on your wiki and are still using node 0.1x, please upgrade your node version by April 2017. The Wikimedia cluster runs node v4.6 right now and will soon be upgraded to node v6.x [4]. Parsoid has been tested with node 0.1x, node v4.x and node v6.x and works with all these versions. However, we are dropping support for node 0.1x right away from the master branch of Parsoid. Going forward, the Parsoid codebase will adopt ES6 features available in node v4.x and higher which aren't supported in node 0.1x and will constitute a breaking change. Subramanya Sastry (Subbu), Technical Lead and Manager, Parsing Team, Wikimedia Foundation. [1] Node.js Long Term Support schedule @ https://github.com/nodejs/LTS [2] https://www.mediawiki.org/wiki/Parsoid/Releases [3] https://www.npmjs.com/package/parsoid [4] https://phabricator.wikimedia.org/T149331

1 1

Fwd: [Commons-l] Programmatically categorizing media in the Commons with Machine Learning
by Pine W 04 Apr '17

04 Apr '17

Forwarding. Pine ---------- Forwarded message ---------- From: "Jordan Adler" <jmadler(a)google.com> Date: Aug 11, 2016 13:06 Subject: [Commons-l] Programmatically categorizing media in the Commons with Machine Learning To: "commons-l(a)wikimedia.org" <commons-l(a)lists.wikimedia.org> Cc: "Ray Sakai" <rsakai(a)reactive.co.jp>, "Ram Ramanathan" < ramramanathan(a)google.com>, "Kazunori Sato" <kazsato(a)google.com> Hey folks! A few months back a colleague of mine was looking for some unstructured images to analyze as part of a demo for the Google Cloud Vision API <https://cloud.google.com/blog/big-data/2016/05/explore-the-galaxy-of-images…>. Luckily, I knew just the place <https://commons.wikimedia.org/wiki/Category:Media_needing_categories>, and the resulting demo <http://vision-explorer.reactive.ai/>, built by Reactive Inc., is pretty awesome. It was shared on-stage by Jeff Dean during the keynote <https://www.youtube.com/watch?v=HgWHeT_OwHc&feature=youtu.be&t=2h1m19s> at GCP NEXT 2016. I wanted to quickly share the data from the programmatically identified images so it could be used to help categorize the media in the Commons. There's about 80,000 images worth of data: - map.txt <https://storage.googleapis.com/gcs-samples2-explorer/reprocess/map.txt> (5.9MB): A single text file mapping id to filename in a "id : filename" format, one per line - results.tar.gz <https://storage.googleapis.com/gcs-samples2-explorer/reprocess/results.tar.…> (29.6MB): a tgz'd directory of json files representing the output of the API <https://cloud.google.com/vision/reference/rest/v1/images/annotate#response-…>, in the format "${id}.jpg.json" We're making this data available under the CC0 license, and these links will likely be live for at least a few weeks. If you're interested in working with the Cloud Vision API to tag other images in the Commons, talk to the WMF Community Tech team. Thanks for your help! _______________________________________________ Commons-l mailing list Commons-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/commons-l

2 1

Deployments: upcoming holidays and their impact
by Greg Grossmeier 22 Dec '16

22 Dec '16

It's that time of year again; the holiday season. Different from last year: In the interest of both 1) giving engineers full time off at the end of the year (not needing to be aware of what code they wrote is being deployed) and 2) to keep the site reliable during our end-of-year fundraising pushes there will be 2 weeks (instead of just 1) at the end of December that are NO DEPLOYS weeks. Here's the basic outline from now until mid-January (by week): * Nov 7th: normal * Nov 14th: normal * Nov 21st: (Thanksgiving) ** All week: no train ** Mon/Tues: SWATs only ** Wed/Thur/Fri: NO DEPLOYS * Nov 28th: normal * Dec 5th: normal * Dec 12th: normal * Dec 19th: NO DEPLOYS (XMAS) * Dec 26th: NO DEPLOYS (XMAS) * Jan 2nd: normal (with train) * Jan 9th: no train, SWATs only (but no one from RelEng is garaunteed to * be around) (DevSummit+All Hands) * Jan 16th: resume normalcy (Monday is MLK Day) As always, the canonical location for deployment information is at: https://wikitech.wikimedia.org/wiki/Deployments Best, Greg -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | Release Team Manager A18D 1138 8E47 FAC8 1C7D |

4 8

Deploying the Linter extension to Wikimedia wikis
by Legoktm 11 Dec '16

11 Dec '16

Hi, Arlo and myself have been working on a new MediaWiki extension to expose Parsoid's "lint errors" to users. A little bit of background, Parsoid has a linter that identifies some issues in wikitext that while may not result in user-facing errors, are still not wanted in wikitext. An example might be [[File:Example.png|foo|bar|baz]]. In this case, "foo" and "bar" are ignored, and "baz" is the actual caption - but the bogus image options should be removed. Other errors include missing end tags, obsolete HTML tags, fostered content, etc. The main advantage of this over tracking categories is that we know the location in the wikitext so it should be easier to identify the error and fix it, as well as knowing whether the issue was caused via a template or not. The main ticket tracking deployment is <https://phabricator.wikimedia.org/T148609>. -- Legoktm

4 5

Introduction for Gsoc 2017
by MAYANK JINDAL 10 Dec '16

10 Dec '16

Hi, My name is Mayank Jindal. I am third year undergraduate student currently studying at Indian Institute of Technology, Kharagpur. I want to take part in Gsoc-2017 from Wikimedia. I have knowledge of C, C++, JAVA, Python, Android app development and Web development and beginner in *Machine Learning, Artificial Intelligence.* I am very enthusiastic to learn new skills which would be required. Kindly guide me to proceed further. -- Kind Regards, Mayank Jindal, Third year undergraduate student, Indian Institute of Technology Kharagpur Mobile : +91- 7076670299 || 8875432718

4 5

Google Code-in 2016 just started and we need your help!
by Andre Klapper 04 Dec '16

04 Dec '16

Google Code-in 2016 just started: https://www.mediawiki.org/wiki/Google_Code-in_2016 In the next seven weeks, many young people are going to make their first contributions to Wikimedia. Expect many questions on IRC and mailing lists by onboarding newcomers who have never used IRC or lists before. Your help and patience is welcome to provide a helping hand! Thanks to all mentors who have already registered & provided tasks! You have not become a mentor yet? Please do consider it. It is fun and we do need more tasks! :) * Think of easy tasks in your area that you could mentor. Areas are: Code, docs/training, outreach/research, quality assurance, and user interface. "Easy" means 2-3h to complete for you, or less technical ~30min "beginner tasks" for onboarding). * OR: provide an easy 'clonable' task (a task that is generic and could be repeated many times by different students). * Note that you commit to answer to students' questions and to evaluate their work within 36 hours (but the better your task description the less questions. No worries, we're here to help!) For the full info, please check out https://www.mediawiki.org/wiki/Google_Code-in_2016/Mentors and ask if something is unclear! Thank you again for giving young contributors the opportunity to learn about and work on all aspects of Free & Open Source Software projects! Cheers, andre -- Andre Klapper | Wikimedia Bugwrangler http://blogs.gnome.org/aklapper/

3 2

Wikimedia Developer Summit Weekly
by Quim Gil 02 Dec '16

02 Dec '16

Original version: https://www.mediawiki.org/wiki/Topic:Td5wfd70vptn8eu4 The Wikimedia Developer Summit 2017 is 13 weeks away! PARTICIPANTS * 68 people have requested an invitation. * 19 of them have requested travel sponsorship as well. Join us: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit. The deadline to request travel sponsorship is Monday, October 24th. PROPOSALS * 8 proposals submitted - https://phabricator.wikimedia. org/project/view/2205/ ** 1 in backlog ** 2 in Unconference ** 2 missing basic information ** 3 to be pre-scheduled https://www.mediawiki.org/wiki/Wikimedia_Developer_ Summit/Call_for_participation The deadline for submitting new proposals is Monday, October 31. MAIN TOPICS https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Program On track: * A plan for the Community Wishlist 2016 top results * Building on Wikimedia services: APIs and Developer Resources * How to manage our technical debt Missing basic information * A unified vision for editorial collaboration * Building a sustainable user experience together * How to grow our technical community Missing two facilitators * A unified vision for editorial collaboration * How to grow our technical community Missing one facilitator * Handling wiki content beyond plaintext * Building a sustainable user experience together * Artificial Intelligence to build and navigate content ORGANIZATION * Meet the Program committee. https://www.mediawiki.org/ wiki/Wikimedia_Developer_Summit/Program_committee -- Quim Gil Engineering Community Manager @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil

4 12

December 2016 CREDIT Showcase in a week
by Adam Baso 01 Dec '16

01 Dec '16

Hello all - The next CREDIT showcase is a week from today - Wednesday, 7-December-2016 at 1900 UTC (1100 San Francisco). https://www.mediawiki.org/wiki/CREDIT_showcase If you have a demo, please add it here: https://etherpad.wikimedia.org/p/CREDIT Last month there were demos on offline, IFTTT, and File properties search & new WDQS visualizations. https://www.youtube.com/watch?v=NmfqtP3pr2Y&t=47 We welcome demos from Wikimedia community members and staff alike - from polished features to something you just got working last night. Hope to see you soon! If you would like to invite anyone to CREDIT, feel free to use this template. *Hi <FNAME>* *I hope all is well with you! I wanted to let you know about CREDIT, a monthly demo series that we’re running to showcase open source tech projects from Wikimedia’s Community, Reading, Editing, Discovery, Infrastructure and Technology teams. * *CREDIT is open to the public, and we welcome questions and discussion. The next CREDIT will be held on December 7th at 11am PT / 2pm ET / 19:00 UTC. * *There’s more info on MediaWiki <https://www.mediawiki.org/wiki/CREDIT_showcase>, and on Etherpad <https://etherpad.wikimedia.org/p/CREDIT>, which is where we take notes and ask questions. You can also ask questions on IRC in the Freenode chatroom #wikimedia-office (web-based access here <https://webchat.freenode.net/?channels=%23wikimedia-office>). Links to video will become available at these locations shortly before the event.* *Please feel free to pass this information along to any interested folks. Our projects tend to focus on areas that might be of interest to folks working across the open source tech community: language detection, numerical sort, large data visualizations, maps, and all sorts of other things.* *If you have any questions, please let me know! Thanks, and I hope to see you at CREDIT.* *-YOURNAME*

1 0

2016-11-30 Scrum of Scrums meeting notes
by Grace Gellerman 01 Dec '16

01 Dec '16

https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-11-30 = 2016-11-30 = == Product == === Reading === ==== Android native app ==== * Last week: ** Continuing Q2 goals for Wikidata descriptions ** New fundraising announcement explore feed card in progress ** Fixing login and editing issues * Next week (https://phabricator.wikimedia.org/project/view/2352/): ** More Q2 goals for Wikidata descriptions (tutorial and polish) ====Web==== * Current sprint: https://phabricator.wikimedia.org/tag/reading-web-sprint-86-%F0%9F%94%AA%F0… * This week: ** Stopped HoverCards A/B tests from Russian and Italian Wikipedias ** New readers work ** MobileFrontend regression fixes ** Make PageImages return the image in the lead section ** MobileFrontend tech debt ** Trending service ** Hovercards rewrite ====iOS==== * Last Week: ** Found and fixed notification bugs, crashes found in 5.3.0, ready for 5.3.1 release ** Finished announcement card updates, ready for 5.3.1 release ** Finished data layer updates, ready for beta in 5.3.2 ** Continued dynamic font size updates to the app * This week: 5.3.1 https://phabricator.wikimedia.org/project/view/2357/ & 5.3.2 https://phabricator.wikimedia.org/project/view/2281/ ** Ship 5.3.1 (any remaining bug fixes) ** Beta 5.3.2 ==== Mobile Content Service (MCS) ==== * Board: https://phabricator.wikimedia.org/tag/mobile-content-service/ * added Wikibase ID to mobile-sections output * introducing eslint as part of the node template update ==== Reading Infrastructure ==== * working on random MediaWiki tech debt * not blocking * Still looking for reviews on the API error/warning i18n patches (see last week) === Community tech === * Blocked: Nope * Blocking: Nope * Updates: * Continued improvements to Copypatrol ( https://tools.wmflabs.org/copypatrol) * Community wishlist survey is in voting phase: https://meta.wikimedia.org/wiki/2016_Community_Wishlist_Survey == Editing == === Parsing === * Blocked: Tim asked for an ops person to work with him to puppetize HTML5Depurate service; yet to hear back as of y'day * Tim working on https://phabricator.wikimedia.org/T151702 with Ops; * Arlo's work on Parsoid-native <gallery> extension is now merged. Once we update the specs at https://www.mediawiki.org/wiki/Specs/HTML/1.3.0 and chat with Services folks, we'll deploy this. * Kunal continuing to work on the Linter extension (scheduled to be deployed in Q3 2016-17). * Scott busy with Dev Summit Planning this previous (short) week, but trying to finish up patches in Parsoid to parse language variant markup. == Technology == === Technical Operations === * Will not be able to attend, apologies! * '''Blocked''' ** None * '''Blocking''' ** None * Updates ** API outage investigation and actionables ongoing ** Q3 Goals discussion started, please inform ops of dependencies on them ** Goals on track, work ongoing === Analytics === * Auditing old logs on stat1002 for *deletion*, including: mobile-sampled, api-usage, sampled, edits (from webrequest), glam_nara, zero. More at https://phabricator.wikimedia.org/T84618 * Mediawiki history reconstruction project milestone: standard metrics ran against the imported data give very similar results with much simpler queries, test dataset loaded into Druid: https://pivot.wikimedia.org/#edit-history-test * Adding send-to-graphite functionality to Reportupdater * Wikistats 2.0 design starting tomorrow * New hire engineer will start December 12th * Ongoing: ** upgrading stat boxes and other out-of-warranty boxes ** optimizing last step of mediawiki history reconstruction ** public event streams work ** varnishkafka integration testing ** reaching out to analysts and evengelizing reportupdater and pivot === Release Engineering === * '''Blocked''' ** None * '''Blocking''' ** None? * '''Updates''' ** Reminder that the [ https://lists.wikimedia.org/pipermail/wikitech-l/2016-November/086936.html deployments schedule gets weird] between now and the end of the year ** Phabricator replication should be a bit faster thanks to Mukunda ** Scap 3.4 is out -- see the [ https://phabricator.wikimedia.org/diffusion/MSCA/browse/release/debian/chan…] for details ** MediaWiki and extensions 1.28.0 tarball is now available === Security === * Security Reviews * LoginNotify completing from last week * Recommendations API started this week * One item need code review * CentralAuth special pages: https://gerrit.wikimedia.o <https://gerrit.wikimedia.o3rg/r/#/c/319055/>3 <https://gerrit.wikimedia.o3rg/r/#/c/319055/>rg/r/#/c/319055/ <https://gerrit.wikimedia.o3rg/r/#/c/319055/> === Fundraising Tech === * Watching logs, tuning fraud settings and queue jobs * Tweaking internal dashboard * Updated thank you email ** Parser output had some unexpected <br>s, wondering if that changed lately ** https://meta.wikimedia.org/w/api.php?page=Fundraising/Translation/Thank_you… === Discovery === * Load-testing crosswiki searching backend code proceeds, looks fine so far * New analysis of query features and performance published: https://commons.wikimedia.org/wiki/File:Query_Features_and_Search_Performan… * New analysis of BM25 scoring in https://phabricator.wikimedia.org/T147500 - still working on how to best tokenize zh

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l November 2016