Wikitech-l March 2017

wikitech-l@lists.wikimedia.org

79 participants
80 discussions

CREDIT next week, and a survey
by Adam Baso 18 Apr '17

18 Apr '17

Hi everybody! As a reminder the CREDIT Showcase is next week on Wednesday, 1-February-2017 (see https://www.mediawiki.org/wiki/CREDIT_showcase for details). Also, as I mentioned previously we're conducting a survey about CREDIT. We'd appreciate your feedback! Here is a link to the survey (which is hosted on a third-party service), and, for information about privacy and data handling, the survey privacy statement. https://docs.google.com/a/wikimedia.org/forms/d/e/1FAIpQLSedAtyPfcEhT6OVd26… https://wikimediafoundation.org/wiki/CREDIT_Feedback_Survey_Privacy_Stateme… . This email is being sent to several mailing lists in order to reach multiple audiences. As always, please follow the list link at the very bottom of this email in case you want to manage your list subscription options such as digest, unsubscribe, and so on. And, as usual, if you'd like to share the news about the upcoming CREDIT, here's some suggested verbiage. *Hi <FNAME>* *I hope all is well with you! I wanted to let you know about CREDIT, a monthly demo series that we’re running to showcase open source tech projects from Wikimedia’s Community, Reading, Editing, Discovery, Infrastructure and Technology teams. * *CREDIT is open to the public, and we welcome questions and discussion. The next CREDIT will be held on February 1st at 11am PT / 2pm ET / 19:00 UTC. * *There’s more info on MediaWiki <https://www.mediawiki.org/wiki/CREDIT_showcase>, and on Etherpad <https://etherpad.wikimedia.org/p/CREDIT>, which is where we take notes and ask questions. You can also ask questions on IRC in the Freenode chatroom #wikimedia-office (web-based access here <https://webchat.freenode.net/?channels=%23wikimedia-office>). Links to video will become available at these locations shortly before the event.* *Please feel free to pass this information along to any interested folks. Our projects tend to focus on areas that might be of interest to folks working across the open source tech community: language detection, numerical sort, large data visualizations, maps, and all sorts of other things.* *If you have any questions, please let me know! Thanks, and I hope to see you at CREDIT.* *YOURNAME* Thanks! Adam Baso Director of Engineering, Reading Wikimedia Foundation abaso(a)wikimedia.org

1 2

Parsoid: node 0.1x deprecated now; node 0.1x support will end March 31st, 2017
by Subramanya Sastry 07 Apr '17

07 Apr '17

The Parsing team at the Wikimedia Foundation that develops the Parsoid service is deprecating support for node 0.1x. Parsoid is the service that powers VisualEditor, Content Translation, and Flow. If you don't run a MediaWiki install that uses VisualEditor, then this announcement does not affect you. Node 0.10 has reached end of life on October 31st, 2016 [1] and node 0.12 is scheduled to reach end of life December 31st, 2016 [1]. Yesterday, we released a 0.6.1 debian package [2] and a 0.6.1 npm version of Parsoid [3]. This will be the last release that will have node 0.1x support. We'll continue to provide any necessary critical bug fixes and security fixes for the 0.6.1 release till March 31st 2017 and will be completely dropping support for all node versions before node v4.x starting April 2017. If you are running a Parsoid service on your wiki and are still using node 0.1x, please upgrade your node version by April 2017. The Wikimedia cluster runs node v4.6 right now and will soon be upgraded to node v6.x [4]. Parsoid has been tested with node 0.1x, node v4.x and node v6.x and works with all these versions. However, we are dropping support for node 0.1x right away from the master branch of Parsoid. Going forward, the Parsoid codebase will adopt ES6 features available in node v4.x and higher which aren't supported in node 0.1x and will constitute a breaking change. Subramanya Sastry (Subbu), Technical Lead and Manager, Parsing Team, Wikimedia Foundation. [1] Node.js Long Term Support schedule @ https://github.com/nodejs/LTS [2] https://www.mediawiki.org/wiki/Parsoid/Releases [3] https://www.npmjs.com/package/parsoid [4] https://phabricator.wikimedia.org/T149331

1 1

Fwd: [Commons-l] Programmatically categorizing media in the Commons with Machine Learning
by Pine W 03 Apr '17

03 Apr '17

Forwarding. Pine ---------- Forwarded message ---------- From: "Jordan Adler" <jmadler(a)google.com> Date: Aug 11, 2016 13:06 Subject: [Commons-l] Programmatically categorizing media in the Commons with Machine Learning To: "commons-l(a)wikimedia.org" <commons-l(a)lists.wikimedia.org> Cc: "Ray Sakai" <rsakai(a)reactive.co.jp>, "Ram Ramanathan" < ramramanathan(a)google.com>, "Kazunori Sato" <kazsato(a)google.com> Hey folks! A few months back a colleague of mine was looking for some unstructured images to analyze as part of a demo for the Google Cloud Vision API <https://cloud.google.com/blog/big-data/2016/05/explore-the-galaxy-of-images…>. Luckily, I knew just the place <https://commons.wikimedia.org/wiki/Category:Media_needing_categories>, and the resulting demo <http://vision-explorer.reactive.ai/>, built by Reactive Inc., is pretty awesome. It was shared on-stage by Jeff Dean during the keynote <https://www.youtube.com/watch?v=HgWHeT_OwHc&feature=youtu.be&t=2h1m19s> at GCP NEXT 2016. I wanted to quickly share the data from the programmatically identified images so it could be used to help categorize the media in the Commons. There's about 80,000 images worth of data: - map.txt <https://storage.googleapis.com/gcs-samples2-explorer/reprocess/map.txt> (5.9MB): A single text file mapping id to filename in a "id : filename" format, one per line - results.tar.gz <https://storage.googleapis.com/gcs-samples2-explorer/reprocess/results.tar.…> (29.6MB): a tgz'd directory of json files representing the output of the API <https://cloud.google.com/vision/reference/rest/v1/images/annotate#response-…>, in the format "${id}.jpg.json" We're making this data available under the CC0 license, and these links will likely be live for at least a few weeks. If you're interested in working with the Cloud Vision API to tag other images in the Commons, talk to the WMF Community Tech team. Thanks for your help! _______________________________________________ Commons-l mailing list Commons-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/commons-l

2 1

Discovery Weekly Update for the week starting 2017-03-20
by Chris Koerner 31 Mar '17

31 Mar '17

It's Spring in the Northern Hemisphere. Seasonal allergies ahoy! This is the weekly update from the Discovery department for the week starting 2017-03-20 and the week of 2017-03-27. == Highlights == * Finished up the analysis of the first A/B test for the sister project search results [10] [11] == Discussions == === Search === * Updates from work done in previous weeks (apologies for the delay!): ** Fixed an issue where subpages were mistakenly preferred over parent pages if you didn't type the whole title [0] ** Fixed multiple issues with the how Elasticsearch integrates with the Translate extension [1] [2] [3] ** Test wiki with new Chinese Traditional-to-Simplified conversion, better word segmentation, and using the new BM25 is available in labs. [4] (It has the index and snippets from Chinese Wikipedia, but not the actual articles.) Additional analysis from native-speaker review is also on MediaWiki. More feedback always welcome. [5] * Fixed regressions which seem to be related to Elasticsearch 5 upgrade: ** Fixed an issue causing language-converted suggestions to not show in the search box [6] * Fixed an issue where using certain special characters in advanced search syntax would unexpectedly give zero results [7] * Finished the second A/B test for the sister project search results. * The analysis of new Ukrainian language analyzer is on MediaWiki, and there's lots of good discussion on Phabricator. More feedback welcome! [8] [9] === Analysis === * Finished up the analysis of the first A/B test for the sister project search results (task T156300) read it here [10] [11] === Wikidata Query Service === * Enabled SPARQL Federation [12] * Enabled parametrized query templates in the GUI [13] [0] https://phabricator.wikimedia.org/T156840 [1] https://phabricator.wikimedia.org/T132076 [2] https://phabricator.wikimedia.org/T132315 [3] https://phabricator.wikimedia.org/T158070 [4] http://zh-wp-smartcn-relforge.wmflabs.org/w/index.php?title=Special:搜索 [5] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Chinese_Analyzer_Ana… [6] https://phabricator.wikimedia.org/T160896 [7] https://phabricator.wikimedia.org/T147636 [8] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Ukrainian_Morfologik… [9] https://phabricator.wikimedia.org/T160106 [10] https://phabricator.wikimedia.org/T156300 [11] https://commons.wikimedia.org/wiki/File:A_Test_Of_Cross-wiki_Search_-_Helpi… ’s_Sister_Projects.pdf [12] https://www.wikidata.org/wiki/Wikidata:Project_chat#Federation_in_the_Wikid… [13] https://www.wikidata.org/wiki/Wikidata:Project_chat#Query_Service:_you_can_… --- The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Problem with updateSearchIndexConfig.php --startover
by Aran 31 Mar '17

31 Mar '17

Hello, I have some wikis running elastic search and after some development updates I find I need to rebuild the search indexes which I do as follows: .../CirrusSearch/maintenance/forceSearchIndex.php --skipLinks --indexOnSkip .../CirrusSearch/maintenance/forceSearchIndex.php --skipParse But I found that this doesn't work for new wikis that don't have indexes at all and so I put the following command before the other two: .../CirrusSearch/maintenance/updateSearchIndexConfig.php --startOver Later I found that this resulted in only the first namespace in each wiki being included in any search results... What is the proper command I should run to completely nuke and rebuild all indexes on a wiki? Thanks, Aran

2 3

Draft Proposal : Adding Data storage feature and upgrading Quiz extension - Gsoc17
by Harjot Bhatia 31 Mar '17

31 Mar '17

Hi, I'd like to share my Project proposal for Gsoc17 Draft Proposal Link: https://docs.google.com/document/d/1P-R68dlBrKaWrYZWts8twIztOj5mHjKJ8BsDFkQ… Phabricator Link : https://phabricator.wikimedia.org/T160257 I'd appreciate comments, feedback and suggestions. Thanks, Harjot Singh

1 0

Gsoc 2017 Draft Proposal
by Harjot Bhatia 31 Mar '17

31 Mar '17

Hi, I have earlier shared my proposal made on phabricator ( T160257 ). I'd like to share the draft proposal for Gsoc program site for feedback and suggestions. Link : Draft | | | | | | | | | | | Proposal : Adding Data storage feature and upgrading Quiz extension - Gsoc17 Profile Name : Harjot Singh Bhatia Time zone : UTC +5:30 Email : harjot4995(a)gmail.com Website : harjot4995.com I... | | | | Thanks. Harjot Singh

2 1

CREDIT showcase, Wednesday, 5-April-2017
by Lani Goto 31 Mar '17

31 Mar '17

Hi! I'd like to welcome you to join us at the CREDIT showcase next week, Wednesday, 5-April-2017 at 1800 UTC / 1100 Pacific Time. We'd like to see your demos, whether they're rough works in progress or polished production material, or even just a telling of something you've been studying recently. For more information on the upcoming event, as well as recordings of previous events, please visit the following page: https://www.mediawiki.org/wiki/CREDIT_showcase And if you'd like to share the news about the upcoming CREDIT showcase, here's some suggested verbiage. Thanks! *Hi <FNAME>* *I hope all is well with you! I wanted to let you know about CREDIT, a monthly demo series that we’re running to showcase open source tech projects from Wikimedia’s Community, Reading, Editing, Discovery, Infrastructure and Technology teams. * *CREDIT is open to the public, and we welcome questions and discussion. The next CREDIT will be held on April 5th at 11am PT / 2pm ET / 18:00 UTC. * *There’s more info on MediaWiki, and on Etherpad, which is where we take notes and ask questions. You can also ask questions on IRC in the Freenode chatroom #wikimedia-office (web-based access here). Links to video will become available at these locations shortly before the event.* *Please feel free to pass this information along to any interested folks. Our projects tend to focus on areas that might be of interest to folks working across the open source tech community: language detection, numerical sort, large data visualizations, maps, and all sorts of other things.* *If you have any questions, please let me know! Thanks, and I hope to see you at CREDIT.* *YOURNAME* -- Lani Goto Project Assistant, Engineering Admin

1 0

Call for candidates to form the first Code of Conduct Committee
by Quim Gil 30 Mar '17

30 Mar '17

Dear Wikimedia technical community members, https://www.mediawiki.org/wiki/Code_of_Conduct The review of the Code of Conduct for Wikimedia technical spaces has been completed and now it is time to bootstrap its first committee. The Technical Collaboration team is looking for five candidates to form the Committee plus five additional auxiliary members. One of them could be you or someone you know! You can propose yourself as a candidate and you can recommend others *privately* at techconductcandidates AT wikimedia DOT org We want to form a very diverse list of candidates reflecting the variety of people, activities, and spaces in the Wikimedia technical community. We are also open to other candidates with experience in the field. Diversity in the Committee is also a way to promote fairness and independence in their decisions. This means that no matter who you are, where you come from, what you work on, or for how long, you are a potential good member of this Committee. The main requirements to join the Committee are a will to foster an open and welcoming community and a commitment to making participation in Wikimedia technical projects a respectful and harassment-free experience for everyone. The committee will handle reports of unacceptable behavior, will analyze the cases, and will resolve on them according to the Code of Conduct. The Committee will also handle proposals to amend the Code of Conduct for the purpose of increasing its efficiency. The term of this first Committee will be one year. Once we have a list of 5 + 5 candidates, we will announce it here for review. You can learn more about the Committee and its selection process at https://www.mediawiki.org/wiki/Code_of_Conduct/Committee and you can ask questions in the related Talk page (preferred) or here. You can also track the progress of this bootstrapping process at https://www.mediawiki.org/wiki/Talk:Code_of_Conduct#Bootstrapping_the_Code_… PS: We have many technical spaces and reaching to all people potentially interested is hard! Please help spreading this call. -- Quim Gil Engineering Community Manager @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil

1 3

2017-03-29 Scrum of Scrums meeting notes
by Grace Gellerman 29 Mar '17

29 Mar '17

https://www.mediawiki.org/wiki/Scrum_of_scrums/2017-03-29 = 2017-03-29= contact: https://www.mediawiki.org/wiki/Wikimedia_Engineering == Call outs == * New version of the PDF render service in beta: https://en.wikipedia.beta.wmflabs.org/api/rest_v1/#!/Page_content/get_page_…, please verify it's all good! == Product == === Reading === ====Android==== * Last week: ** New saved page cache implementation in review \o/ T156917 ** Upgraded Mapbox SDK * Next week (https://phabricator.wikimedia.org/project/view/2352/ ): ** Reading on site ** Continue 2.5.x release https://phabricator.wikimedia.org/project/board/2667/ ==== iOS ==== * Last Week ** Released 5.4 https://phabricator.wikimedia.org/project/view/2326/ * This week ** Reading offsite ==== Community Tech ==== * Unable to attend hangout due to poor WiFi in WMCON hotel * Collaborating with WMDE's Technical Wishes team in testing and providing feedback on prototypes of features both teams are working on * No other major updates, most of our team is out this week for WMCON or on holiday ==== Multimedia ==== * Not blocking anything, waiting for some help from Services on deploying 3d2png, based on their recommendation to use the service template deploy method. Expect that to be done soon. ====Reading Infrastructure==== * TemplateStyles: ** CSS sanitizer library https://gerrit.wikimedia.org/r/#/q/project:css-sanitizer * Mobile Content Service: https://phabricator.wikimedia.org/project/view/2445/ ** Last week: adding more languages to TFA (Today's Featured Article) in aggregated feed T150806 ** This week: deploy ^ + reading offsite ==== Web ==== * Last week: Reading Web off-site * This week: Reading off-site === Editing === ==== UI Standardization ==== ** Continued work on style guide components. Both technical and content-wise https://phabricator.wikimedia.org/tag/wikimediaui_style_guide/ ** Unify padding/position of labels/icons/indicators across OOjs UI widgets in a generalized manner https://phabricator.wikimedia.org/T161177 ** Align the style for lists of pages https://phabricator.wikimedia.org/T153417 * Updates: ** OOjs UI: *** Release of v0.20.1 (still undergoing release) https://phabricator.wikimedia.org/diffusion/GOJU/browse/master/History.md **** Implement `grunt add-theme` task to ease theme creation (Bartosz Dziewoński) ==== Collaboration ==== * Blocked * Blocking * Updates ** RecentChanges Filters *** Port Wikidata to new RC Filters system. *** Add conflicts (tracking which filters don't make sense together) *** Continued enhancements to EventLogging for RC Filters *** Deployed as Beta Feature to Polish Wikipedia and Portuguese Wikipedia ** Flow *** Fix for long-standing DB issue *** Cleanup unused library ==== Parsing ==== * https://tools.wmflabs.org/wikitext-deprecation/ dashboard now tracks results from the Linter extension as well. Working through some bugs in the Linter extension and will continue with additional rollout this / next week. * Doing a bunch of CSS tweaking as part of visual diff tests ( https://parsoid-vd-tests.wikimedia.org/ ) comparing Parsoid rendering and PHP parser rendering. Continuing to update https://www.mediawiki.org/wiki/Parsoid/Known_differences_with_PHP_parser_ou… based on results of investigating the differences. * Going through a bunch of bugs filed against Parsoid and fixing them. * Parser Migration extension is ready for rollout -- probably next week. ==== Language ==== * Work on OOjs migration for Content Translation continue. == Technology == === Security === * Reviews: ** 2FA for iOS continues ** CodeMirror planned for next week * Meeting this week to discuss handoff of MW === Services === * Blockers: none * Updates: ** New version of electron PDF render service in BETA *** https://en.wikipedia.beta.wmflabs.org/api/rest_v1/#!/Page_content/get_page_… *** First update in half a year, big changes. Reading, WMDE, all parties interested - please beta-test *** Moving to production in next several days ** Cassandra client encryption enforced for the RESTBase cluster === Discovery === * No blockers * Elasticsearch 5 upgrade complete for search * Logstash upgraded on beta cluster, production update will follow soon * There are some issues with realtime suggestion updates, so it is stalled for now * Second A/B test for displaying sister wiki results finished, analysis in progress * First A/B test for sister wiki results analyzed: https://commons.wikimedia.org/wiki/File:A_Test_Of_Cross-wiki_Search_-_Helpi… * WDQS federation (calling out to other services) has been enabled, with small whitelist for starters === Technical Operations === * '''Blocking''' ** No one * '''Blocked''' ** DBAs asking for help on https://phabricator.wikimedia.org/T159319 * Updates ** Work on switchover goal concluding this week. Services are being switchover to discovery URLs across the infrastructure ** Precise Pangolin (Ubuntu 12.04) is officially removed from production (last instance done today) and the last few VMs that have not been migrated by their owners in labs will be forcefully shutdown ** Removal of precise supporting code from puppet has begun. ** HHVM 3.18 upgrade started, and then stalled. 3.18 follows strict Zend PHP 7.0 behavior, mediawiki code needs to be updated T161095 ** ganglia officially deprecated (since last Q), removal has started === Fundraising Tech === * Blockers: none * Mostly focused on the job-runner script to replace Jenkins * testing on some simple jobs ** Trickiest thing so far is streaming the subprocess's stdout to a file * More investigations about importing 3rd-party mailer data into CiviCRM * CentralNotice banner sequence feature: https://phabricator.wikimedia.org/T144453 * More PayPal express checkout fixes === Analytics === * Now serving pagecounts per project from AQS (legacy metric) [still beta - subject to changes] * Finalizing the replacement of the reportcard using the mentioned new AQS metric and others * More work on the Wikistats2.0 prototype, adding visual stubs for all functionalities before asking for feedback * Deploying right now, replacing user agent string by user agent map in EventLogging

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l March 2017