[discovery] Discovery Weekly Update for the week starting 2018-01-29

Hello again, Below you'll find the weekly update from the Search Platform team for the week starting 2018-01-29. == Highlights== * Several members of the Search Platform team participated in the annual Developer Summit held January 22-23, 2018 in Berkeley, California. [0] * During the WMF's All Hand's two day event, the Search Platform team met with the WMDE team and the Multimedia team to talk about the future of search and how the teams can work together on Structured Data on Commons. * The Search Platform team had their team offsite the two days after the All Hands meetings — lots of great conversations about the future of search were had. * Q3 Goals can be found on MediaWiki.org [1] == Discussions == === Search === * Chris Schilling opened a Phab ticket (T185721) on the difficulties of searching in the Khmer script. [2] [3] The Unicode encoding for the script uses many diacritics, and (with proper font support) the same glyph can be properly written with the underlying Unicode characters in any of several different orders, which complicates searching. If you are interested in learning more—or if you have any experience with computing in Khmer—please check out the Phab ticket. * Erik made a new utility script that reads in the spark dataframe and emits binary xgboost datasets to hdfs; all in order to switch Mjolnir to file based training [4] * Gehel cleaned up multiple definitions of logstash endpoint in puppet / hiera so that almost all references to the logstash host are now consolidated in a single variable [5] * Stas added hidden status to category dumps, to be deployed on Feb 7 [6] === Analysis === * Chelsy finished up a draft analysis result on a MLR test on Hebrew wiki that is being reviewed by the Search Platform team [7] [8] === WDQS === * Gehel took on the large-ish task of defining the constraints of the new WDQS cluster and getting that information on wiki. [9] [10] * Stas added support for continuations to WDQS queries to Mediawiki API [11] == Did you know? == English has a very large vocabulary—possibly larger than any other language. [12] In part this is because English likes to "[pursue] other languages down alleyways to beat them unconscious and rifle their pockets for new vocabulary," [13] and in part because of its history [14]—particularly the Norman conquest, which brought in a ruling class that spoke a dialect of Old French. An interesting consequence of this history is that English has distinct words for the animal, "cow," and for the meat of that animal, "beef." Surprisingly, both come from the same Proto-Indo-European [15] root: "gʷṓws". [16] The word "cow" derives from Proto-Germanic "kūz," while "beef" was borrowed from Old French "boef", derived in turn from Latin "bōs". Both "kūz" and "bōs" come from PIE "gʷṓws," though they obviously followed very different sound change [17] paths along the way. [0] https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2018 [1] https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2017-18_Q3#Progra… [2] https://phabricator.wikimedia.org/T185721 [3] https://en.wikipedia.org/wiki/Khmer_alphabet [4] https://phabricator.wikimedia.org/T184547 [5] https://phabricator.wikimedia.org/T182304 [6] https://phabricator.wikimedia.org/T173980 [7] https://phabricator.wikimedia.org/T183024 [8] https://analytics.wikimedia.org/datasets/discovery/reports/Second_MLR_Test_… [9] https://phabricator.wikimedia.org/T184083 [10] https://www.mediawiki.org/wiki/Wikidata_query_service/Implementation#Usage_… [11] https://phabricator.wikimedia.org/T178712 [12] https://en.wikipedia.org/wiki/English_language#Vocabulary [13] https://en.wikiquote.org/wiki/English_language [14] https://en.wikipedia.org/wiki/History_of_English [15] https://en.wikipedia.org/wiki/Proto-Indo-European_language [16] https://en.wiktionary.org/wiki/Reconstruction:Proto-Indo-European/g%CA%B7%E… [17] https://en.wikipedia.org/wiki/Sound_change --- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison Wikimedia Foundation