Greetings, This is the weekly update from the Search Platform team for the week starting 2018-07-30. As always, feedback and questions welcome. == Discussions == === Search === * David did a lot of work surrounding the upgrade to ElasticSearch 6.3: ** updated an extra plugin[0] ** updated the LTR plugin [1] ** worked on an older issue that we were waiting to fix for an experimental plugin that was throwing an exception on some requests [2] ** upgraded the experimental highlighter [3] ** upgraded the extra-analysis plugin [4] * Trey fixed an address stemming issue in Polish analyzer for search [5] and it'll be fully in production when a re-index occurs [6] * Trey also worked on a review of the Esperanto Morphological Libraries when a volunteer wanted to help in this effort (and knew Esperanto) [7] * David worked on a few cirrus integration tests are were randomly and consistently failing (V => Venom > V:N) [8] * Trey took on the task of reviewing the work of applying Indonesian Analysis Chain for Malay (writeup and review) [9] [10] * David took on the fairly massive task of changing how SpecialSearch/SearchEngine handles the 'prefix' URI param set by the InputBox extension with lots of help from others [11] * David did quite a bit of work on deprecating the SearchEngine::replacePrefixes with several related patches [12] * Trey worked on exploring the potential applications for NLP to be applied in Search (review and write-up [13] [14] * Stas implemented fulltext search for Lexemes when Lexeme namespace is requested in the search [15] * Stas added collecting click data from Wikidata prefix search into event logs so that we know what the users are selecting when using Wikidata prefix search [16] and it'll be in production late this week with the train * Erik and David worked on the completion suggester code which didn't gracefully handle shard failures during the fetch phase, which caused the response received by cirrus to not contain the necessary information [17] and it'll be in production late this week with the train * After a lot of discussion, Erik fixed an issue where intitle search didn't match the stop words [18] and it'll be in production late this week with the train * Gehel and Erik worked on ensuring discovery.query_clicks_* data is purged per privacy policy, with adding support for hourly or daily partition dropping [19] === Wikidata Query Service === * Wikibase Constraints violations are now loaded into WDQS database and are queryable. [20] [0] https://phabricator.wikimedia.org/T198067 [1] https://phabricator.wikimedia.org/T198066 [2] https://phabricator.wikimedia.org/T178425 [3] https://phabricator.wikimedia.org/T198069 [4] https://phabricator.wikimedia.org/T198068 [5] https://phabricator.wikimedia.org/T186046 [6] https://phabricator.wikimedia.org/T200037 [7] https://phabricator.wikimedia.org/T197240 [8] https://phabricator.wikimedia.org/T199445 [9] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Analysis_of_Applying… [10] https://phabricator.wikimedia.org/T196780 [11] https://phabricator.wikimedia.org/T198318 [12] https://phabricator.wikimedia.org/T198860 [13] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Potential_Applicatio… [14] https://phabricator.wikimedia.org/T193070 [15] https://phabricator.wikimedia.org/T196188 [16] https://phabricator.wikimedia.org/T196186 [17] https://phabricator.wikimedia.org/T180807 [18] https://phabricator.wikimedia.org/T68969 [19] https://phabricator.wikimedia.org/T189845 [20] https://phabricator.wikimedia.org/T172380 --- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Relations Specialist Wikimedia Foundation