Discovery March 2019

discovery@lists.wikimedia.org

3 participants
7 discussions

Upcoming Search Platform Office Hours—April 3rd

by Trey Jones

The Search Platform Team <https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> usually holds office hours the first Wednesday of each month. Come ask us anything about Wikimedia search! Feel free to add your items to the Etherpad Agenda for the next meeting. Details for our next meeting (note that we're on daylight savings time, so the time has shifted relative to GMT): Date: Wednesday, April 3rd, 2019 Time: 15:00-16:00 GMT / 08:00-9:00 PDT / 11:00-12:00 EDT / 17:00-18:00 CEST Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours Google Meet link: https://meet.google.com/vyc-jvgq-dww *N.B.:* Google Meet System Requirements <https://support.google.com/meet/answer/7317473> Hope to talk to you in a week! Trey Jones Sr. Software Engineer, Search Platform Wikimedia Foundation

4 years, 7 months

Discovery Weekly Update for the week starting 2019-03-18

by Chris Koerner

Hello, This is the weekly update from the Search Platform team for the week starting 2019-03-18. As always, feedback and questions are welcome. == Discussions == === Search === * Nearly done with the upgrade to ElasticSearch v6: ** Upgrade relforge to elasticsearch 6.5.4 [0] ** Upgrade deployment-elastic* to 6.5.4 [1] ** Build wmf-elasticsearch-search-plugins.deb that supports elastic 6.5.4 [2] * There was a minor bug in not getting a search result for "garrett marquis" or "Garrett Marquis"; Erik investigated and found it was a minor html issue and corrected it. FYI, it will take ~8 weeks for all content across all wikis to be reindexed. [3] * As part of the work done to separate the CirrusSearch/Elastic specific code from Wikibase code base, Stas deployed WikibaseCirrusSearch in production and all searches are now running through it [4] === WDQS === * New Release of Blazegraph (2.1.5) is announced. We have released 0.3.1 version of WDQS build with it. [5] [0] https://phabricator.wikimedia.org/T218116 [1] https://phabricator.wikimedia.org/T218247 [2] https://phabricator.wikimedia.org/T199791 [3] https://phabricator.wikimedia.org/T218001 [4] https://phabricator.wikimedia.org/T218715 [5] https://phabricator.wikimedia.org/T218746 ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

4 years, 8 months

Discovery Weekly Update for the week starting 2019-03-11

by Chris Koerner

Hello, This is the weekly update from the Search Platform team for the week starting 2019-03-11. As always, feedback and questions are welcome. == Discussions == === Search === * As part of the ElasticSearch v6 upgrade: ** uploaded ElasticSearch 6.5.4 packages to reprepro [0] ** Adapted the ElasticSearch puppet module [1] ** Tested that we can run multiple ElasticSearch clusters on same hardware [2] * We discovered that the Maven Wrapper does not support XDG_CACHE_HOME, so we updated java8: to actually use the new image; fixing the mvn wrapper cache issue [3] * Upon unpacking the Greek analysis chain, we found that the filter isn't properly handling language-specific lowercasing in language analyzers; fixed it by adding Greek empty-token filter and keep lang-specific lowercasing [4] * As part of the lang-specific lowercasing, we also upgraded the ICU normalization [5] [6] * We decided we should use a better/cleaner approach for icinga elasticsearch check and merged http and https elasticsearch icinga checks into one [7] * We also updated the alerts for mjolnir daemons, so we can tell when it starts misbehaving [8] [0] https://phabricator.wikimedia.org/T218113 [1] https://phabricator.wikimedia.org/T217196 [2] https://phabricator.wikimedia.org/T193654 [3] https://phabricator.wikimedia.org/T218099 [4] https://phabricator.wikimedia.org/T217602 [5] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Greek_and_Unexpected… [6] https://phabricator.wikimedia.org/T203117 [7] https://phabricator.wikimedia.org/T215587 [8] https://phabricator.wikimedia.org/T214494 ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

4 years, 8 months

Discussion around CirrusSearch extensibility

by David Causse

Hi, I've put a couple pages on mw.org to start discussing about the problem we are currently facing as to how extensions can interact with CirrusSearch without stepping on each other toes. To describe the problem briefly: some extension provides additional data to the wiki structure (generally through custom Content Handlers) and CirrusSearch itself is not aware of the best ways to take benefits of this additional data to ameliorate ranking or results display. Cirrus has provided (organically) various hooks to let us build what we have today: - wikidata search integration for Entities, Properties & Lexemes - search keywords But as more and more integrations have to be done (SDoC) we need to step back and decide on better ways to let extensions augment the search experience. Beware that the discussion at this stage may only be relevant to developers who worked on these extensions. A page describes the current "query construction" mechanism[1] (with an emphasis on parts that poses problems at the moment) and a first list of use cases and a first set of solutions[2]. This is just a starting point for the discussion. Thanks for your input. [1] https://www.mediawiki.org/wiki/Extension:CirrusSearch/Query_Construction [2] https://www.mediawiki.org/wiki/Extension:CirrusSearch/Query_Construction/Us…

4 years, 8 months

Discovery Weekly Update for the week starting 2019-03-04

by Chris Koerner

Good to see you, This is the weekly update from the Search Platform team for the week starting 2019-03-04. As always, feedback and questions are welcome. == Discussions == === Search === * We've updated Textcat with the new Russian wrong-keyboard and wrong-encoding models and now need CirrusSearch to be up to date as well [0] * In order to ease migration to ES6 we should migrate to 5.6.14 first (epic task is now completed) [1] ** As part of the ES6 update, we needed to upgrade logstash and the logstash elasticsearch cluster to 5.6.14 [2] ** We also upgraded logstash plugin to 5.6.14 as part of the prep for elasticsearch upgrade [3] * We've deployed & tested WikibaseCirrusSearch on beta cluster [4] * We've also deployed & tested WikibaseCirrusSearch on testwikidata and found things to be good! [5] * We found that incigna monitoring for Elasticsearch doesn't seem to notice when an out-of-memory error has happened on a node — this will be fully fixed with the next cluster restart [6] * There was some logspam due to deduplicating I18N messages between Wikibase and WikibaseCirrusSearch that we resolved [7] * A discovery was made with a WMFTimeoutException in Special:Search that was resolved by decreasing timeouts and removing our timeout hack for regexes [8] * An umbrella task to track patches of merge commits to activate elastic6 support code [9] * A ticket created way back in 2016 asked: 'the new substring search is great on mw.o. Can we have it on office wiki too?' and yes, now it is [10] === Wikidata Query Service === * We've created CI testing environment for Blazegraph on our Jenkins CI [11] * Working on hiring a contractor to help us with Blazegraph tasks [12] * We noticed that metrics from wdqs updater JMX should be prefixed while we were investigating something else, metrics exposed by jmx_exporter running on wdqs-updater should be prefixed with wdqs_updater_ to make it more clear what they are referring to (done!) [13] == Other Noteworthy Stuff == * Read Trey's new blog post about 'the anatomy of search: a place for my stuff' [14] [0] https://phabricator.wikimedia.org/T216083 [1] https://phabricator.wikimedia.org/T215931 [2] https://phabricator.wikimedia.org/T216052 [3] https://phabricator.wikimedia.org/T216993 [4] https://phabricator.wikimedia.org/T215684 [5] https://phabricator.wikimedia.org/T217276 [6] https://phabricator.wikimedia.org/T76090 [7] https://phabricator.wikimedia.org/T216839 [8] https://phabricator.wikimedia.org/T216860 [9] https://phabricator.wikimedia.org/T217043 [10] https://phabricator.wikimedia.org/T150153 [11] https://phabricator.wikimedia.org/T216855 [12] https://phabricator.wikimedia.org/T200558 [13] https://phabricator.wikimedia.org/T208215 [14] https://wikimediafoundation.org/2019/03/12/the-anatomy-of-search-a-place-fo… ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

4 years, 8 months

Upcoming Search Platform Office Hours—March 6th

by Trey Jones

The Search Platform Team <https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> usually holds office hours the first Wednesday of each month. Come ask us anything about Wikimedia search! We’re particularly interested in: * Opportunities for collaboration—internally or externally to the Wikimedia Foundation * Challenges you have with on-wiki search, in any of the languages we support But we're happy to talk about anything search-related. Feel free to add your items to the Etherpad Agenda for the next meeting. Details for our next meeting: Date: Wednesday, March 6th, 2018 Time: 16:00-17:00 GMT / 08:00-9:00 PST / 11:00-12:00 EST / 17:00-18:00 CET Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours Google Meet link: https://meet.google.com/vyc-jvgq-dww *N.B.:* Google Meet System Requirements <https://support.google.com/meet/answer/7317473> Trey Jones Sr. Software Engineer, Search Platform Wikimedia Foundation

4 years, 8 months

Discovery Weekly Update for the week starting 2019-02-25

by Chris Koerner

Hello, This is the weekly update from the Search Platform team for the week starting 2019-02-25. As always, feedback and questions are welcome. == Discussions == === Search === * New extension WikibaseCirrusSearch [0], encapsulating search code from Wikibase [1], is deployed on Beta [2] and will be tested on production next week [3]. *New phan checks are enabled for CirrusSearch and WikibaseCirrusSearch === Wikidata Query Service === * Stas fixed issues with small negative data display [4] and breakage in short URLs due to Bootstrap security fixes [5]. == Events and News == * We'll be holding our next Search Platform Office Hours on Wednesday, March 6th: **Time: 16:00-17:00 GMT / 08:00-9:00 PST / 11:00-12:00 EST / 17:00-18:00 CET **Etherpad: [6] **Google Meet link: [7] [0] https://www.mediawiki.org/wiki/Extension:WikibaseCirrusSearch [1] https://phabricator.wikimedia.org/T190022 [2] https://phabricator.wikimedia.org/T215684 [3] https://phabricator.wikimedia.org/T217276 [4] https://phabricator.wikimedia.org/T211007 [5] https://phabricator.wikimedia.org/T217161 [6] https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours [7] https://meet.google.com/vyc-jvgq-dww ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

4 years, 8 months

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery March 2019