Discovery April 2019

discovery@lists.wikimedia.org

3 participants
6 discussions

Search Platform Team Office Hours May 1, 2019

by Trey Jones

The Search Platform Team <https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> usually holds office hours the first Wednesday of each month. Come talk to us about anything related to Wikimedia search! Feel free to add your items to the Etherpad Agenda for the next meeting. Details for our next meeting: Date: Wednesday, May 1st, 2019 Time: 15:00-16:00 GMT / 08:00-9:00 PDT / 11:00-12:00 EDT / 17:00-18:00 CEST Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours Google Meet link: https://meet.google.com/vyc-jvgq-dww *N.B.:* Google Meet System Requirements <https://support.google.com/meet/answer/7317473> Hope to talk to you next week! —Trey Trey Jones Sr. Software Engineer, Search Platform Wikimedia Foundation

3 years, 10 months

Discovery Weekly Update for the week starting 2019-04-15

by Chris Koerner

Greetings, This is the weekly update from the Search Platform team for the week starting 2019-04-15. As always, feedback and questions are welcome. == Highlights == * The Wikidata Query Service URL shortener has been switched to Wikimedia shortener. [9] This means you can now post short URLs on wiki! == Discussions == === Search === * Porting of search code from Wikibase to WikibaseCirrusSearch is finished. [0] and [1] * Trey worked on updating our Analysis Analysis Tools with some recent ad hoc improvements [2] * Mathew and Gehel created a cookbook to copy data between WDQS servers to copy data between wdqs servers [3] * A volunteer developer asked for our help to to be able to sort search results by date, as well as by 'relevance'. Upon looking at this again, we realized we had already implemented this functionality [4] but probably need some UI updates that are detailed in [5] * Erik and David worked on getting search sorted by creation date to include some missing items, it's been complete for a while but our mass indexing process takes about 8 weeks to finish up, and now it appears that it's all done [6] * Erik found that metrics were being reported for the wrong cluster when $wgCirrusSearchClusterOverrides is used and fixed it [7] * An error was found where a fatal error occurred when submitting "insource" query on mediawiki.org [8] === Wikidata Query Service === * UI URL shortener has been switched to Wikimedia shortener. [9] This means you can now post short URLs on wiki! [0] https://phabricator.wikimedia.org/T190022 [1] https://phabricator.wikimedia.org/T190022 [2] https://phabricator.wikimedia.org/T220124 [3] https://phabricator.wikimedia.org/T213401 [4] https://phabricator.wikimedia.org/T18237 [5] https://phabricator.wikimedia.org/T197525 [6] https://phabricator.wikimedia.org/T215487 [7] https://phabricator.wikimedia.org/T221019 [8] https://phabricator.wikimedia.org/T221058 [9] https://phabricator.wikimedia.org/T112715 ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

3 years, 11 months

Pre-run of my haystack talk on interleaved AB testing

by Erik Bernhardson

I'm going to be doing a run through of my haystack talk <https://haystackconf.com/2019/variance/> this Thursday at 9:30-10:30 PDT. The calendar even is scheduled for an hour, but the talk needs to fit in 45 minutes and I'm expecting closer to 35 minutes + questions. I've invited people currently or previously in discovery, but anyone interested is free to attend and ask some questions. The URL is pretty ugly, but this is what the gcal publish event gave me: https://calendar.google.com/event?action=TEMPLATE&tmeid=M2U5b2llMm5sOWYyM2N… Erik B.

3 years, 11 months

Discovery Weekly Update for the week starting 2019-04-08

by Chris Koerner

Greetings, This is the weekly update from the Search Platform team for the week starting 2019-04-08. As always, feedback and questions are welcome. == Discussions == === Search === * New WikibaseCirrusSearch code is deployed and enabled on all wikis, including Wikidata & Commons. [0] *We re-indexed all Greek, Turkish, and Irish-language wikis with better handling of Greek accents [1], Turkish İ/ı [2], and Irish initial mutation [3] when lowercasing searches in quotes. [4] * We noticed that indices can move to readonly state when disk space is low and needs to be reset manually once disk space is available again, so we made it a cookbook [5] * David wrote a patch to help with an exception from CirrusSearch/Sanity/Checker (cannot fetch ids from index) [6] * We fixed/finalized more things with the ES6 upgrade: ** Create checks that alerts on cirrussearch update lags [7] ** Elasticsearch 6: the classic similarity is deprecated [8] ** Setup elasticsearch on cloudelastic100[1-4] [9] ** Rack/setup/install cloudelastic100[1-4].eqiad.wmnet systems [10] * Erik converted mjolnir from KafkaRDD to direct kafka-python usage, because KafkaRDD in pyspark only supports the 0.8 api [11] === Wikidata Query Service === * Deployed patch to fix an issue in Blazegraph workbench [12] * Working on improving cache behavior of Updater [13] [0] https://phabricator.wikimedia.org/T218954 [1] https://en.wikipedia.org/wiki/Greek_diacritics#Accents [2] https://en.wikipedia.org/wiki/Dotted_and_dotless_I [3] https://en.wikipedia.org/wiki/Irish_initial_mutations [4] https://phabricator.wikimedia.org/T217806 [5] https://phabricator.wikimedia.org/T219799 [6] https://phabricator.wikimedia.org/T218945 [7] https://phabricator.wikimedia.org/T219601 [8] https://phabricator.wikimedia.org/T219268 [9] https://phabricator.wikimedia.org/T214921 [10] https://phabricator.wikimedia.org/T194186 [11] https://phabricator.wikimedia.org/T219932 [12] https://phabricator.wikimedia.org/T219406 [13] https://phabricator.wikimedia.org/T217897 ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

3 years, 11 months

Discovery Weekly Update for the week starting 2019-03-25

by Chris Koerner

Greetings, This is the weekly update from the Search Platform team for the week starting 2019-03-25 and 2019-04-01. As always, feedback and questions are welcome. == Discussions == === Search === * ElasticSearch upgrade to v6: ** incident [0] *Trey finished a deep dive into the performance of language identification for cross-wiki searching [1] (example [2]) and punctuation-related problems, and discovered things are working pretty well overall, but the Chinese language model is a bit off. * Erik noticed that the inlabel / incaption keywords should highlight the label/caption but were not [3] * David worked on fixing an error code that Elasticsearch 6 nested_path and nested_filter are deprecated [4] and _retry_on_conflict was deprecated [5] * We worked on migrating mjolnir to stdout/syslog/cee logging output [6] * The team worked on upgrade to elasticsearch 6.5.4 for cirrus / codfw (specifically) [7] and for eqiad [8] * Erik worked on the implementation and testing of glent m0 integration with wmf infrastructure [9] * David did a lot of work to update the mw-config to use the psi&omega elastic clusters [10] * David found that the auto_generate_phrase_queries is deprecated and ineffective [11] * The team fixed an old bug where we were getting fatal errors - "cannot perform this operation with arrays" from CirrusSearch/ElasticaWrite (using JobQueueDB) [12] * Gehel worked to make spicerack more robust when unfreezing writes to elasticsearch / cirrus [13] as well as creating a cookbook to reset frozen write state on elasticsearch / cirrus [14] * Stas moved WikibaseLexeme search code to WikibaseLexemeCirrusSearch extension [15] * We noticed that Elasticsearch indices went read-only, causing a huge lag [16] * We also saw where search exceptions handling was printing response information on the screen [17] * The team fixed an issue where mwgrep was not working [18] * We also fixed an issue where Elasticsearch 6 needed to silence deprecation warnings to avoid logspam [19] * We needed to create an extra elasticsearch clusters in the beta cluster [20] * We also needed some alerts so we know if mjolnir starts misbehaving [21] * We also converted check_elasticsearch.py icinga plugin to py3 [22] * We needed to start using local nginx reverse proxy for connections reuse [23] * The version of curator that we currently use (5.2.0) isn't compatible with elasticsearch 6. Which causes issues in a few cron on logtash servers (see blelow). Version 5.6.0 supports both elasticsearch 5 and 6.....so...we updated it [24] * We also did some cleanup of the reprepro configuration for elasticsearch-curator [25] * Getting a centralized way to inspect the content of the search profiles might be helpful when investigating search behaviors. In the same vein as other dump debug APIs (mapping/settings/cirrusdoc) David suggested that we should add a new simple API to dump the profiles (cirrus-profiles-dump) [26] * David also found that a call to a member function toArray() on a non-object (null) in vendor/ruflin/elastica/lib/Elastica/Client.php:736 and fixed it [27] [0] https://wikitech.wikimedia.org/wiki/Incident_documentation/20190327-elastic… report [1] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Review_of_Language_I… [2] https://en.wikipedia.org/w/index.php?search=%D0%93%D0%B0%D1%80%D1%80%D0%B8+… [3] https://phabricator.wikimedia.org/T217809 [4] https://phabricator.wikimedia.org/T219266 [5] https://phabricator.wikimedia.org/T219265 [6] https://phabricator.wikimedia.org/T218833 [7] https://phabricator.wikimedia.org/T218878 [8] https://phabricator.wikimedia.org/T218879 [9] https://phabricator.wikimedia.org/T218164 [10] https://phabricator.wikimedia.org/T210381 [11] https://phabricator.wikimedia.org/T219267 [12] https://phabricator.wikimedia.org/T124196 [13] https://phabricator.wikimedia.org/T219640 [14] https://phabricator.wikimedia.org/T219638 [15] https://phabricator.wikimedia.org/T216206 [16] https://phabricator.wikimedia.org/T219364 [17] https://phabricator.wikimedia.org/T216959 [18] https://phabricator.wikimedia.org/T219162 [19] https://phabricator.wikimedia.org/T219269 [20] https://phabricator.wikimedia.org/T213940 [21] https://phabricator.wikimedia.org/T214494 [22] https://phabricator.wikimedia.org/T215439 [23] https://phabricator.wikimedia.org/T215491 [24] https://phabricator.wikimedia.org/T218991 [25] https://phabricator.wikimedia.org/T216235 [26] https://phabricator.wikimedia.org/T218682 [27] https://phabricator.wikimedia.org/T217402 ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

3 years, 11 months

Upcoming Search Platform Office Hours—April 3rd

by Trey Jones

The Search Platform Team <https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> usually holds office hours the first Wednesday of each month. Come ask us anything about Wikimedia search! Feel free to add your items to the Etherpad Agenda for the next meeting. Details for our next meeting (note that we're on daylight savings time, so the time has shifted relative to GMT): Date: Wednesday, April 3rd, 2019 Time: 15:00-16:00 GMT / 08:00-9:00 PDT / 11:00-12:00 EDT / 17:00-18:00 CEST Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours Google Meet link: https://meet.google.com/vyc-jvgq-dww *N.B.:* Google Meet System Requirements <https://support.google.com/meet/answer/7317473> Hope to talk to you in a week! Trey Jones Sr. Software Engineer, Search Platform Wikimedia Foundation

3 years, 11 months

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery April 2019