Discovery

discovery@lists.wikimedia.org

748 discussions

Discovery Weekly Update for the week starting 2019-04-15

by Chris Koerner

Greetings, This is the weekly update from the Search Platform team for the week starting 2019-04-15. As always, feedback and questions are welcome. == Highlights == * The Wikidata Query Service URL shortener has been switched to Wikimedia shortener. [9] This means you can now post short URLs on wiki! == Discussions == === Search === * Porting of search code from Wikibase to WikibaseCirrusSearch is finished. [0] and [1] * Trey worked on updating our Analysis Analysis Tools with some recent ad hoc improvements [2] * Mathew and Gehel created a cookbook to copy data between WDQS servers to copy data between wdqs servers [3] * A volunteer developer asked for our help to to be able to sort search results by date, as well as by 'relevance'. Upon looking at this again, we realized we had already implemented this functionality [4] but probably need some UI updates that are detailed in [5] * Erik and David worked on getting search sorted by creation date to include some missing items, it's been complete for a while but our mass indexing process takes about 8 weeks to finish up, and now it appears that it's all done [6] * Erik found that metrics were being reported for the wrong cluster when $wgCirrusSearchClusterOverrides is used and fixed it [7] * An error was found where a fatal error occurred when submitting "insource" query on mediawiki.org [8] === Wikidata Query Service === * UI URL shortener has been switched to Wikimedia shortener. [9] This means you can now post short URLs on wiki! [0] https://phabricator.wikimedia.org/T190022 [1] https://phabricator.wikimedia.org/T190022 [2] https://phabricator.wikimedia.org/T220124 [3] https://phabricator.wikimedia.org/T213401 [4] https://phabricator.wikimedia.org/T18237 [5] https://phabricator.wikimedia.org/T197525 [6] https://phabricator.wikimedia.org/T215487 [7] https://phabricator.wikimedia.org/T221019 [8] https://phabricator.wikimedia.org/T221058 [9] https://phabricator.wikimedia.org/T112715 ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

5 years, 1 month

Pre-run of my haystack talk on interleaved AB testing

by Erik Bernhardson

I'm going to be doing a run through of my haystack talk <https://haystackconf.com/2019/variance/> this Thursday at 9:30-10:30 PDT. The calendar even is scheduled for an hour, but the talk needs to fit in 45 minutes and I'm expecting closer to 35 minutes + questions. I've invited people currently or previously in discovery, but anyone interested is free to attend and ask some questions. The URL is pretty ugly, but this is what the gcal publish event gave me: https://calendar.google.com/event?action=TEMPLATE&tmeid=M2U5b2llMm5sOWYyM2N… Erik B.

5 years, 1 month

Discovery Weekly Update for the week starting 2019-04-08

by Chris Koerner

Greetings, This is the weekly update from the Search Platform team for the week starting 2019-04-08. As always, feedback and questions are welcome. == Discussions == === Search === * New WikibaseCirrusSearch code is deployed and enabled on all wikis, including Wikidata & Commons. [0] *We re-indexed all Greek, Turkish, and Irish-language wikis with better handling of Greek accents [1], Turkish İ/ı [2], and Irish initial mutation [3] when lowercasing searches in quotes. [4] * We noticed that indices can move to readonly state when disk space is low and needs to be reset manually once disk space is available again, so we made it a cookbook [5] * David wrote a patch to help with an exception from CirrusSearch/Sanity/Checker (cannot fetch ids from index) [6] * We fixed/finalized more things with the ES6 upgrade: ** Create checks that alerts on cirrussearch update lags [7] ** Elasticsearch 6: the classic similarity is deprecated [8] ** Setup elasticsearch on cloudelastic100[1-4] [9] ** Rack/setup/install cloudelastic100[1-4].eqiad.wmnet systems [10] * Erik converted mjolnir from KafkaRDD to direct kafka-python usage, because KafkaRDD in pyspark only supports the 0.8 api [11] === Wikidata Query Service === * Deployed patch to fix an issue in Blazegraph workbench [12] * Working on improving cache behavior of Updater [13] [0] https://phabricator.wikimedia.org/T218954 [1] https://en.wikipedia.org/wiki/Greek_diacritics#Accents [2] https://en.wikipedia.org/wiki/Dotted_and_dotless_I [3] https://en.wikipedia.org/wiki/Irish_initial_mutations [4] https://phabricator.wikimedia.org/T217806 [5] https://phabricator.wikimedia.org/T219799 [6] https://phabricator.wikimedia.org/T218945 [7] https://phabricator.wikimedia.org/T219601 [8] https://phabricator.wikimedia.org/T219268 [9] https://phabricator.wikimedia.org/T214921 [10] https://phabricator.wikimedia.org/T194186 [11] https://phabricator.wikimedia.org/T219932 [12] https://phabricator.wikimedia.org/T219406 [13] https://phabricator.wikimedia.org/T217897 ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

5 years, 1 month

Discovery Weekly Update for the week starting 2019-03-25

by Chris Koerner

Greetings, This is the weekly update from the Search Platform team for the week starting 2019-03-25 and 2019-04-01. As always, feedback and questions are welcome. == Discussions == === Search === * ElasticSearch upgrade to v6: ** incident [0] *Trey finished a deep dive into the performance of language identification for cross-wiki searching [1] (example [2]) and punctuation-related problems, and discovered things are working pretty well overall, but the Chinese language model is a bit off. * Erik noticed that the inlabel / incaption keywords should highlight the label/caption but were not [3] * David worked on fixing an error code that Elasticsearch 6 nested_path and nested_filter are deprecated [4] and _retry_on_conflict was deprecated [5] * We worked on migrating mjolnir to stdout/syslog/cee logging output [6] * The team worked on upgrade to elasticsearch 6.5.4 for cirrus / codfw (specifically) [7] and for eqiad [8] * Erik worked on the implementation and testing of glent m0 integration with wmf infrastructure [9] * David did a lot of work to update the mw-config to use the psi&omega elastic clusters [10] * David found that the auto_generate_phrase_queries is deprecated and ineffective [11] * The team fixed an old bug where we were getting fatal errors - "cannot perform this operation with arrays" from CirrusSearch/ElasticaWrite (using JobQueueDB) [12] * Gehel worked to make spicerack more robust when unfreezing writes to elasticsearch / cirrus [13] as well as creating a cookbook to reset frozen write state on elasticsearch / cirrus [14] * Stas moved WikibaseLexeme search code to WikibaseLexemeCirrusSearch extension [15] * We noticed that Elasticsearch indices went read-only, causing a huge lag [16] * We also saw where search exceptions handling was printing response information on the screen [17] * The team fixed an issue where mwgrep was not working [18] * We also fixed an issue where Elasticsearch 6 needed to silence deprecation warnings to avoid logspam [19] * We needed to create an extra elasticsearch clusters in the beta cluster [20] * We also needed some alerts so we know if mjolnir starts misbehaving [21] * We also converted check_elasticsearch.py icinga plugin to py3 [22] * We needed to start using local nginx reverse proxy for connections reuse [23] * The version of curator that we currently use (5.2.0) isn't compatible with elasticsearch 6. Which causes issues in a few cron on logtash servers (see blelow). Version 5.6.0 supports both elasticsearch 5 and 6.....so...we updated it [24] * We also did some cleanup of the reprepro configuration for elasticsearch-curator [25] * Getting a centralized way to inspect the content of the search profiles might be helpful when investigating search behaviors. In the same vein as other dump debug APIs (mapping/settings/cirrusdoc) David suggested that we should add a new simple API to dump the profiles (cirrus-profiles-dump) [26] * David also found that a call to a member function toArray() on a non-object (null) in vendor/ruflin/elastica/lib/Elastica/Client.php:736 and fixed it [27] [0] https://wikitech.wikimedia.org/wiki/Incident_documentation/20190327-elastic… report [1] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Review_of_Language_I… [2] https://en.wikipedia.org/w/index.php?search=%D0%93%D0%B0%D1%80%D1%80%D0%B8+… [3] https://phabricator.wikimedia.org/T217809 [4] https://phabricator.wikimedia.org/T219266 [5] https://phabricator.wikimedia.org/T219265 [6] https://phabricator.wikimedia.org/T218833 [7] https://phabricator.wikimedia.org/T218878 [8] https://phabricator.wikimedia.org/T218879 [9] https://phabricator.wikimedia.org/T218164 [10] https://phabricator.wikimedia.org/T210381 [11] https://phabricator.wikimedia.org/T219267 [12] https://phabricator.wikimedia.org/T124196 [13] https://phabricator.wikimedia.org/T219640 [14] https://phabricator.wikimedia.org/T219638 [15] https://phabricator.wikimedia.org/T216206 [16] https://phabricator.wikimedia.org/T219364 [17] https://phabricator.wikimedia.org/T216959 [18] https://phabricator.wikimedia.org/T219162 [19] https://phabricator.wikimedia.org/T219269 [20] https://phabricator.wikimedia.org/T213940 [21] https://phabricator.wikimedia.org/T214494 [22] https://phabricator.wikimedia.org/T215439 [23] https://phabricator.wikimedia.org/T215491 [24] https://phabricator.wikimedia.org/T218991 [25] https://phabricator.wikimedia.org/T216235 [26] https://phabricator.wikimedia.org/T218682 [27] https://phabricator.wikimedia.org/T217402 ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

5 years, 1 month

Upcoming Search Platform Office Hours—April 3rd

by Trey Jones

The Search Platform Team <https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> usually holds office hours the first Wednesday of each month. Come ask us anything about Wikimedia search! Feel free to add your items to the Etherpad Agenda for the next meeting. Details for our next meeting (note that we're on daylight savings time, so the time has shifted relative to GMT): Date: Wednesday, April 3rd, 2019 Time: 15:00-16:00 GMT / 08:00-9:00 PDT / 11:00-12:00 EDT / 17:00-18:00 CEST Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours Google Meet link: https://meet.google.com/vyc-jvgq-dww *N.B.:* Google Meet System Requirements <https://support.google.com/meet/answer/7317473> Hope to talk to you in a week! Trey Jones Sr. Software Engineer, Search Platform Wikimedia Foundation

5 years, 2 months

Discovery Weekly Update for the week starting 2019-03-18

by Chris Koerner

Hello, This is the weekly update from the Search Platform team for the week starting 2019-03-18. As always, feedback and questions are welcome. == Discussions == === Search === * Nearly done with the upgrade to ElasticSearch v6: ** Upgrade relforge to elasticsearch 6.5.4 [0] ** Upgrade deployment-elastic* to 6.5.4 [1] ** Build wmf-elasticsearch-search-plugins.deb that supports elastic 6.5.4 [2] * There was a minor bug in not getting a search result for "garrett marquis" or "Garrett Marquis"; Erik investigated and found it was a minor html issue and corrected it. FYI, it will take ~8 weeks for all content across all wikis to be reindexed. [3] * As part of the work done to separate the CirrusSearch/Elastic specific code from Wikibase code base, Stas deployed WikibaseCirrusSearch in production and all searches are now running through it [4] === WDQS === * New Release of Blazegraph (2.1.5) is announced. We have released 0.3.1 version of WDQS build with it. [5] [0] https://phabricator.wikimedia.org/T218116 [1] https://phabricator.wikimedia.org/T218247 [2] https://phabricator.wikimedia.org/T199791 [3] https://phabricator.wikimedia.org/T218001 [4] https://phabricator.wikimedia.org/T218715 [5] https://phabricator.wikimedia.org/T218746 ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

5 years, 2 months

Discovery Weekly Update for the week starting 2019-03-11

by Chris Koerner

Hello, This is the weekly update from the Search Platform team for the week starting 2019-03-11. As always, feedback and questions are welcome. == Discussions == === Search === * As part of the ElasticSearch v6 upgrade: ** uploaded ElasticSearch 6.5.4 packages to reprepro [0] ** Adapted the ElasticSearch puppet module [1] ** Tested that we can run multiple ElasticSearch clusters on same hardware [2] * We discovered that the Maven Wrapper does not support XDG_CACHE_HOME, so we updated java8: to actually use the new image; fixing the mvn wrapper cache issue [3] * Upon unpacking the Greek analysis chain, we found that the filter isn't properly handling language-specific lowercasing in language analyzers; fixed it by adding Greek empty-token filter and keep lang-specific lowercasing [4] * As part of the lang-specific lowercasing, we also upgraded the ICU normalization [5] [6] * We decided we should use a better/cleaner approach for icinga elasticsearch check and merged http and https elasticsearch icinga checks into one [7] * We also updated the alerts for mjolnir daemons, so we can tell when it starts misbehaving [8] [0] https://phabricator.wikimedia.org/T218113 [1] https://phabricator.wikimedia.org/T217196 [2] https://phabricator.wikimedia.org/T193654 [3] https://phabricator.wikimedia.org/T218099 [4] https://phabricator.wikimedia.org/T217602 [5] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Greek_and_Unexpected… [6] https://phabricator.wikimedia.org/T203117 [7] https://phabricator.wikimedia.org/T215587 [8] https://phabricator.wikimedia.org/T214494 ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

5 years, 2 months

Discussion around CirrusSearch extensibility

by David Causse

Hi, I've put a couple pages on mw.org to start discussing about the problem we are currently facing as to how extensions can interact with CirrusSearch without stepping on each other toes. To describe the problem briefly: some extension provides additional data to the wiki structure (generally through custom Content Handlers) and CirrusSearch itself is not aware of the best ways to take benefits of this additional data to ameliorate ranking or results display. Cirrus has provided (organically) various hooks to let us build what we have today: - wikidata search integration for Entities, Properties & Lexemes - search keywords But as more and more integrations have to be done (SDoC) we need to step back and decide on better ways to let extensions augment the search experience. Beware that the discussion at this stage may only be relevant to developers who worked on these extensions. A page describes the current "query construction" mechanism[1] (with an emphasis on parts that poses problems at the moment) and a first list of use cases and a first set of solutions[2]. This is just a starting point for the discussion. Thanks for your input. [1] https://www.mediawiki.org/wiki/Extension:CirrusSearch/Query_Construction [2] https://www.mediawiki.org/wiki/Extension:CirrusSearch/Query_Construction/Us…

5 years, 2 months

Discovery Weekly Update for the week starting 2019-03-04

by Chris Koerner

Good to see you, This is the weekly update from the Search Platform team for the week starting 2019-03-04. As always, feedback and questions are welcome. == Discussions == === Search === * We've updated Textcat with the new Russian wrong-keyboard and wrong-encoding models and now need CirrusSearch to be up to date as well [0] * In order to ease migration to ES6 we should migrate to 5.6.14 first (epic task is now completed) [1] ** As part of the ES6 update, we needed to upgrade logstash and the logstash elasticsearch cluster to 5.6.14 [2] ** We also upgraded logstash plugin to 5.6.14 as part of the prep for elasticsearch upgrade [3] * We've deployed & tested WikibaseCirrusSearch on beta cluster [4] * We've also deployed & tested WikibaseCirrusSearch on testwikidata and found things to be good! [5] * We found that incigna monitoring for Elasticsearch doesn't seem to notice when an out-of-memory error has happened on a node — this will be fully fixed with the next cluster restart [6] * There was some logspam due to deduplicating I18N messages between Wikibase and WikibaseCirrusSearch that we resolved [7] * A discovery was made with a WMFTimeoutException in Special:Search that was resolved by decreasing timeouts and removing our timeout hack for regexes [8] * An umbrella task to track patches of merge commits to activate elastic6 support code [9] * A ticket created way back in 2016 asked: 'the new substring search is great on mw.o. Can we have it on office wiki too?' and yes, now it is [10] === Wikidata Query Service === * We've created CI testing environment for Blazegraph on our Jenkins CI [11] * Working on hiring a contractor to help us with Blazegraph tasks [12] * We noticed that metrics from wdqs updater JMX should be prefixed while we were investigating something else, metrics exposed by jmx_exporter running on wdqs-updater should be prefixed with wdqs_updater_ to make it more clear what they are referring to (done!) [13] == Other Noteworthy Stuff == * Read Trey's new blog post about 'the anatomy of search: a place for my stuff' [14] [0] https://phabricator.wikimedia.org/T216083 [1] https://phabricator.wikimedia.org/T215931 [2] https://phabricator.wikimedia.org/T216052 [3] https://phabricator.wikimedia.org/T216993 [4] https://phabricator.wikimedia.org/T215684 [5] https://phabricator.wikimedia.org/T217276 [6] https://phabricator.wikimedia.org/T76090 [7] https://phabricator.wikimedia.org/T216839 [8] https://phabricator.wikimedia.org/T216860 [9] https://phabricator.wikimedia.org/T217043 [10] https://phabricator.wikimedia.org/T150153 [11] https://phabricator.wikimedia.org/T216855 [12] https://phabricator.wikimedia.org/T200558 [13] https://phabricator.wikimedia.org/T208215 [14] https://wikimediafoundation.org/2019/03/12/the-anatomy-of-search-a-place-fo… ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner (he/him) Community Relations Specialist Wikimedia Foundation

5 years, 2 months

Upcoming Search Platform Office Hours—March 6th

by Trey Jones

The Search Platform Team <https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> usually holds office hours the first Wednesday of each month. Come ask us anything about Wikimedia search! We’re particularly interested in: * Opportunities for collaboration—internally or externally to the Wikimedia Foundation * Challenges you have with on-wiki search, in any of the languages we support But we're happy to talk about anything search-related. Feel free to add your items to the Etherpad Agenda for the next meeting. Details for our next meeting: Date: Wednesday, March 6th, 2018 Time: 16:00-17:00 GMT / 08:00-9:00 PST / 11:00-12:00 EST / 17:00-18:00 CET Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours Google Meet link: https://meet.google.com/vyc-jvgq-dww *N.B.:* Google Meet System Requirements <https://support.google.com/meet/answer/7317473> Trey Jones Sr. Software Engineer, Search Platform Wikimedia Foundation

5 years, 2 months

← Newer
1
...
7
8
9
10
11
12
13
...
75
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery