Discovery December 2017

discovery@lists.wikimedia.org

5 participants
7 discussions

Discovery Weekly Update for the week starting 2017-12-11

by Chris Koerner

Hello, This is the weekly update from the Search Platform team for the week starting 2017-12-11. *Note*: Because of the holidays the next weekly update will be sent out on 8 January 2018. See you in the new year! == Highlights == * We've officially closed out the Discovery Department wiki page (to include the portal work) and added a new Search Platform page. [0] [1] [2] * The automation of the Wikipedia.org Portal page is now complete. Going forward, as part of the automation process, two tickets will continue to be updated for the stats and translations weekly updates. More information on how the automation is done can be found in the documentation on diffusion. 🎁 🎉 [3] [4] [5] * "Gnomes and trolls and hobgoblins (oh my!)—Failed queries and the vicarious fear of missing out" ** Check out Trey's latest blog post! [6] == Discussions == === Search === * We closed out the large ticket that tracked a lot of the work that we've done so far for backend data engineering and plumbing for machine learning to rank [7] * We also closed out a ticket that fixed an issue with some features that were being hidden by the completion suggester such as sub phrase matching on wikisource. [8] * We've ported all our Selenium tests from Ruby to Node.js for the Search Platform team [9] * Trey finished up a long review of the Serbian Morphological Libraries that are available [10] [11] * We did some tuning to Wikidata prefix search [12] * Added ability for searches to return extra match data in API responses (Wikidata will use it) [13] === WDQS === * |Categories are now updated automatically into Blazegraph each week on Monday. [14] === Portal === * Jan finished up several small tasks related to the automation of the Wikipedia.org portal page [15]: ** Update technical documentation for portals repository [16] ** Fixed a minor UI problem with rendering the small print in English and Russian [17] == FY 2017-18 Q2 (Oct-Dec) goals == This status was last updated 2017-12-12 [18] [0] https://www.mediawiki.org/wiki/Wikimedia_Discovery [1] https://www.mediawiki.org/wiki/Wikipedia.org_Portal [2] https://www.mediawiki.org/wiki/Wikimedia_Search_Platform [3] https://phabricator.wikimedia.org/T128546 [4] https://phabricator.wikimedia.org/T142582 [5] https://phabricator.wikimedia.org/diffusion/WPOR/browse/master/README.md [6] https://blog.wikimedia.org/2017/12/12/failed-queries-fear-of-missing-out/ [7] https://phabricator.wikimedia.org/T162053 [8] https://phabricator.wikimedia.org/T178474 [9] https://phabricator.wikimedia.org/T178474 [10] https://phabricator.wikimedia.org/T178926 [11] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Serbian_Stemmer_Anal… [12] https://phabricator.wikimedia.org/T182136 [13] https://phabricator.wikimedia.org/T182145 [14] https://www.mediawiki.org/wiki/Wikidata_query_service/Categories [15] https://phabricator.wikimedia.org/T140159 [16] https://phabricator.wikimedia.org/T182119 [17] https://phabricator.wikimedia.org/T178744 [18] https://www.mediawiki.org/wiki/Discovery/Status_updates/2017-12-11#FY_2017-… --- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison Wikimedia Foundation

5 years, 11 months

Elasticsearch metrics moving to Prometheus

by Guillaume Lederrey

Hello! We are moving all our metrics from Graphite [1] to Prometheus [2]. See the master task for some details [3]. We are now working on porting all those metrics for elasticsearch [4]. This means that we'll need to update our dashboards to start using those new metrics. During the transition, I'll keep the existing dashboards, but create a copy to migrate the metrics. For example, the "Elasticsearch Percentiles" [5] dashboard has a copy named "Elasticsearch Percentiles - Prometheus" [6]. Those new dashboards should behave in just the same way, but please let me know if you find any incoherence! Have fun! Guillaume [1] https://wikitech.wikimedia.org/wiki/Graphite [2] https://wikitech.wikimedia.org/wiki/Prometheus [3] https://phabricator.wikimedia.org/T177195 [4] https://phabricator.wikimedia.org/T181627 [5] https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=1 [6] https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles-promet… -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST

5 years, 11 months

Discovery Weekly Update for the week starting 2017-12-04

by Chris Koerner

Hello, This is the weekly update from the search team at the foundation for the week starting 2017-12-04. == Discussions== === Search === * Upgrading to ElasticSearch 5.5.x took a lot of smaller sections of work to be completed [0] ** Complete a Kibana security release [1] ** Upgrading Logstash cluster to elastic 5.5.x [2] ** Upgrading all log producers to use the Logstash LVS endpoint [3] * There was a HP RAID Battery issue on elastic2004 that has been resolved [4] * There was an issue with forceSearchIndex.php hanging at the end of the process when running on large wikis, so we disabled statsd collection instead of replacing statsd [5] === Portal === * Fixed an issue where the logo on www.wikipedia.org is misaligned for RTL languages on very small devices [6] * Automation is nearly done for the Wikipedia.org portal site [7] == FY 2017-18 Q2 (Oct-Dec) goals]== This status was last updated 2017-12-08. [8] [0] https://phabricator.wikimedia.org/T174662 [1] https://phabricator.wikimedia.org/T173685 [2] https://phabricator.wikimedia.org/T178412 [3] https://phabricator.wikimedia.org/T175242 [4] https://phabricator.wikimedia.org/T181412 [5] https://phabricator.wikimedia.org/T181716 [6] https://phabricator.wikimedia.org/T179273 [7] https://phabricator.wikimedia.org/T140159 [8] https://www.mediawiki.org/wiki/Discovery/Status_updates/2017-12-04#FY_2017-… --- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison Wikimedia Foundation

5 years, 12 months

OpenSearch Return Value

by Gheorghe Postelnicu

Hello, I am writing with a question regarding the OpenSearch API, which is super useful for extracting summary information about a page. For a lot of pages, the return items contain: - text - url - description - image e.g. https://en.wikipedia.org/w/api.php?action=opensearch&search=Angela%20Merkel… However, there are quite a few cases where the image property is missing, although the page features an image in the infobox. Example: https://en.wikipedia.org/w/api.php?action=opensearch&search=Solaris_(2002_f… In cases where the opensearch extension doesn't return an image, is there another API which allows extracting the infobox image if one is present? Many thanks in advance for the help, Gheorghe

5 years, 12 months

Commons Fulltext Search

by Gheorghe Postelnicu

Hello, First, many thanks for maintaining this API! I am writing as I am having trouble using the full text search API on Commons. Namely, I would like to get the following list back via the API: https://commons.wikimedia.org/w/index.php?title=Special:Search&profile=imag… however the full text search doesn't produce the same results when using the API: https://commons.wikimedia.org/w/api.php?action=query&list=search&srsearch=d… A suggestion to nudge me in the right direction would be greatly appreciated. Many thanks in advance for the help. Best regards, Gheorghe

6 years

Logstash upgrade coming up

by Guillaume Lederrey

Hello! Our logstash servers are the last servers to upgrade to elasticsearch 5.5.2. I'm planning to do this upgrade on Thursday December 7, between 9:00 and 10:00 UTC (10:00 to 11:00 CET, 01:00 to 02:00 PST). Testing on labs went well, so I'm not expecting any issue. Now that all servers are behind LVS, we should not even loose a single message. That being said, there is a risk. If you see disapearing log messages during that period, please let me know! Have fun! Guillaume -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST

6 years

Discovery Weekly Update for the week starting 2017-11-27

by Chris Koerner

Greetings, Here is the Discovery weekly update for 2017-11-27. As always, feedback and questions are welcome. == Discussions == === Search === * More work has been done on upgrading elasticsearch to 5.5.x [0] * We re-indexed English language wikis to incorporate the Katakana and Hiragana searches [1] * Issues were found where search.wikimedia.org was the source of lots of 500 errors, it's been resolved [2] * Indexing wikidata descriptions has been enabled (but still working on using that in the search) [3] [4] === Analysis === * Chelsy's analysis is complete for testing on 18 wikis with > 1% of search traffic [5] [6] [7] === Portal === * The name change from "Sámegiella" to "Davvisámegiella" has been pushed into production on Nov 20 [8] * The portal translations and stats have been updated — via our new automated process (it's still a work in process) [9] [10] [0] https://phabricator.wikimedia.org/T178411 [1] https://phabricator.wikimedia.org/T179945 [2] https://phabricator.wikimedia.org/T179266 [3] https://phabricator.wikimedia.org/T176903 [4] https://phabricator.wikimedia.org/T178851 [5] https://phabricator.wikimedia.org/T176493 [6] https://phabricator.wikimedia.org/T176493 [7] https://analytics.wikimedia.org/datasets/discovery/reports/Experiement_with… [8] https://phabricator.wikimedia.org/T178981 [9] https://phabricator.wikimedia.org/T128546 [10] https://phabricator.wikimedia.org/T142582 -- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison Wikimedia Foundation

6 years

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery December 2017