Discovery October 2018

discovery@lists.wikimedia.org

10 participants
8 discussions

Upcoming Search Platform Office Hours—November 7th

by Trey Jones

The Search Platform Team <https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> will be holding office hours the first Wednesday of each month, starting next week. Come ask us anything about Wikimedia search! We’re particularly interested in: * Opportunities for collaboration—internally or externally to the Wikimedia Foundation * Challenges you have with on-wiki search, in any of the languages we support But we're happy to talk about anything search-related. Details for our next meeting: Date: Wednesday, November 7th, 2018 Time: 16:00 GMT / 08:00 PST / 11:00 EST / 17:00 CET Google Meet link: https://meet.google.com/vyc-jvgq-dww *N.B.:* Google Meet System Requirements <https://support.google.com/meet/answer/7317473> Hope to see you there! —Trey Trey Jones Sr. Software Engineer, Search Platform Wikimedia Foundation

5 years, 5 months

Fwd: [MediaWiki-l] New episode of Between the Brackets: "BTB Digest"

by Deb Tankersley

Reposting here! :) Cheers, Deb -- deb tankersley program manager, engineering Wikimedia Foundation ---------- Forwarded message --------- From: Yaron Koren <yaron57(a)gmail.com> Date: Tue, Oct 30, 2018 at 2:57 PM Subject: [MediaWiki-l] New episode of Between the Brackets: "BTB Digest" To: MediaWiki announcements and site admin list < mediawiki-l(a)lists.wikimedia.org> Hi, The latest episode of the MediaWiki podcast Between the Brackets has been released. This one just contains some highlights from previous episodes, so you can get a condensed version of five recent episodes (featuring Stas Malyshev, Remco De Boer, Dustin Phillips, Nikhil Kumar & Yashdeep Thorat, and Bartosz Dziewonski) in just 20 minutes or so. You can listen to the episode here: http://betweenthebrackets.libsyn.com/episode-20-btb-digest-3 -Yaron _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

5 years, 5 months

Supporting researchers

by Erik Bernhardson

While at the conference this week I met up with an professor and he had an interesting proposal. Essentially the idea would be they could build containers (following our OSS requirements) that we could deploy, and for some small percent of search traffic pass our top 100 results to their container for reranking. Certainly more details would have to be worked out, but does this seem reasonable? Overall I still think working directly with academics can be beneficial to our own work, although I'm not sure what our overhead would be.

5 years, 5 months

Playing with SonarCloud

by Guillaume Lederrey

Hello! I took some time to upload a first analysis of our Java based projects to SonarCloud [1]. This is just experimentation at this point, but there are a few interesting findings that should be corrected. What's missing: * run the analysis regularly, so that we can see trends * host it on our own infrastructure instead of depending on SonarCloud * some discussion on whether we agree with the metrics / issues * lot more Have fun! Guillaume [1] https://sonarcloud.io/organizations/wmftest/projects -- Guillaume Lederrey Operations Engineer, Search Platform Wikimedia Foundation UTC+2 / CEST

5 years, 6 months

BlazeGraph + Elasticsearch?

by Trey Jones

Hi Everyone, I'm at WikiConference NA today, and I was chatting with someone from OCLC <https://en.wikipedia.org/wiki/OCLC>, and he mentioned that BlazeGraph can be configured to call out to a full-text search engine. It looks like it only works with SOLR out of the box, but the documentation <https://wiki.blazegraph.com/wiki/index.php/ExternalFullTextSearch> mentions that Elasticsearch is a candidate search endpoint. Obviously it wouldn't be worth doing any real work on investigating this until the BlazeGraph/Amazon situation is clearer, and maybe Stas or others have looked at it in the past and already know why it isn't worth the added complexity, but there are some interesting use cases where combining full text and SPARQL would be useful—for example if you are looking for a person, you know part of their name, and some facts about them. In general, any full-text search with additional structured data constraints. Anyone already know anything about the capacity of BlazeGraph? Thanks, —Trey Trey Jones Sr. Software Engineer, Search Platform Wikimedia Foundation

5 years, 6 months

Discovery Weekly Update for the week starting 2018-10-15

by Chris Koerner

This is the weekly update from the Search Platform team for the week starting 2018-10-15. As always, feedback and questions welcome. == Discussions == === Search === * Trey completed his analysis of the Nori language analyzer for Korean. [0] With appropriate customizations, it should improve search on Korean-language wikis, though we won't be able to use it until we upgrade to Elasticsearch 6.4.2. Thanks to Baha and Revi for their help reviewing the output of the language analyzer. [1] * Mathew and Gehel worked on reconfigure=ing Icinga alert for elasticsearch_shard_size to reduce false positive alerts [2], as well as working on refactoring to 'use_git_deploy' in wdqs puppet module to cater for scap3 and autodeployment modes [3] * Erik worked on repairing some search satisfaction browser tests that were no longer automagically running [4] === WIkidata Query Service === * Query Lexemes are now in production for the Wikidata Query Service that entailed a large amount of work and dedication from Stas [5] * Mathew and Gehel created an auto-deployment test server for WDQS [6] == Did you know? == * The writing system for Korean, called Hangeul or Hangul, [7] consists of 24 characters (14 consonants and 10 vowels), which can be combined into 11,172 modern syllable blocks. Each of the modern syllable blocks has its own Unicode character, while obsolete syllable blocks can be composed of individual characters (if you have a font that is smart enough to know what to do [8]). Hangeul thus combines the features of an alphabet and a syllabary, and has been called a "syllabic alphabet" and an "alphabetic syllabary". [0] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Nori_Analyzer_Analys… [1] https://phabricator.wikimedia.org/T178925 [2] https://phabricator.wikimedia.org/T206187 [3] https://phabricator.wikimedia.org/T206597 [4] https://phabricator.wikimedia.org/T206232 [5] https://lists.wikimedia.org/pipermail/wikidata//2018-October/012545.html [6] https://phabricator.wikimedia.org/T197187 [7] https://en.wikipedia.org/wiki/Hangul [8] https://commons.wikimedia.org/wiki/File:Korean_Composition_Example.png ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Relations Specialist Wikimedia Foundation

5 years, 6 months

Discovery Weekly Update for the week starting 2018-10-01

by Chris Koerner

Hello again. It's been a while. This is the weekly update from the Search Platform team for the week starting 2018-09-17 through 2018-10-01. As always, feedback and questions welcome. == Discussions == === Search === * Implemented indexing statement values as part of main data in Wikidata, so that statement values are now searchable without special syntax [0] * Reindexed wikidata which also enables qualifier indexing [1] * Mathew worked on resolving an elasticsearch shard size alert by doing an in place reindex [2] * There was a lot of work done to investigate a brief outage of CirrusSearch (mw exception spike for api.php) [3], but it's resolved enough for now. * Gehel and others worked on refactoring puppet to support multiple elasticsearch instances on same node [4] * Erik worked on an issue where the text content of wiki page in search index can merge words making them unfindable [5] * Stas updated the search engine of Wikidata to enable searching by author name string [6] * David and Erik worked together on evaluating adding an image quality score to media search result ranking [7] * Stas added X-Search-Id to WikidataCompletionSearchClicks events [8] * David added a way to configure timeouts of autocomplete queries [9] * Erik upgraded saneitizer to constantly re-index documents [10] * David investigated why interwiki cache hit/miss was no longer reported (since 2017) and decided to drop the support for caching interwiki queries [11] * Mathew and Gehel worked on raising the alert level on disk space for old elasticsearch servers [12] * Erik worked to correct issues where the Cirrus MLT cache had a 0% hit rate on switchover [13] === WDQS === * Added new NTriples RDF dump (which makes it easier to do per-line processing) [14] * Internal cluster switched to Kafka events as change source, public cluster next [15] == Did you know? == * Different languages can have a different number of sounds they use; the set of sounds used in a particular language is called its “phonemic inventory”. [16] The numbers of sounds can range from 11 to over 140! Having more sounds than letters, or different sounds than the usual sound associated with a letter, can be the source of unusual orthographies and/or transliteration schemes—including "q" formerly being used as a vowel in Natqgu (now Natügu), a language of the Solomon Islands. [0] https://phabricator.wikimedia.org/T163642 [1] https://phabricator.wikimedia.org/T193407 [2] https://phabricator.wikimedia.org/T204362 [3] https://phabricator.wikimedia.org/T204776 [4] https://phabricator.wikimedia.org/T198351 [5] https://phabricator.wikimedia.org/T195389 [6] https://phabricator.wikimedia.org/T179815 [7] https://phabricator.wikimedia.org/T202339 [8] https://phabricator.wikimedia.org/T205597 [9] https://phabricator.wikimedia.org/T204959 [10] https://phabricator.wikimedia.org/T203622 [11] https://phabricator.wikimedia.org/T191961 [12] https://phabricator.wikimedia.org/T204361 [13] https://phabricator.wikimedia.org/T204148 [14] https://phabricator.wikimedia.org/T144103 [15] https://phabricator.wikimedia.org/T189458 [16] https://en.wikipedia.org/wiki/Phonemic_inventory ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Relations Specialist Wikimedia Foundation

5 years, 6 months

Upcoming Search Platform Office Hours—October 3rd

by Trey Jones

The Search Platform Team <https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> will be holding office hours the first Wednesday of each month, starting next week. Come ask us anything about Wikimedia search! We’re particularly interested in: * Opportunities for collaboration—internally or externally to the Wikimedia Foundation * Challenges you have with on-wiki search, in any of the languages we support But we're happy to talk about anything search-related. Details for our next meeting: Date: Wednesday, October 3rd, 2018 Time: 15:00 GMT / 08:00 PDT / 11:00 EDT / 17:00 CEST Google Meet link: https://meet.google.com/vyc-jvgq-dww *N.B.:* Google Meet System Requirements <https://support.google.com/meet/answer/7317473> See you there! —Trey Trey Jones Sr. Software Engineer, Search Platform Wikimedia Foundation

5 years, 6 months

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery October 2018