Discovery

discovery@lists.wikimedia.org

756 discussions

Elasticsearch sharding
by Guillaume Lederrey 16 Sep '16

16 Sep '16

Hello all! We had an interesting discussion yesterday with David about the way we do sharding of our indices on elasticsearch. Here are a few notes for whoever finds the subject interesting and wants to jump in the discussion: Context: We recently activated row aware shard allocation on our elasticsearch search clusters. This means that we now have one additional constraint on shard allocation: spread copies of shards across multiple datacenter rows, so that if we loose a full row, we still have a copy of all the data. During an upgrade of elasticsearch, another constraint comes into play: a shard can move from a node with an older version of elasticsearch to a node with a newer version, but not the other way around. This leads to elasticsearch struggling to allocate all shards during the recent codfw upgrade to elasticsearch 2.3.5. While it is not the end of the world (we can still server traffic if some indices don't have all shards allocated), this is something we need to improve. Number of shards / number of replicas: An elasticsearch index is split at creation in a number of shards. A number of replica per shard is configured [1]. The total number of shards for an index is "number_of_shards * (number_of_replicas + 1)". Increasing the number of shards per index allow to execute read operation in parallel over the different shards and aggregate the results at the end, improving response time Increasing the number of replicas allow to distribute the read load over more nodes (and provides some redundancy in case we loose one server). As term frequency [2] is calculated over a shard and not over the full index, There is some black magic involved in how we shard our indices, but most of it is documented [3] The enwiki_content example: enwiki_content index is configured to have 6 shards and 3 replicas, for a total number of 24 shards. It also has the additional constraint that there is at most 1 enwiki_content per node. This ensures a maximum spread of enwiki_content shards over the cluster. Since enwiki_content is one of the index with the most traffic, this ensure that the load is well distributed over the cluster. Now the bad news: for codfw, which is a 24 node cluster, it means that reaching this perfect equilibrium of 1 shard per node is a serious challenge if you take into account the other constraint in place. Even with relaxing the constraint to 2 enwiki shards per node, we have seen unassigned shards during elasticsearch upgrade. Potential improvements: While ensuring that a large index has a number of shards close to the number of nodes in the cluster allows for optimally spreading load over the cluster, it degrade fast if all the stars are not aligned perfectly. There are 2 opposite solutions 1) decrease the number of shards to leave some room to move them around 2) increase the number of shards and allow multiple shards of the same index to be allocated on the same node 1) is probably impractical on our large indices, enwiki_content shards are already ~30Gb and this makes it impractical to move them around during relocation and recovery 2) is probably our best bet. More smaller shards means that a single query load will be spread over more nodes, potentially improving response time. Increasing number of shards for enwiki_content from 6 to 20 (total shards = 80) means we have 80 / 24 = 3.3 shards per node. Removing the 1 shards per node constraint and letting elasticsearch spread the shards as best as it can means that in case 1 node is missing, or during an upgrade, we still have the ability to move shards around. Increasing this number even more might help keep the load evenly spread across the cluster (the difference between 8 or 9 shards per node is smaller than the difference between 3 or 4 shards per node). David is going to do some tests to validate that those smaller shards don't impact the scoring (smaller shards mean worse frequency analysis). I probably forgot a few points, but this email is more than long enough already... Thanks to all of you who kept reading until the end! MrG [1] https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_conc… [2] https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.… [3] https://wikitech.wikimedia.org/wiki/Search#Estimating_the_number_of_shards_… -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST

3 2

Weekly update for the week starting 2016-09-05
by Deborah Tankersley 12 Sep '16

12 Sep '16

Hello, Here is the week's update from the Discovery department - enjoy the read and your weekend! == Discussions == * Trey completed the analysis for optimizing language identification for the Dutch Wikipedia (nlwiki). The results were good (F0.5 = 82.3%) but not great. The small proportions of queries in the Romance languages and in German led to many more false positives than true positives and so they had to be excluded. Future work on improving confidence may help. [1] * We could use help translating (via translatewiki) the relevant "showing results from" messages into Dutch. We'll need English, Chinese, Arabic, Korean, Greek, Hebrew, Japanese, and Russian translations. [2] * The Analysis team had a discussion on how to use better wording for phrases like "users were 1.07 times more likely to do X" and decided on using phrases similar to "we can expect 2-9 more sessions to click on a search result when they have the new feature" [3] * The Search team wrapped up research into the ElasticSearch instabilities on the eqiad search cluster that occurred on Aug 6, 2016; nothing conclusive was found. [4] == Events and News == === Interactive === * <maplink> has been enabled on all wikis (announced via email to wikitech-l) [5] * Geoshapes data service is now integrated into all maps [6] === Search === * Turned off BM25 A/B test, awaiting analysis [7] * Pushed into production a change that implemented ascii-folding for French [8] * Improved balance of nodes across rows for ElasticSearch eqiad cluster [9] === Portal === * Currently blocked on this check-in to gerrit [10] == Other Noteworthy Stuff' == * Our elasticsearch clusters now have "row aware shard allocation". This means that we can theoretically lose one row of servers in our datacenter and still serve search traffic. [11] * The Search team sent out a request for comment article that was posted to various Village Pumps asking for it to be translated. [12] ** This was in reference to the cross-wiki search results new functionality and design articles on MediaWiki. [13], [14] == Did you know? == * A study came out yesterday showing that giraffes are actually four distinct species, rather than one (article and BBC report). [15], [16] ** Of course, the English and German Wikipedia pages on giraffes have already been updated! [17], [18] [1] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/TextCat_Optimization… [2] https://phabricator.wikimedia.org/T143354 [3] https://phabricator.wikimedia.org/T140187 [4] https://phabricator.wikimedia.org/T142506 [5] https://lists.wikimedia.org/pipermail/wikitech-l/2016-September/086490.html [6] https://www.mediawiki.org/wiki/Help:Extension:Kartographer#GeoShapes_extern… [7] https://phabricator.wikimedia.org/T143588 [8] https://phabricator.wikimedia.org/T144429 [9] https://phabricator.wikimedia.org/T143685 [10] https://gerrit.wikimedia.org/r/#/c/306241/ [11] https://phabricator.wikimedia.org/T143571 [12] https://meta.wikimedia.org/wiki/User:DTankersley_(WMF)/translation_request_… [13] https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements [14] https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/Design [15] http://www.cell.com/current-biology/fulltext/S0960-9822(16)30787-4 [16] http://www.bbc.com/news/science-environment-37311716 [17] https://en.wikipedia.org/wiki/Giraffe [18] https://de.wikipedia.org/wiki/Giraffe ---- The full update, and archive of past updates, can be found on Mediawiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as Easy or volunteer needed in Phabricator: [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Cheers! -- Deb Tankersley Product Manager, Discovery IRC: debt Wikimedia Foundation

3 3

Discernatron lunch - 12pm (SF time) Tue 13th Sep
by Dan Garry 10 Sep '16

10 Sep '16

The Search Team in Discovery needs your help! Discernatron [1] is a search relevance tool developed by the Discovery department. Its goal is to help improve search relevance - showing articles that are most relevant to search queries - with human assistance. We need your help grading search results! Join us for lunch at 12pm (SF time) in the 5th floor lounge on Tuesday 13th September! In the Discernatron lunch, we'll give a brief overview of what the Discernatron is, then ask people to get rating queries, so bring your laptops! We're hoping a limited amount of food will be provided for the event, but you can only eat it if you agree to rate queries for us. ;-) We'll also be set up for remote participation on Hangouts and IRC, and the session will be recorded. Hangout: https://hangouts.google.com/hangouts/_/7pcv3gtfcbczzhaxbezyqhedfee YouTube stream: https://www.youtube.com/watch?v=q4W9t6IcjWk Thanks! If there are any questions, let me know! Dan [1]: https://www.mediawiki.org/wiki/Discernatron -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation

2 1

Multimedia search on Commons
by Pine W 07 Sep '16

07 Sep '16

Hi Discovery, I'm wondering if there are any significant improvements coming in the next +/- 18 months for multimedia search on Commons. Finding images can be a very time-consuming job for Wikimedians and other users of the site. For example, it would be nice to be able to do a "join" search with categories, so that only media files that appear in 2+ selected categories are shown in search results. As an example, suppose that I want an image of a blue tile roof in China. The search "blue tile roof China" doesn't show any results that interest me on the first page. However, https://commons.wikimedia.org/wiki/File:Wuhan_University_-_roof_tiles.JPG is an image that would interest me. That file is several layers deep in subcategories under the China category. It would be nice to be able to find that image by searching a join of the China category and its subcategories, with the "blue roof" category. Thanks, Pine

2 1

New functionality: cross-wiki search results
by Deborah Tankersley 07 Sep '16

07 Sep '16

The Discovery Search Team wants to enable search results on Wikipedia that will include articles gathered across all sister wiki projects – within the same language – but we need your feedback. Please read t he specifics of how this new functionality <https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements> [1] might work and add comments, concerns, or alternative ideas for design options <https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/Design> [2] on the talk pages . See an image <https://www.mediawiki.org/wiki/File:Search_results_page-enwiki_right-hand-b…> [3] that shows one of many example display options that have been mocked up, after considering what other wiki communities have done. Thank you for your time and c heers from the Search Team! [1] https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements [2] https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/Design [3] *https://www.mediawiki.org/wiki/File:Search_results_page-enwiki_right-hand-box-general-projects.png <https://www.mediawiki.org/wiki/File:Search_results_page-enwiki_right-hand-b…>* -- Deb Tankersley Product Manager, Discovery IRC: debt Wikimedia Foundation

4 6

Discovery Weekly Update for the week starting 2016-08-29
by Chris Koerner 03 Sep '16

03 Sep '16

Hello, It's Friday where I'm at. I hope you have a good weekend. Here is this week's update from the Discovery department. == Analysis == * add a 'most commonly clicked section per visit' metric on Portal dashboard [1] [2] == Search == * a small subsection of the team met to go over the upcoming Discernatron demo / luncheon and how we can improve the user interface for it [3] * turned on BM25 A/B test after re-indexing the codfw seach cluster first [4] [5] * worked on updating the prose and user flow of the Discernatron introduction and instructions page [6] [3] * added patch to ensure timeouts always work on CirrusSearch [7] * evaluated BM25 with new fulltext query and weighted sum with PaulScore and Discernatron [8] * added patch to properly handle acronyms in fulltext query builder [9] * tested and analyzed the effect of adding ascii-folding to fr.wiki [10] * Use top_terms_blended_freq for wildcard queries [11] * updated fulltext query builder to enhance title [12] * turned off 'did you mean' on Wikidata [13] == Maps == * We tried to enable <maplink> on every Wikipedia, but ran into some blockers. Will try again next week. Track it at phab:T144062 [14] [1] https://phabricator.wikimedia.org/T141061 [2] http://discovery.wmflabs.org/portal/#most_common [3] https://discernatron.wmflabs.org/ [4] https://phabricator.wikimedia.org/T143586 [5] https://phabricator.wikimedia.org/T143590 [6] https://phabricator.wikimedia.org/T133359 [7] https://phabricator.wikimedia.org/T143006 [8] https://phabricator.wikimedia.org/T139579 [9] https://phabricator.wikimedia.org/T143541 [10] https://phabricator.wikimedia.org/T142620 [11] https://phabricator.wikimedia.org/T143537 [12] https://phabricator.wikimedia.org/T143537 [13] https://phabricator.wikimedia.org/T143260 [14] https://phabricator.wikimedia.org/T144062 ---- The full update, and archive of past updates, can be found on Mediawiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as Easy or volunteer needed in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Training bias in machine learning
by Pine W 01 Sep '16

01 Sep '16

Perhaps of interest: http://www.seattletimes.com/business/microsoft/how-linkedins-search-engine-… "“Histories of discrimination can live on in digital platforms,” Kate Crawford, a Microsoft researcher, wrote <http://www.nytimes.com/2016/06/26/opinion/sunday/artificial-intelligences-w…> in The New York Times earlier this year. “And if they go unquestioned, they become part of the logic of everyday algorithmic systems.”" Pine

1 0

Re: [discovery] [Translators-l] Translations requested: "showing results from"
by Deborah Tankersley 29 Aug '16

29 Aug '16

Thanks, Takashi! :) -- Deb Tankersley Product Manager, Discovery IRC: debt Wikimedia Foundation On Fri, Aug 26, 2016 at 8:29 PM, Takashi OTA < supertakot+translators(a)gmail.com> wrote: > HI Deborah, > > Done for Japanese messages. > Thanks. > > --Takashi [[U:Takot]] > > 2016年8月26日(金) 13:23 Takashi OTA <supertakot+translators(a)gmail.com>: > >> Hi Deborah, >> >> I'll take on JA this weekend. >> Please wait for a while. >> >> Cheers, >> >> --Takot >> >> On Tue, Aug 23, 2016 at 05:57 Deborah Tankersley < >> dtankersley(a)wikimedia.org> wrote: >> >>> Hi Stryn, >>> >>> We really only need these languages to be translated for: >>> >>> Portuguese: PT, EN, RU, HE, AR, ZH, KO, EL >>> Japanese: JA, EN, RU, KO, AR, HE >>> >>> >>> As those are the only languages that we're looking for on the PT and JA >>> wiki sites in order to determine if the user typed in one of those >>> languages. We didn't enable the feature for ALL languages. >>> >>> Hope that helps! >>> >>> Cheers, >>> >>> Deb >>> >>> >>> -- >>> Deb Tankersley >>> Product Manager, Discovery >>> IRC: debt >>> Wikimedia Foundation >>> >>> On Mon, Aug 22, 2016 at 8:23 AM, Stryn <strynwiki(a)gmail.com> wrote: >>> >>>> Hmm, there is really no easier way to translate those? >>>> >>>> Showing results from [[:als:Main_Page|Alemannic Wikipedia]]. >>>> Showing results from [[:cbk-zam:Main_Page|Chavacano Wikipedia]] >>>> >>>> There are many languages where I don't know the Finnish translation, so >>>> I would need to Google them first. >>>> >>>> Do we have system messages for language names? Like "Chavacano"? >>>> Then if do, would only need to translate the "showing results from" >>>> part and the rest could come from the system message? Or is it too complex? >>>> >>>> ... Ok I just found it, we have messages "MediaWiki:Project-localized- >>>> name-akwiki/fi", can't we merge in some way those messages? >>>> Now we have messages for "Wikimedia Project Names" and "Wikimedia >>>> Interwiki Search Results". I mean it just makes so much work for >>>> translators to translate those both. >>>> We can't use directly the messages from the "Wikimedia Project Names" >>>> groups for interwiki search texts because of conjugates, but something >>>> like "Showing results from: {{MediaWiki:Project-localized- >>>> name-akwiki/fi}}" would be nice. >>>> >>>> *Stryn* >>>> >>>> *Suomenkielisen Wikipedian ylläpitäjä & osoitepaljastaja / Admin and >>>> checkuser on the Finnish Wikipedia* >>>> >>>> *Wikidatan ylläpitäjä / Admin on Wikidata* >>>> >>>> *Meta-Wikin ylläpitäjä / Admin on Meta-Wiki* >>>> *Ylivalvoja / Steward* >>>> >>>> 2016-08-18 22:45 GMT+03:00 Deborah Tankersley < >>>> dtankersley(a)wikimedia.org>: >>>> >>>>> >>>>> The Discovery team has been making good progress in enabling cross >>>>> language search results on several wiki's and now we need help in >>>>> translating a phrase: "*showing results from"*. >>>>> >>>>> >>>> _______________________________________________ >>>> Translators-l mailing list >>>> Translators-l(a)lists.wikimedia.org >>>> https://lists.wikimedia.org/mailman/listinfo/translators-l >>>> >>>> >>> _______________________________________________ >>> Translators-l mailing list >>> Translators-l(a)lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/translators-l >>> >> > _______________________________________________ > Translators-l mailing list > Translators-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/translators-l > >

1 0

Discovery Weekly Update for the week starting 2016-08-22
by Chris Koerner 27 Aug '16

27 Aug '16

Happy Friday, Here is this week's update from the Discovery department. It's a big one, so grab your favorite drink and settle in. == Events and News == * Interactive team plans to deploy <maplink> support to all Wikipedia next week. T144062 [1] == Other Noteworthy Stuff == * Updated the Wikipedia.org Portal page with new information on quarterly goals and recent production releases [2] [3] * Updated the Help:CirrusSearch documentation to reflect the great work Cspiral did in organizing the information in a more approachable and logical way. Kept existing translation tags in place and have asked for review. This is in preparation for this small fix to the "Help" link on every * * * * * Wikimedia Special:Search page. [4] [5] * Analysis: completed a retro of how the wikipedia.org portal page has evolved [6] * Analysis: completed a review of why portal pageviews increased in June/July 2016 (phab ticket) [7] [8] * Analysis: updated pageview counting on the portal dashboard (dashboard page showing new documented changes) [9] [10] * Analysis: added an easy to use page to show metrics year over year and month over month (dashboard page) [11] [12] * Analysis: added a portal dashboard page to show clicks/users by language (dashboard page) [13] [14] * Analysis: updated dashboard to take into account a new schema for mobile app events (dashboard page) [15] [16] * Analysis: added DuckDuckGo to the search engine referral dashboard page (dashboard page) [17] [18] * Portal: added several new small visual enhancements and optimizations for wikipedia.org along with an article stats and translations updates for code review [19] * Portal: Clean up project portals (community involvement) [20] * Search: deployed ascii folding / stemming re-ordering changes [21] * Search: disable an old search schema since a new one has already been deployed [22] * Search: implement an updated full text search query [23] * Search: Refactor GeoData elasticsearch mapping using the new search field definitions from core [24] * Search: update wildcard queries for BM25 [25] * Search: Centralize document id and namespace generation [26] * Search: Enable BM25 by default in cirrus and evaluate impact [27] * Search: Add a "reverse" suggestion field [28] * Interactive: Large number of small maps usability fixes T144008, T143883, T137357, T143179, T144001 [29] [30] [31] [32] [33] * Interactive: Added a box frame and captioning to <mapframe> (similar to framed images) [34] == Did you know? == * Friday, August 26th is National Dog Day - go pet a pooch! [35] [1] https://phabricator.wikimedia.org/T144062 [2] https://www.mediawiki.org/wiki/Wikipedia.org_Portal#Current_Quarter_Goals [3] https://www.mediawiki.org/wiki/Wikipedia.org_Portal#Portal_Updates:_April_-… [4] https://www.mediawiki.org/wiki/Help:CirrusSearch [5] https://phabricator.wikimedia.org/T141003 [6] https://commons.wikimedia.org/wiki/File:Impact_of_Wikipedia.org_Portal_Chan… [7] https://github.com/wikimedia-research/Discovery-Research-Portal/blob/master… [8] https://phabricator.wikimedia.org/T143045 [9] https://phabricator.wikimedia.org/T143064 [10] http://discovery-beta.wmflabs.org/portal/#pageview_tab [11] https://phabricator.wikimedia.org/T124098 [12] http://discovery.wmflabs.org/metrics/#monthly_metrics [13] https://phabricator.wikimedia.org/T140816 [14] http://discovery.wmflabs.org/portal/#languages_visited [15] https://phabricator.wikimedia.org/T143447 [16] http://discovery.wmflabs.org/metrics/#app_events [17] https://phabricator.wikimedia.org/T143287 [18] http://discovery.wmflabs.org/external/#traffic_by_engine [19] https://gerrit.wikimedia.org/r/306241 [20] https://phabricator.wikimedia.org/T135929 [21] https://phabricator.wikimedia.org/T142721 [22] https://phabricator.wikimedia.org/T130321 [23] https://phabricator.wikimedia.org/T128073 [24] https://phabricator.wikimedia.org/T139021 [25] https://phabricator.wikimedia.org/T143537 [26] https://phabricator.wikimedia.org/T139496 [27] https://phabricator.wikimedia.org/T139576 [28] https://phabricator.wikimedia.org/T107006 [29] https://phabricator.wikimedia.org/T144008 [30] https://phabricator.wikimedia.org/T143883 [31] https://phabricator.wikimedia.org/T137357 [32] https://phabricator.wikimedia.org/T143179 [33] https://phabricator.wikimedia.org/T144001 [34] https://phabricator.wikimedia.org/T128913 [35] https://www.google.com/#hl=en&q=national+dog+day+images ---- The full update, and archive of past updates, can be found on Mediawiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as Easy or volunteer needed in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Re: [discovery] Phab autocomplete (was: FYI: Minor change)
by Kevin Smith 26 Aug '16

26 Aug '16

The phab folks so far have resisted really fixing autocomplete. And they won't accept patches that don't align with their own product vision. Thus, at least so far, the answer has been "you can't get there from here." Kevin Smith Agile Coach, Wikimedia Foundation On Fri, Aug 26, 2016 at 9:43 AM, Jonathan Morgan <jmorgan(a)wikimedia.org> wrote: > Grrr. > > https://phabricator.wikimedia.org/T99739 > > Hey who can we Barnstar to get this fixed? > > J > >

2 1

← Newer
1
...
33
34
35
36
37
38
39
...
76
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery