Discovery July 2017

discovery@lists.wikimedia.org

6 participants
6 discussions

Puppetized Discovery Dashboards and Shiny Server module for Puppet

by Mikhail Popov

Howdy, Happy to report that production[1] and development[2] sets of Discovery Dashboards are up and running again, this time managed by Puppet. (There was a bug with web proxies and DNS settings that delayed this announcement.) Theoretically they should be snappier to use now because there is no longer an extra virtualization (Vagrant) layer and they are running directly on Labs instances. R is a software and programming language mainly used for statistical inference, machine learning, and data wrangling & visualization. RStudio's Shiny[3] is a framework for developing web applications in R, and it's what Discovery's dashboards are written in. The Reading::Discovery::Analysis team (with guidance and help from Guillaume Lederrey) is proud to announce a new module available in Ops' Puppet repo: shiny_server[4], which installs & configures RStudio's Shiny Server[5] for serving R/Shiny applications. The module also provides resources for installing R packages from CRAN, GitHub, and other remote git repositories like Gerrit. For a practical example, refer to Discovery Dashboards base[6] and production[7] profiles. Cheers, Mikhail on behalf of Discovery Analysts [1] https://discovery.wmflabs.org [2] https://discovery-beta.wmflabs.org/ [3] https://shiny.rstudio.com/ [4] https://github.com/wikimedia/puppet/tree/production/modules/shiny_server [5] https://www.rstudio.com/products/shiny/shiny-server/ [6] https://github.com/wikimedia/puppet/blob/production/modules/profile/manifes… [7] https://github.com/wikimedia/puppet/blob/production/modules/profile/manifes…

5 years, 7 months

Discovery Weekly Update for the week starting 2017-07-24

by Chris Koerner

Greetings, This is the weekly update for Discovery. As always, feedback and questions are welcome. == Discussions == === Search === * Erik's bugfixes: T171009 (don't issue a near match query with empty query), T171155 (match redirects with the ''intitle'' keyword). [0] [1] * Stas' Wikidata entity search with Elastic is in testing [2] [3] * Stas' work on the archive search is in final call, to be enabled next week. [4] * David worked on collecting features from the sltr query by using a custom Accumulator to collect feature names and extracting feature vectors from the ltr plugin [5] * Trey's language analysis analysis tools will soon be released in RelForge. See phab ticket T171516. [6] Comments on the extensive/excessive documentation copied to MediaWiki are welcome. [7] === Analysis === * Chelsy found a bug in the ''renderMenu'' function in ''shinydashboard'' package that was fixed—but we need to wait for latest version to be uploaded to CRAN before we can patch our instance [8] === Interactive === * Mapframe was enabled for cswiki by a volunteer contributor. [0] [0] https://phabricator.wikimedia.org/T171009 [1] https://phabricator.wikimedia.org/T171155 [2] https://www.wikidata.org/wiki/Wikidata:Project_chat#wbsearchentities_with_E… [3] http://elastic-wikidata.wmflabs.org/wb.html [4] https://phabricator.wikimedia.org/T163235 [5] https://phabricator.wikimedia.org/T168813 [6] https://phabricator.wikimedia.org/T171516 [7] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Analysis_Analysis_To… [8] https://phabricator.wikimedia.org/T170724 [9] https://phabricator.wikimedia.org/T171588 -- The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison Wikimedia Foundation

5 years, 7 months

Discovery Weekly Update for the week starting 2017-07-17

by Chris Koerner

Hey all, This is the Discovery weekly update for the week of 2017-07-17 _including_ updates from the week of 2017-07-10 because I'm a dummy and forgot to send the update email last week! Feedback and questions are welcome. ==Highlights== * Quarterly Review check-in's were presented this week (links forthcoming). ==Discussions== * Had a good conversation on how we want to do an upcoming A/B test to have search results interleaved [0] * Also had a really interesting conversation on how we should move forward with metrics and KPIs for the Search Platform and Discovery. === Search === * After extensive help and feedback from the Japanese Wikipedia community, we decided ''not'' to deploy the new Japanese language analyzer. [1] * After testing and analysis, we are ''not'' going to deploy the Vietnamese language analyzer [2] * Did some quick analysis of the RelForge impact of initial learning-to-rank model [3] * Completed a high level task of organizing necessary adjustments to the ElasticsSearch learning-to-rank plugin [4] * Updated the Wikidata code to take advantage of nested fields noop script [5] * Analyzing the new Kuromoji Japanese language analyzer is done, just waiting to merge the updated configuration [6] * A/B test for the new search results Explore Similar feature continues after fixing a minor bug [7] [8] * Fixed a mapping bug when using text for 'did you mean' suggestions [9] * Updated the 'cirrussearch-explore-similar-languages-none' message to be more meaningful (will be deployed the week of July 25, 2017) [10] === Analysis === * To track sister project search feature, the Search Metrics dashboard now shows traffic to various Wikimedia projects from Wikipedia search results pages [11] [12] * We found what lowered full-text search PaulScores – it was a change of event logging sampling rates (decreasing enwiki and increasing all other wikis). [13] [14] * Investigated the seemingly odd statistic of mobile web having the highest usage of the sister project snippets in the search results page; conclusion: nothing to worry about, everything is being logged correctly. [15] === Portal === * The Wikipedia portal stats and translations were updated on July 18, 2017 [16] [17] === Maps === * A usage policy was recently changed for the relief data layer that we use in maps to now require an API key; a key was procured and Max S helped us get it into production. This only effected maps on Wikivoyage. [18] [0] https://phabricator.wikimedia.org/T150032 [1] https://phabricator.wikimedia.org/T166731 [2] https://phabricator.wikimedia.org/T170423 [3] https://phabricator.wikimedia.org/T169103 [4] https://phabricator.wikimedia.org/T162062 [5] https://phabricator.wikimedia.org/T166589 [6] https://phabricator.wikimedia.org/T166731#3439190 [7] https://phabricator.wikimedia.org/T149809 [8] https://gerrit.wikimedia.org/r/364834 [9] https://phabricator.wikimedia.org/T155489 [10] https://phabricator.wikimedia.org/T169302 [11] https://discovery.wmflabs.org/metrics/ [12] https://discovery.wmflabs.org/metrics/#sister_search_traffic [13] https://discovery.wmflabs.org/metrics/#paulscore_approx [14] https://phabricator.wikimedia.org/T168466 [15] https://phabricator.wikimedia.org/T170183 [16] https://phabricator.wikimedia.org/T128546 [17] https://phabricator.wikimedia.org/T142582 [18] https://phabricator.wikimedia.org/T170976 --- The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison Wikimedia Foundation

5 years, 8 months

Wired article "Search Algorithms Kept Me From My Sister for 14 Years"

by Chris Koerner

Thanks to Erica Litrenta for sharing this with me. I thought I'd share if forward. "It was because of the letter K that I found my younger sister, but for 14 years, it was also the letter K that kept us apart." https://www.wired.com/story/search-algorithms-kept-me-from-my-sister-for-14… Yours, Chris Koerner Community Liaison Wikimedia Foundation

5 years, 8 months

Discovery Weekly Update for the week starting 2017-07-03

by Chris Koerner

Hello, This is the Discovery update for last week. Apologies for the delay in getting it out. == Discussions == === Search === * Created a method for the Kafka consumer to take 'learn to rank' queries from a queue and run them against ElasticSearch to generate relevance labels [0] * Added in the ability to use kafka in our LTRank feature generation queries and pushing them into ElasticSearch for analysis [1] * Added ability to extract TF and IDF based features in the ElasticSearch 'learning to rank' plugin [2] * A/B test still in progress 'explore similar' links, but we're running into a few bugs that will be sorted out next week [3] * Fixed a bug where searching for phrase queries did not highlight page content [4] === Analysis === * Fixed a bug with the sister project snippets and eventlogging [5] * Finished up analysis for determining what is a reasonable per-IP ratelimit for maps [6] * Fixed a minor dashboard bug (splines) [7] [0] https://phabricator.wikimedia.org/T162059 [1] https://phabricator.wikimedia.org/T162072 [2] https://phabricator.wikimedia.org/T167437 [3] https://phabricator.wikimedia.org/T164856 [4] https://phabricator.wikimedia.org/T167798 [5] https://phabricator.wikimedia.org/T168916 [6] https://phabricator.wikimedia.org/T169175 [7] https://phabricator.wikimedia.org/T169125 Yours, Chris Koerner Community Liaison Wikimedia Foundation

5 years, 8 months

Re: [discovery] Wired article "Search Algorithms Kept Me From My Sister for 14 Years"

by Øystein Håberg

Very interesting! 10. jul. 2017 16:34 skrev "Chris Koerner" <ckoerner(a)wikimedia.org>: Thanks to Erica Litrenta for sharing this with me. I thought I'd share if forward. "It was because of the letter K that I found my younger sister, but for 14 years, it was also the letter K that kept us apart." https://www.wired.com/story/search-algorithms-kept-me- from-my-sister-for-14-years Yours, Chris Koerner Community Liaison Wikimedia Foundation _______________________________________________ discovery mailing list discovery(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery

5 years, 8 months

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery July 2017