Discovery

discovery@lists.wikimedia.org

1 participants
755 discussions

Discovery Weekly Update for the week starting 2017-12-11
by Chris Koerner 18 Dec '17

18 Dec '17

Hello, This is the weekly update from the Search Platform team for the week starting 2017-12-11. *Note*: Because of the holidays the next weekly update will be sent out on 8 January 2018. See you in the new year! == Highlights == * We've officially closed out the Discovery Department wiki page (to include the portal work) and added a new Search Platform page. [0] [1] [2] * The automation of the Wikipedia.org Portal page is now complete. Going forward, as part of the automation process, two tickets will continue to be updated for the stats and translations weekly updates. More information on how the automation is done can be found in the documentation on diffusion. 🎁 🎉 [3] [4] [5] * "Gnomes and trolls and hobgoblins (oh my!)—Failed queries and the vicarious fear of missing out" ** Check out Trey's latest blog post! [6] == Discussions == === Search === * We closed out the large ticket that tracked a lot of the work that we've done so far for backend data engineering and plumbing for machine learning to rank [7] * We also closed out a ticket that fixed an issue with some features that were being hidden by the completion suggester such as sub phrase matching on wikisource. [8] * We've ported all our Selenium tests from Ruby to Node.js for the Search Platform team [9] * Trey finished up a long review of the Serbian Morphological Libraries that are available [10] [11] * We did some tuning to Wikidata prefix search [12] * Added ability for searches to return extra match data in API responses (Wikidata will use it) [13] === WDQS === * |Categories are now updated automatically into Blazegraph each week on Monday. [14] === Portal === * Jan finished up several small tasks related to the automation of the Wikipedia.org portal page [15]: ** Update technical documentation for portals repository [16] ** Fixed a minor UI problem with rendering the small print in English and Russian [17] == FY 2017-18 Q2 (Oct-Dec) goals == This status was last updated 2017-12-12 [18] [0] https://www.mediawiki.org/wiki/Wikimedia_Discovery [1] https://www.mediawiki.org/wiki/Wikipedia.org_Portal [2] https://www.mediawiki.org/wiki/Wikimedia_Search_Platform [3] https://phabricator.wikimedia.org/T128546 [4] https://phabricator.wikimedia.org/T142582 [5] https://phabricator.wikimedia.org/diffusion/WPOR/browse/master/README.md [6] https://blog.wikimedia.org/2017/12/12/failed-queries-fear-of-missing-out/ [7] https://phabricator.wikimedia.org/T162053 [8] https://phabricator.wikimedia.org/T178474 [9] https://phabricator.wikimedia.org/T178474 [10] https://phabricator.wikimedia.org/T178926 [11] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Serbian_Stemmer_Anal… [12] https://phabricator.wikimedia.org/T182136 [13] https://phabricator.wikimedia.org/T182145 [14] https://www.mediawiki.org/wiki/Wikidata_query_service/Categories [15] https://phabricator.wikimedia.org/T140159 [16] https://phabricator.wikimedia.org/T182119 [17] https://phabricator.wikimedia.org/T178744 [18] https://www.mediawiki.org/wiki/Discovery/Status_updates/2017-12-11#FY_2017-… --- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison Wikimedia Foundation

1 0

Elasticsearch metrics moving to Prometheus
by Guillaume Lederrey 14 Dec '17

14 Dec '17

Hello! We are moving all our metrics from Graphite [1] to Prometheus [2]. See the master task for some details [3]. We are now working on porting all those metrics for elasticsearch [4]. This means that we'll need to update our dashboards to start using those new metrics. During the transition, I'll keep the existing dashboards, but create a copy to migrate the metrics. For example, the "Elasticsearch Percentiles" [5] dashboard has a copy named "Elasticsearch Percentiles - Prometheus" [6]. Those new dashboards should behave in just the same way, but please let me know if you find any incoherence! Have fun! Guillaume [1] https://wikitech.wikimedia.org/wiki/Graphite [2] https://wikitech.wikimedia.org/wiki/Prometheus [3] https://phabricator.wikimedia.org/T177195 [4] https://phabricator.wikimedia.org/T181627 [5] https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=1 [6] https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles-promet… -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST

2 2

Discovery Weekly Update for the week starting 2017-12-04
by Chris Koerner 11 Dec '17

11 Dec '17

Hello, This is the weekly update from the search team at the foundation for the week starting 2017-12-04. == Discussions== === Search === * Upgrading to ElasticSearch 5.5.x took a lot of smaller sections of work to be completed [0] ** Complete a Kibana security release [1] ** Upgrading Logstash cluster to elastic 5.5.x [2] ** Upgrading all log producers to use the Logstash LVS endpoint [3] * There was a HP RAID Battery issue on elastic2004 that has been resolved [4] * There was an issue with forceSearchIndex.php hanging at the end of the process when running on large wikis, so we disabled statsd collection instead of replacing statsd [5] === Portal === * Fixed an issue where the logo on www.wikipedia.org is misaligned for RTL languages on very small devices [6] * Automation is nearly done for the Wikipedia.org portal site [7] == FY 2017-18 Q2 (Oct-Dec) goals]== This status was last updated 2017-12-08. [8] [0] https://phabricator.wikimedia.org/T174662 [1] https://phabricator.wikimedia.org/T173685 [2] https://phabricator.wikimedia.org/T178412 [3] https://phabricator.wikimedia.org/T175242 [4] https://phabricator.wikimedia.org/T181412 [5] https://phabricator.wikimedia.org/T181716 [6] https://phabricator.wikimedia.org/T179273 [7] https://phabricator.wikimedia.org/T140159 [8] https://www.mediawiki.org/wiki/Discovery/Status_updates/2017-12-04#FY_2017-… --- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison Wikimedia Foundation

1 0

OpenSearch Return Value
by Gheorghe Postelnicu 11 Dec '17

11 Dec '17

Hello, I am writing with a question regarding the OpenSearch API, which is super useful for extracting summary information about a page. For a lot of pages, the return items contain: - text - url - description - image e.g. https://en.wikipedia.org/w/api.php?action=opensearch&search=Angela%20Merkel… However, there are quite a few cases where the image property is missing, although the page features an image in the infobox. Example: https://en.wikipedia.org/w/api.php?action=opensearch&search=Solaris_(2002_f… In cases where the opensearch extension doesn't return an image, is there another API which allows extracting the infobox image if one is present? Many thanks in advance for the help, Gheorghe

2 1

Commons Fulltext Search
by Gheorghe Postelnicu 06 Dec '17

06 Dec '17

Hello, First, many thanks for maintaining this API! I am writing as I am having trouble using the full text search API on Commons. Namely, I would like to get the following list back via the API: https://commons.wikimedia.org/w/index.php?title=Special:Search&profile=imag… however the full text search doesn't produce the same results when using the API: https://commons.wikimedia.org/w/api.php?action=query&list=search&srsearch=d… A suggestion to nudge me in the right direction would be greatly appreciated. Many thanks in advance for the help. Best regards, Gheorghe

2 2

Logstash upgrade coming up
by Guillaume Lederrey 06 Dec '17

06 Dec '17

Hello! Our logstash servers are the last servers to upgrade to elasticsearch 5.5.2. I'm planning to do this upgrade on Thursday December 7, between 9:00 and 10:00 UTC (10:00 to 11:00 CET, 01:00 to 02:00 PST). Testing on labs went well, so I'm not expecting any issue. Now that all servers are behind LVS, we should not even loose a single message. That being said, there is a risk. If you see disapearing log messages during that period, please let me know! Have fun! Guillaume -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST

1 0

Discovery Weekly Update for the week starting 2017-11-27
by Chris Koerner 01 Dec '17

01 Dec '17

Greetings, Here is the Discovery weekly update for 2017-11-27. As always, feedback and questions are welcome. == Discussions == === Search === * More work has been done on upgrading elasticsearch to 5.5.x [0] * We re-indexed English language wikis to incorporate the Katakana and Hiragana searches [1] * Issues were found where search.wikimedia.org was the source of lots of 500 errors, it's been resolved [2] * Indexing wikidata descriptions has been enabled (but still working on using that in the search) [3] [4] === Analysis === * Chelsy's analysis is complete for testing on 18 wikis with > 1% of search traffic [5] [6] [7] === Portal === * The name change from "Sámegiella" to "Davvisámegiella" has been pushed into production on Nov 20 [8] * The portal translations and stats have been updated — via our new automated process (it's still a work in process) [9] [10] [0] https://phabricator.wikimedia.org/T178411 [1] https://phabricator.wikimedia.org/T179945 [2] https://phabricator.wikimedia.org/T179266 [3] https://phabricator.wikimedia.org/T176903 [4] https://phabricator.wikimedia.org/T178851 [5] https://phabricator.wikimedia.org/T176493 [6] https://phabricator.wikimedia.org/T176493 [7] https://analytics.wikimedia.org/datasets/discovery/reports/Experiement_with… [8] https://phabricator.wikimedia.org/T178981 [9] https://phabricator.wikimedia.org/T128546 [10] https://phabricator.wikimedia.org/T142582 -- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison Wikimedia Foundation

1 0

Vote in the Community Wishlist Survey (2017)
by Deborah Tankersley 28 Nov '17

28 Nov '17

Hello all, For your information, the voting phase of the WMF Community Wishlist Survey has begun and will close on December 10th, 2017. There are more than 200 proposals in several main categories (i.e.: Search); click on the categories to view the proposal's write-up and discussions. More information on the community wishlist survey is in the forwarded email below. -- deb tankersley Program Manager, Engineering Wikimedia Foundation ---------- Forwarded message ---------- From: Johan Jönsson <jjonsson(a)wikimedia.org> Date: Mon, Nov 27, 2017 at 11:01 PM Subject: [Wikimedia-l] You can now vote in the Community Wishlist Survey Hey everyone, The voting phase of the 2017 Community Wishlist Survey has now started. Read the proposals and support the ones you want to support to make the wikis better: https://meta.wikimedia.org/wiki/2017_Community_Wishlist_Survey Click on the categories to find the proposals. The voting will close on December 10. That's the important part of this email. Feel free to follow the link above and starting voting right now. The longer version: The Community Wishlist Survey decides what the Wikimedia Foundation Community Tech team will work on over the next year. The team is responsible for addressing the top 10 wishes on the list, as well as some wishes from smaller groups and projects that are doing important work, but don't have the numbers to get their proposal into the top 10. The Wishlist is also used by volunteer developers and other teams, who want to find projects to work on that the community really wants. Come help set the agenda. If you want to see what the team has done in 2017, see the status report from last month: https://meta.wikimedia.org/wiki/2016_Community_Wishlist_ Survey/Status_report_1 What you can do now: *) Vote. This is the most important thing. *) Spread the word. We really want people to find this, of course, and we'll work on finding the best balance between spreading the news to everyone and not being annoying, but please do help to spread the information in your local community – Village Pump equivalents, IRC channels, social media groups and so on. *) Help translating the pages. We want the process to be as available as possible for everyone. It's not every available if it's only in English. *) If you want to get short updates through the notification system, you can sign up for the Community Tech Newsletter: https://www.mediawiki.org/wiki/Newsletter:The_Community_Tech_Newsletter //Johan Jönsson --

1 0

AdvancedSearch beta feature now on Mediawiki
by Birgit Müller 21 Nov '17

21 Nov '17

(sorry for cross-posting ...) Hello all, we're really happy to announce that the new AdvancedSearch interface got deployed as a beta feature to Mediawiki.org and test just 2 hours ago. [1] AdvancedSearch enhances Special:Search through an advanced parameters form and aims to make existing search options more visible and accessible for everyone. [2] The feature is a project by WMDE's Tech team and originates from the Technical Wishes project. Many other people and teams helped making this project happen, so stay tuned for the thank you section at the end of this email :-) [3] *Why AdvancedSearch* The Search has great options to perform advanced queries, but often even experienced editors don't know about it - this is what we found out when we were conducting a workshop series on advanced search in several cities in Germany in 2016. Together with contributors from German Wikipedia we've discussed their desired search queries, explained how the syntax works and how keywords like "hastemplate", "filetype" or "intitle" can be used and combined to get the desired results. The idea for the AdvancedSearch feature results out of these workshops, where we not only discussed search but also designed first mocks for an advanced search interface. [4] *The first version, more deployments and other next steps*AdvancedSearch supports some of the special search options the WMF's search team has been implemented in the last years. The way the interface works, users don't have to know the syntax behind each search field, but they can learn about it if they want to. The first version of the feature comes with a first selection of advanced search options, e.g. including support for "hastemplate" or "intitle". The WMF's search team has started to work on a 'deepcat' functionality to make sub category search happen. We plan to add support for this in the future, too. If all goes well, we plan to deploy the beta feature on German and Arabic Wikipedia by Wednesday, Nov 29. [5] In the next 2-3 months we'd love to invite everyone to test the new feature on those wikis: Comments, thoughs, bug reports ... - any feedback is much appreciated! Please see https://www.mediawiki.org/wiki/Help:AdvancedSearch for how to use AdvancedSearch. Deployments to other wikis can follow later. *Thank you (finally!)* This project was developed with the support from many different people from the start, and this makes it very special to us: We would like to thank the folks from the German Wikipedia community who participated in the workshops and who were discussing and designing first ideas together with us. We would like to thank all other people who gave valuable feedback on Phabricator, onwiki and in real life. We would like to thank the Arabic Wikipedia community for getting interested in AdvancedSearch, and for offering to test this feature in an early stage. The credit for the development work on AdvancedSearch goes to our FUN team, supported by the team that usually works on Technical Wishes. Last but really not least we would like to thank the great folks from the WMF's search team who have supported this project from the start and who have done (and still do) the necessary backend work that now got supported by the AdvancedSearch extension. Sorry for the long email, and thanks for reading :-) Birgit (for the Technical Wishes team) [1] https://phabricator.wikimedia.org/T180147 (Deployment ticket group 0) [2] https://www.mediawiki.org/wiki/Help:AdvancedSearch [3] https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/AdvancedSearch [4] https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/AdvancedSearch/Worksh… [5] https://phabricator.wikimedia.org/T180128 (Deployment ticket arwiki and dewiki) -- Birgit Müller Community Communications Manager Software Development and Engineering Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin Tel. (030) 219 158 26-0 http://wikimedia.de Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen Wissens frei teilhaben kann. Helfen Sie uns dabei! http://spenden.wikimedia.de/ Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

1 0

Discovery Weekly Update for the week starting 2017-11-13
by Chris Koerner 20 Nov '17

20 Nov '17

Hello, Here is the Discovery weekly update for 2017-10-02. As always, feedback and questions are welcome. *Programing note*: Due to the US holiday there will be no status update for the week of 20 November. Updates will resume the week of 27 November. == Highlights == * There is a survey on French Wikipedia asking if they should replace static maps with dynamic maps and to define best practices. [0] * An update on the Discovery team and projects was shared [1] == Discussions == === Search === * Erik did quite a bit of research into whether we should expand the rescore windows for learning to rank — it was decided it wouldn't be helpful [2] * David fixed a bug where the JobQueue was trying to report update metrics when response was not valid [3] * Gehel paid down some technical debt by creating a dedicated elastic component in the APT repository — which will help get the next upgrade more smoothly [4] * Stas fixed bug where removed Wikidata description was still showing when searching [5] === Analysis === * Mikhail provided analysis of external & internal WDQS traffic as preparation for sizing of internal WDQS cluster. [6] === WDQS === * Stas reloaded WDQS database to enable external links and performance improvements [7] [0] https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Sondage/Remplacement_des_carte… [1] https://lists.wikimedia.org/pipermail/discovery/2017-November/001601.html [2] https://phabricator.wikimedia.org/T162369 [3] https://phabricator.wikimedia.org/T180298 [4] https://phabricator.wikimedia.org/T179964 [5] https://phabricator.wikimedia.org/T180382 [6] https://phabricator.wikimedia.org/T179850 [7] https://phabricator.wikimedia.org/T176593 --- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison Wikimedia Foundation

1 0

← Newer
1
...
18
19
20
21
22
23
24
...
76
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery