Discovery August 2018

discovery@lists.wikimedia.org

4 participants
6 discussions

Discovery Weekly Update for the week starting 2018-08-20

by Chris Koerner

Hello, This is the weekly update from the Search Platform team for the week starting 2018-08-20. As always, feedback and questions welcome. == Discussions == === Search === * Gehel did a lot of work on getting the servers ready for ElasticSearch 6: resolving a deprecated data directory structure [0], migrating from Jessie to Stretch [1], and migrating the cirrus cluster to RAID0 [2] * Erik and Gehel fixed an issue where the mjolnir-kafka-bulk-daemon was failing on all elastic / eqiad nodes [3] * Erik also added stats collection for observability of mjolnir daemons using prometheus [4] * We had an issue where the warning message was not very informative when doing a search with greater than 10,000 results; Erik updated the warning to now read: "''A warning has occured while searching: Could not retrieve results. Up to 10000 search results are supported, but results starting at 10000 were requested.''" [5] * When the ElasticSearch server went down recently, there were various PHP notices spamming the logs. Erik was able to fix this by better handling the null result from link count msearch [6] * Trey worked on creating and merging, a Esperanto Elasticsearch plugin using Esperanto morphological libraries [7] * Stas worked on a enabling indexing all entity-valued statements [8] which is now enabled for newly edited items and will be added to all items on next reindex [9] === WDQS === * Stas has fixed some bugs in categories daily RDF diffs generation [10] and wrote puppet recipies to enable auto-loading them on WDQS servers. Switch from weekly to daily loading to start on Monday 27th. ---- [0] https://phabricator.wikimedia.org/T199389 [1] https://phabricator.wikimedia.org/T193649 [2] https://phabricator.wikimedia.org/T198391 [3] https://phabricator.wikimedia.org/T202120 [4] https://phabricator.wikimedia.org/T201948 [5] https://phabricator.wikimedia.org/T201360 [6] https://phabricator.wikimedia.org/T105887 [7] https://phabricator.wikimedia.org/T200099 [8] https://phabricator.wikimedia.org/T199884 [9] https://phabricator.wikimedia.org/T147505 [10] https://phabricator.wikimedia.org/T201216 ---- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Relations Specialist Wikimedia Foundation

5 years, 8 months

Fwd: [Wikitech-l] VPS users, please claim your projects (second notice)

by Deb Tankersley

FYI: please read email below if you use a VPS project and want to keep it. Cheers, Deb -- deb tankersley program manager, engineering Wikimedia Foundation ---------- Forwarded message --------- From: Andrew Bogott <abogott(a)wikimedia.org> Date: Sun, Aug 26, 2018 at 10:33 PM Subject: [Wikitech-l] VPS users, please claim your projects (second notice) To: <Cloud-announce(a)lists.wikimedia.org>, Wikimedia developers < wikitech-l(a)lists.wikimedia.org> In an attempt to identify abandoned VPS projects, I've created a wiki page that lists all existing projects, here: https://wikitech.wikimedia.org/wiki/Cloud_VPS_2018_Purge Currently 85 projects[2] on that list are unclaimed. If you are a VPS user, please visit that page and mark any projects that you use as {{Used}}. Note that it's not necessary for you to be a project admin to mark something -- if you know that you're currently using a resource and want to keep using it, go ahead and mark it accordingly. If you /are/ a project admin, please take a moment to mark which VMs are or aren't used in your projects. When October arrives, I will shut down and begin the process of reclaiming unused projects. If you think you use a VPS project but aren't sure which, I encourage you to poke around on https://tools.wmflabs.org/openstack-browser/ to see what looks familiar. You can also log in to http://horizon.wikimedia.org which will provide you with a handy menu of projects that you are currently a member of. Thank you! -Andrew and the WMCS team [1] https://phabricator.wikimedia.org/phame/post/view/112/neutron_is_finally_co… [2] Here, for good measure, is that list. Every one of these projects is currently a candidate for deletion: aicaptcha analytics bots chicotestproject ci-staging codereview collection-alt-renderer commonsarchive community-labs-monitoring contributors dashiki deep-learning-services discourse discourse-wam download etytree fastcci getstarted glampipe globaleducation hat-imagescalers hound iiab iiifls1 ircd jupyter kubernetes-testing maps maps-team math matrix mcr-dev mediahandler-tests mediawiki-docker multimedia mw-api-testing mw-extension-ids mwfuzz mwoffliner newsletter nonfreewiki openocr ores orig otrs pagemigration paws phlogiston piwik project-smtp rcm rdfiodev reading-lists reading-web-staging recommendation-api redirects reportcard sciencesource sentry services services-testbed t136871 test-twemproxy thumbor tor traffic twl utrs video wcdo wdq-mm wikibase-nearest-neighbors wikibrain wikidata-federation wikidata-page-banner wikidata-primary-sources-tool wikidumpparse wikifactmine wikimetrics wikisource-tools wikistream wikitolearn-dev wildcat wlmjudging wmf-research-tools _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

5 years, 8 months

Incorrect info in interface on numbers of results returned on Wikimedia search page?

by Michael Khoo

Hello all, First, I'm a newcomer here so apologies if this is not the best place to post this (and any advice on where I should be going would be much appreciated - thanks!). Anyway I think there's a small discrepancy on the Wikimedia search results page, regarding the number of results that are displayed on a page. - The 'View' links at the bottom of the page are all in multiples of 10 - e.g 20, 50, 100 - The actual number of returned results all appear to be multiples of 10 - The /reported/ numbers of returned results are all ((some multiple of 10)+1) So for example I carry out a search, and I see the first twenty results. At the bottom of the page there are links for 'next 20,' etc. However at the top right of the page it says 'Results 1-21" which is 21 results. If I view the next 20 results, I see 20 results, but the top right of the page says 'Results 21-41" which is 21 results again, and so on. Not sure where this is arising from, maybe it's an interface issue, although I can't find a specific interface group at Wikimedia. Apologies if this has already been brought up! Best wishes, Mick.

5 years, 8 months

Discovery Weekly Update for the week starting 2018-08-13

by Chris Koerner

Hello, This is the weekly update from the Search Platform team for the week starting 2018-08-13. As always, feedback and questions welcome. == Discussions == === Search === * We re-indexed wikis in Malay, Indonesian, and Polish, adding stemming to Malay and ICU normalization to Indonesian [0], and fixing the worst pathological stemming errors in Polish [1]. ** Reindexing tickets are [2] and [3]. *Trey wrote a blog post about the difficulties of tokenization (breaking text up into words, more or less). [4] It's the first in a series covering the basics of search. == Did you know? == * Auto-antonyms (or contronyms, among other names) are words that have contradictory meanings. [5] Some cases arise from words that were originally distinct, but came to sound the same through normal sound change, such as “cleave”, which means both to hold fast (from Old English “clifian”) and to cut in two (from Old English “clēofan”). Other times, words just acquire novel but contradictory senses, such as “fast”, which means both to be securely attached to, and able to move quickly. Often the sense is quite clear from context, but when it is not, words can become so confusing that careful writers avoid them; these are sometimes called “skunked” words. [6] --- [0] https://phabricator.wikimedia.org/T196780 [1] https://phabricator.wikimedia.org/T186046 [2] https://phabricator.wikimedia.org/T200037 [3] https://phabricator.wikimedia.org/T200204 [4] https://wikimediafoundation.org/2018/08/07/anatomy-search-token-affection/ [5] https://en.wikipedia.org/wiki/Auto-antonym [6] https://en.wikipedia.org/wiki/Skunked_term Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R As always, feedback and questions welcome. Yours, Chris Koerner Community Relations Specialist Wikimedia Foundation

5 years, 8 months

Discovery Weekly Update for the week starting 2018-08-06

by Chris Koerner

Hello, This is the weekly update from the Search Platform team for the week starting 2018-08-06. As always, feedback and questions welcome. == Discussions == === Search === * Erik and David worked on setting up a new branch for elasticsearch 6.x (as a first step) [0], which is mostly done but will need a follow-on task to get into production [1] * Erik worked on a small but weird bug where Cirrussearch on test.wikipedia.org was not able to search on specific non-zero namespaces and fixed it [2] * Stas fixed the error message for haswbstatement without valid query [3] and searching for property without specifying the value [4], e.g. <code>haswbstatement:P231</code>. --- [0] https://phabricator.wikimedia.org/T199256 [1] https://phabricator.wikimedia.org/T200198 [2] https://phabricator.wikimedia.org/T201584 [3] https://phabricator.wikimedia.org/T200824 [4] https://phabricator.wikimedia.org/T199886 Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Relations Specialist Wikimedia Foundation

5 years, 8 months

Discovery Weekly Update for the week starting 2018-07-30

by Chris Koerner

Greetings, This is the weekly update from the Search Platform team for the week starting 2018-07-30. As always, feedback and questions welcome. == Discussions == === Search === * David did a lot of work surrounding the upgrade to ElasticSearch 6.3: ** updated an extra plugin[0] ** updated the LTR plugin [1] ** worked on an older issue that we were waiting to fix for an experimental plugin that was throwing an exception on some requests [2] ** upgraded the experimental highlighter [3] ** upgraded the extra-analysis plugin [4] * Trey fixed an address stemming issue in Polish analyzer for search [5] and it'll be fully in production when a re-index occurs [6] * Trey also worked on a review of the Esperanto Morphological Libraries when a volunteer wanted to help in this effort (and knew Esperanto) [7] * David worked on a few cirrus integration tests are were randomly and consistently failing (V => Venom > V:N) [8] * Trey took on the task of reviewing the work of applying Indonesian Analysis Chain for Malay (writeup and review) [9] [10] * David took on the fairly massive task of changing how SpecialSearch/SearchEngine handles the 'prefix' URI param set by the InputBox extension with lots of help from others [11] * David did quite a bit of work on deprecating the SearchEngine::replacePrefixes with several related patches [12] * Trey worked on exploring the potential applications for NLP to be applied in Search (review and write-up [13] [14] * Stas implemented fulltext search for Lexemes when Lexeme namespace is requested in the search [15] * Stas added collecting click data from Wikidata prefix search into event logs so that we know what the users are selecting when using Wikidata prefix search [16] and it'll be in production late this week with the train * Erik and David worked on the completion suggester code which didn't gracefully handle shard failures during the fetch phase, which caused the response received by cirrus to not contain the necessary information [17] and it'll be in production late this week with the train * After a lot of discussion, Erik fixed an issue where intitle search didn't match the stop words [18] and it'll be in production late this week with the train * Gehel and Erik worked on ensuring discovery.query_clicks_* data is purged per privacy policy, with adding support for hourly or daily partition dropping [19] === Wikidata Query Service === * Wikibase Constraints violations are now loaded into WDQS database and are queryable. [20] [0] https://phabricator.wikimedia.org/T198067 [1] https://phabricator.wikimedia.org/T198066 [2] https://phabricator.wikimedia.org/T178425 [3] https://phabricator.wikimedia.org/T198069 [4] https://phabricator.wikimedia.org/T198068 [5] https://phabricator.wikimedia.org/T186046 [6] https://phabricator.wikimedia.org/T200037 [7] https://phabricator.wikimedia.org/T197240 [8] https://phabricator.wikimedia.org/T199445 [9] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Analysis_of_Applying… [10] https://phabricator.wikimedia.org/T196780 [11] https://phabricator.wikimedia.org/T198318 [12] https://phabricator.wikimedia.org/T198860 [13] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Potential_Applicatio… [14] https://phabricator.wikimedia.org/T193070 [15] https://phabricator.wikimedia.org/T196188 [16] https://phabricator.wikimedia.org/T196186 [17] https://phabricator.wikimedia.org/T180807 [18] https://phabricator.wikimedia.org/T68969 [19] https://phabricator.wikimedia.org/T189845 [20] https://phabricator.wikimedia.org/T172380 --- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Relations Specialist Wikimedia Foundation

5 years, 8 months

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery August 2018