Discovery

discovery@lists.wikimedia.org

756 discussions

Elasticsearch - datacenter switch
by Guillaume Lederrey 05 Apr '17

05 Apr '17

Hello teams! After some discussion with David, we realised that Cirrus / Elasticsearch switch is already more automated than we realised. Cirrus is configured to talk the local Elasticsearch cluster. So if we start serving traffic for Mediawiki from codfw, those mediawiki instances should contact the Elasticsearch codfw cluster. We do have the ability to change that configuration and for the use of a specific cluster. That's what we did during the previous datacenter switch, and what we already do for some maintenance operations (yes, major upgrades of Elasticsearch do require downtime, so we use codfw during those upgrades). Since we have already tested a manual DC switch quite a few times, it is time to check if this automatic switch is working as it should. The only downside is that it increases the number of moving parts during the Mediawiki switch. On a last note, lots of thanks and praise to David and Erik who did think ahead much more than I did and implemented those nice features! Have fun! Guillaume -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST

1 0

Discovery Weekly Update for the week starting 2017-03-20
by Chris Koerner 01 Apr '17

01 Apr '17

It's Spring in the Northern Hemisphere. Seasonal allergies ahoy! This is the weekly update from the Discovery department for the week starting 2017-03-20 and the week of 2017-03-27. == Highlights == * Finished up the analysis of the first A/B test for the sister project search results [10] [11] == Discussions == === Search === * Updates from work done in previous weeks (apologies for the delay!): ** Fixed an issue where subpages were mistakenly preferred over parent pages if you didn't type the whole title [0] ** Fixed multiple issues with the how Elasticsearch integrates with the Translate extension [1] [2] [3] ** Test wiki with new Chinese Traditional-to-Simplified conversion, better word segmentation, and using the new BM25 is available in labs. [4] (It has the index and snippets from Chinese Wikipedia, but not the actual articles.) Additional analysis from native-speaker review is also on MediaWiki. More feedback always welcome. [5] * Fixed regressions which seem to be related to Elasticsearch 5 upgrade: ** Fixed an issue causing language-converted suggestions to not show in the search box [6] * Fixed an issue where using certain special characters in advanced search syntax would unexpectedly give zero results [7] * Finished the second A/B test for the sister project search results. * The analysis of new Ukrainian language analyzer is on MediaWiki, and there's lots of good discussion on Phabricator. More feedback welcome! [8] [9] === Analysis === * Finished up the analysis of the first A/B test for the sister project search results (task T156300) read it here [10] [11] === Wikidata Query Service === * Enabled SPARQL Federation [12] * Enabled parametrized query templates in the GUI [13] [0] https://phabricator.wikimedia.org/T156840 [1] https://phabricator.wikimedia.org/T132076 [2] https://phabricator.wikimedia.org/T132315 [3] https://phabricator.wikimedia.org/T158070 [4] http://zh-wp-smartcn-relforge.wmflabs.org/w/index.php?title=Special:搜索 [5] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Chinese_Analyzer_Ana… [6] https://phabricator.wikimedia.org/T160896 [7] https://phabricator.wikimedia.org/T147636 [8] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Ukrainian_Morfologik… [9] https://phabricator.wikimedia.org/T160106 [10] https://phabricator.wikimedia.org/T156300 [11] https://commons.wikimedia.org/wiki/File:A_Test_Of_Cross-wiki_Search_-_Helpi… ’s_Sister_Projects.pdf [12] https://www.wikidata.org/wiki/Wikidata:Project_chat#Federation_in_the_Wikid… [13] https://www.wikidata.org/wiki/Wikidata:Project_chat#Query_Service:_you_can_… --- The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Help with Ukrainian?
by Trey Jones 31 Mar '17

31 Mar '17

Hey everyone, If you know any Ukrainian, I could use your help reviewing the new Ukrainian language analyzer that we want to deploy to all Ukrainian language wikis. More info is on Phabricator: https://phabricator.wikimedia.org/T160106#3145612 Thanks! —Trey Trey Jones Software Engineer, Discovery Wikimedia Foundation

1 0

Discovery Weekly Update for the week starting 2017-03-13
by Chris Koerner 18 Mar '17

18 Mar '17

A full week in Discovery. Feedback, questions, ruminations on the meaning of life, are welcome. == Highlights == * ElasticSearch 5.1.2 is now serving all wikis from the codfw datacenter, the completion suggester has been disabled during the transition and will be re-enabled very soon. The cluster in eqiad will be upgraded next week; announced on wikitech-l and discovery mail lists [0] [1] * Started a second A/B test for sister project search results on selected wikis, announced on wikitech-l, discovery, and wikitech-ambassadors mail lists. [2] [3] [4] == Discussions == === Search === * Took a wrong turn on some updates to the Swedish analysis chain (T155822), but got set right with some helpful input from native speakers (T160562) and completed the analysis of enabling ICU folding on Swedish, which we are in the process of deploying. Actual updates to the Swedish wikis will follow. [5] [6] [7] * updated several minor UI bugs for second A/B test for sister project search results [8] [9] * turned on second A/B test for sister project search results (task T160005) for a week of testing on 8 wikis: persian, italian, catalan, polish, arabic, french, german, russian; more information in parent ticket (task T160004) [10] [11] === Analysis === * updated the portal dashboard to now show sister project clickthroughs (task T152617) and app links (task T154634) [12] [13] * updated external traffic dashboard to add note about internally referred traffic being mis-categorized [14] * created a new dashboard to display the usage stats for Wikidata Query Service (WDQS) end point (SPARQL and LDF) usage [15] * finalizing the analysis of the first A/B test for the sister project search results (draft link in comments of ticket) [16] === Portal === * completed a variety of updates to the Portal for RTL, icons and caching [17] [18] [19] [20] [21] [22] * updated the portal's translations and statistics [23] [24] === Interactive === * Tasmania was accidentally flooded in Wikimedia maps. Through careful work, we managed to drain the excess water successfully. [25] == Other Noteworthy Stuff == * Live your life like you just burst through the door of your dad's super important live broadcast skype call [26] == Development == * Stas shared some thoughts on design for improved Wikidata search using ElasticSearch backend [27] [0] https://lists.wikimedia.org/pipermail/wikitech-l/2017-March/087801.html [1] https://lists.wikimedia.org/pipermail/discovery/2017-March/001461.html [2] https://lists.wikimedia.org/pipermail/wikitech-l/2017-March/087800.html [3] https://lists.wikimedia.org/pipermail/discovery/2017-March/001460.html [4] https://lists.wikimedia.org/pipermail/wikitech-ambassadors/2017-March/00156… [5] https://phabricator.wikimedia.org/T155822 [6] https://phabricator.wikimedia.org/T160562 [7] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Swedish_Analyzer_Ana… [8] https://phabricator.wikimedia.org/T158937 [9] https://phabricator.wikimedia.org/T158935 [10] https://phabricator.wikimedia.org/T160005 [11] https://phabricator.wikimedia.org/T160004 [12] https://phabricator.wikimedia.org/T152617 [13] https://phabricator.wikimedia.org/T154634 [14] https://phabricator.wikimedia.org/T154722 [15] https://phabricator.wikimedia.org/T156300 [16] https://phabricator.wikimedia.org/T156300 [17] https://phabricator.wikimedia.org/T159601 [18] https://phabricator.wikimedia.org/T129771 [19] https://phabricator.wikimedia.org/T154349 [20] https://phabricator.wikimedia.org/T153764 [21] https://phabricator.wikimedia.org/T158782 [22] https://phabricator.wikimedia.org/T158810 [23] https://phabricator.wikimedia.org/T142582 [24] https://phabricator.wikimedia.org/T128546 [25] https://phabricator.wikimedia.org/T159631 [26] http://deathtodickens.com/post/158370691335/yolo-babies [27] https://www.wikidata.org/wiki/User:Smalyshev_(WMF)/Wikidata_search ---- The full update, and archive of all past updates, can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

New A/B test for sister project search results
by Deborah Tankersley 18 Mar '17

18 Mar '17

Hello, tl;dr - we’re running a second test for displaying sister projects in the search results (with new link highlights) on 8 wikis for a week; starting today, with apologies for sending to multiple email lists. As a follow up to a recent A/B test <https://phabricator.wikimedia.org/T149806> that the Discovery Search <https://www.mediawiki.org/wiki/Wikimedia_Discovery/Search> team conducted, we’ll be running a second test <https://phabricator.wikimedia.org/T160004> for displaying sister project search results in a sidebar on several wiki’s search engine results page (SERP) - starting today. You can read the draft report <https://commons.wikimedia.org/wiki/File:A_Test_Of_Cross-wiki_Search_-_Helpi…> or browse the next two paragraphs for the biggest concerns. We realized that when we display the results from the sister projects, it might not have been intuitive to our visitors that there were links that could be clicked to go to individual articles and to additional search results, so, we’ve added blue links <https://phabricator.wikimedia.org/T158935> to the results. We also discovered that the smaller wikis in which we had originally chosen to test on, might not have had enough relevant results in their sister projects to be displayed. To remedy that, we reached out <https://phabricator.wikimedia.org/T158935> to the community and added 4 larger wikis along with the original 4 wikis, to test on. The list is as follows: Persian, Italian, Catalan, Polish, Russian, French, German, and Arabic. Here are a few links that will not affect our testing, but will demonstrate the new test display: - cawiki: https://ca.wikipedia.org/wiki/Special:Search?search= manifestaci%C3%B3&fulltext=1&cirrusUserTesting=recall_ sidebar_results&searchToken=66xv5qgi97musdtybgwik11sk#manifestaci%C3%B3 <https://ca.wikipedia.org/wiki/Special:Search?search=manifestaci%C3%B3&fullt…> - dewiki: https://de.wikipedia.org/wiki/Special:Search?search= regenbogen&fulltext=1&cirrusUserTesting=recall_ sidebar_results&searchToken=44wgqit3kykji2o1cxtaxlzcs#regenbogen <https://de.wikipedia.org/wiki/Special:Search?search=regenbogen&fulltext=1&c…> - arwiki: https://ar.wikipedia.org/wiki/Special:Search?search=%D9%82% D9%88%D8%B3+%D8%A7%D9%84%D9%85%D8%B7%D8%B1&fulltext=1& cirrusUserTesting=recall_sidebar_results&searchToken= bxfe1fnvj7sav1nue56hpjt69#%D9%82%D9%88%D8%B3%2B%D8%A7%D9%84% D9%85%D8%B7%D8%B1 <https://ar.wikipedia.org/wiki/Special:Search?search=%D9%82%D9%88%D8%B3+%D8%…> You can edit the url to change the search term and also change to a different Wikipedia. For example, try this url for “apple pie” on enwiki: https://en.wikipedia.org/wiki/Special:Search?search=apple+pie&fulltext=1& cirrusUserTesting=recall_sidebar_results&searchToken= 8mavijx36f9cl26l6k2v2o11l#apple%2Bpie Let us know what you think! Cheers from the Discovery Search Team -- deb tankersley irc: debt Product Manager, Discovery Wikimedia Foundation

4 5

ElasticSearch 5 is alive!
by Deborah Tankersley 17 Mar '17

17 Mar '17

Hello, After several months of hard work by the Discovery Search <https://www.mediawiki.org/wiki/Wikimedia_Discovery/Search> team, we're happy to announce that the CirrusSearch <https://www.mediawiki.org/wiki/Extension:CirrusSearch> backend has been upgraded to ElasticSearch <https://www.elastic.co/products/elasticsearch> version 5. This update was bit tedious and difficult for the team; you can browse the various tickets <https://phabricator.wikimedia.org/T154501> in Phabricator if you'd like to know more. This upgrade to ES5 is great, because we now have a new Ukrainian analyzer to experiment with, a new reindexing API and soon, we’ll have an updated completion suggester - all of which makes our backend more standard and reduced a significant amount of technical debt. Cheers from the Discovery Search Team! -- deb tankersley irc: debt Product Manager, Discovery Wikimedia Foundation

1 0

Discovery Weekly Update for the week starting 2017-02-27
by Chris Koerner 15 Mar '17

15 Mar '17

Hello, This is the weekly update for the week starting 2017-02-27 and includes work up till 2017-03-13 due to a scheduling error. Apologies for the confusion. == Highlights == *Jan demo'd a new "explore similar" widget during this month's CREDIT showcase [0] == Discussions == * Posted on French, Arabic, German and Russian Wikipedia Village Pumps to ask if there are any issues with A/B testing the sister project search results on them. [1] === Search === * Finished analysis of potential new Chinese-language analyzers. See T158203. We expect improved word segmentation accuracy, which should lead to better search results, as well as support for searching across Traditional/Simplified character sets. A test index in labs will be available soon. [2] * Elasticsearch 5 upgrade in progress: the beta cluster is now running the new version. Upgrade of production clusters will start next week. === Analysis === * Recap of the migration to ReportUpdater and more! [3] * Analysis of the A/B test for displaying sister projects in search results is ongoing [4] ===Portal === * Portal statistics were updated. [5] * Fix released for bad caching of error page on portal - to be released next week. [6] * Worked on a few small fixes for the portal to include a few edits on RTL language display on the portal - to be released next week. [0] https://www.youtube.com/watch?v=aEGlXIKUXD8 [1] https://phabricator.wikimedia.org/T154525#3091063 [2] https://phabricator.wikimedia.org/T158203 [3] https://lists.wikimedia.org/pipermail/discovery/2017-March/001454.html [4] https://phabricator.wikimedia.org/T156300 [5] https://phabricator.wikimedia.org/T128546 [6] https://phabricator.wikimedia.org/T158782#3064105 [7] https://phabricator.wikimedia.org/T129771 [8] https://phabricator.wikimedia.org/T154349 [9] https://phabricator.wikimedia.org/T158809 ---- The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Ideas from the search retrospective
by Kevin Smith 14 Mar '17

14 Mar '17

This email is aimed at members of the Search team, so anyone else can freely ignore it. It was easier for me to send it to this list, and it seemed like it could be beneficial to have others aware of the issues that came up. Here is a list of all the ideas that came up during the recent search retrospective. I encourage further discussion about them, either in this thread, or in a new thread per item/cluster, or in the talk page for the retro itself[1], or elsewhere. - Do people want more feedback on their work/way [We discussed this, and Erika has an action item to follow up] - In hangouts, make sure you are on screen when speaking [Note that although you see yourself in a rectangle, everyone else only sees you as a square] - do we need a fully documented checklist on how to update the portal? - automate our maven deployment / release process - Find a way to fluently enable new search keywords that need reindex/new data - Running A/B tests is a very slow feedback loop - Getting feedback on things that are still internal, before releasing to the world - Weekly sprint planning meeting has a variable agenda - Does Cross-Wiki work still need its own column on the Search workboard? - We sometime focus too much on having efficient meetings [Kevin has an action item for this] - Product (user?) Testing: I'd like to see if there's ways we could expand our testing "toolbelt" - Do we need to foster more knowledge sharing between areas of expertise? [Erika has an action item for this] - How to bring in outside people/organizations that want to help (converting interest into a contract or other formal agreement) - This quarter has been a little lighter on community discussion [1] https://www.mediawiki.org/wiki/Wikimedia_Discovery/Meetings/Search_retrospe… Kevin Smith Agile Coach, Wikimedia Foundation

3 3

Updates to Discovery Dashboards and Metrics
by Mikhail Popov 10 Mar '17

10 Mar '17

Howdy! Just wanted to let y’all know about some updates to Discovery Dashboards[1]: - All dashboards have been updated to use a new collection of datasets[2] - The Wikipedia.org Portal dashboard[4] now tracks: - Clicks on sister project links (T152617) - Clicks on mobile app links (T154634) - The WDQS dashboard[3] now tracks LDF endpoint usage (T153936) So that’s the front-end news. On the back-end: - We’ve refactored the entire metric-calculation codebase[5] to use Analytics’ Reportupdater infrastructure[6] (T150915) - That codebase is now thoroughly documented[7] - We’ve introduced additional rules for detecting automata to several scripts that calculate traffic and usage metrics - We’ve erased and backfilled all the metrics from 2017-01-01 using all the new definitions and updated UDFs - So if you see drastic changes to a metric starting on 1st January 2017, that’s why :) - Accordingly, all the graphs have been annotated with this info - We are no longer undercounting full-text and geo search API usage (see [8], hence the jump on 2017-01-01) - Adding new metrics and backfilling data is now way easier - We’re already working on adding the ability to look at several search metrics by language and project (T150410) - We can even add predictive models of metrics AS metrics for the codebase to calculate (T112170, [9]) A huge thanks to Deb for her support and patience through this project, and huge thanks to Chelsy for her thorough code review of and help with [10]. Cheers, Mikhail on behalf of Discovery’s Analysis team [1]: https://discovery.wmflabs.org/ [2]: https://datasets.wikimedia.org/aggregate-datasets/discovery/ [3]: https://discovery.wmflabs.org/wdqs/ [4]: https://discovery.wmflabs.org/portal/ [5]: https://phabricator.wikimedia.org/diffusion/WDGO/ [6]: https://wikitech.wikimedia.org/wiki/Analytics/Reportupdater [7]: https://github.com/wikimedia/wikimedia-discovery-golden/blob/master/README.… [8]: https://gerrit.wikimedia.org/r/#/c/315503/ [9]: https://github.com/wikimedia/wikimedia-discovery-golden/blob/master/README.… [10]: https://gerrit.wikimedia.org/r/#/c/325870/ -- Mikhail Popov // Count Logula Discovery, Wikimedia Foundation PGP Public Key: https://people.wikimedia.org/~bearloga/public.asc Fingerprint: B362 76D4 FEBE F715 5F40 0C9C 4BC8 A265 E573 3216

3 2

Fwd: [Affiliates] Recognition of the Wikimaps User Group
by Pine W 08 Mar '17

08 Mar '17

FYI Pine ---------- Forwarded message ---------- From: Kirill Lokshin <kirill.lokshin(a)gmail.com> Date: Tue, Mar 7, 2017 at 8:37 AM Subject: [Affiliates] Recognition of the Wikimaps User Group To: Wikimedia Mailing List <wikimedia-l(a)lists.wikimedia.org>, Wikimedia Movement Affiliates discussion list <affiliates(a)lists.wikimedia.org> Hi everyone! I'm very happy to announce that the Affiliations Committee has recognized the Wikimaps User Group [1] as a Wikimedia User Group. The group aims to gather Wikimedia users engaged in a variety of activities related to geographic information from across the movement, so that people with different ideas for using geographic components in their projects are able to come together, share their expertise, and help each other. Please join me in congratulating the members of this new user group! Regards, Kirill Lokshin Affiliations Committee [1] https://meta.wikimedia.org/wiki/Wikimaps_User_Group _______________________________________________ Affiliates mailing list Affiliates(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/affiliates

1 0

← Newer
1
...
25
26
27
28
29
30
31
...
76
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery