Discovery

discovery@lists.wikimedia.org

1 participants
756 discussions

Discovery Weekly Update for the week starting 2017-10-16
by Deborah Tankersley 24 Oct '17

24 Oct '17

This is the weekly update for the week starting '''2017-10-16''' ==Highlights== * Wikidata Query Service had a partial outage on 18 Oct 2017: a single client started to run an unusually high number of queries on WDQS and the overload was not prevented by our current throttling nor was it automatically detected and isolated—detailed information is on the mailing list. [0] * Discovery and Legal chatted on Oct 18 about supporting only CC0 data in the .map data namespace on Commons, in response to a community conversation; a detailed response will be prepared and released next week. [1] [2] ==Discussions== === Search === * We added more namespaces to the search results for Bengali Wikisource (Special:Search—add) [3] * David fixed an issue where users couldn't click on svwikisource search results when running a crosswiki search on svwiki; this will be in production during the week of Oct 24th [4] * David was also working on finishing up deployment-prep elasticsearch cluster to 5.5.x [5] * David launched the A/B test for testing the relaxing of the retrieval query filter using two different configurations (the test is expected to last a week) [6] * We wrapped up a series of adding content namespaces to wikis that were accidentally excluded from the default results of Special:Search (thanks to TheDJ for discovering this bug and to David for his quick fixing of the issues once the community consultation was completed by Chris K) [7] * Erik deployed a fix for when the rebuildtextindex fails when searchindex on InnoDB (no support for MyISAM) and it went into production on Oct 19 [8] === Analysis === * Mikhail finished up tasks related to establishing the baseline of metrics for SDoC—looking at querying databases [9] === Portal === * The Wikipedia.org portal page stats and translations were updated on Oct 19, 2017 [10] [11] === Maps === * MaxSem and Gehel fixed a weird bug where tileshell wasn't exiting after after being notified of updates, effectively stopping the OSM data in production from being regenerated; data updates are now back and running once a day [12] [0] https://lists.wikimedia.org/pipermail/wikidata/2017-October/011280.html [1] https://commons.wikimedia.org/wiki/Commons:Village_pump/Proposals#Proposal_… [2] https://phabricator.wikimedia.org/T154071 [3] https://phabricator.wikimedia.org/T178041 [4] https://phabricator.wikimedia.org/T177913 [5] https://phabricator.wikimedia.org/T178413 [6] https://phabricator.wikimedia.org/T177502 [7] https://phabricator.wikimedia.org/T170473 [8] https://phabricator.wikimedia.org/T177477 [9] https://phabricator.wikimedia.org/T177356 [10] https://phabricator.wikimedia.org/T128546 [11] https://phabricator.wikimedia.org/T142582 [12] https://phabricator.wikimedia.org/T177389 --- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R -- deb tankersley Product Manager, Discovery Wikimedia Foundation

1 0

Discovery Weekly Update for the week starting 2017-10-09
by Deborah Tankersley 17 Oct '17

17 Oct '17

Hello all, Here is the Discovery weekly update for 2017-10-09. As always, feedback and questions are welcome. ==Highlights== * After active discussions on Spanish Wikipedia, mapframe has been enabled on that site. [0] * * The discussion on English Wikipedia on whether to ask for mapframe to be enabled has been resolved with a consensus of 'yes' to support the use of this functionality on enwiki. [1] * There is an ongoing discussion on English Wikivoyage about the use of mapframe and a draft maps policy has been created. [2] [3] * There is a discussion on Commons about deletion requests, specifically about maps, user-derived work and licensing of that work. [4] * The Search Platform team had a long discussion on how to improve search relevance surveys and A/B tests, highlights from the meeting can be read in and we'll use the information discussed in future tests. [5] [6] ==Discussions== === Search === * Gehel finished the investigation into why elastic@codfw alerted during codfw row B switch upgrade, determined it was that a check done on the LVS endpoint ended up being routed to the server that had just recovered network connectivity. [7] * A 'fix' was released last week that actually had un-intended results on the search results page by adding an icon to every autocomplete result in the main dropdown. Thanks, @Jdforrester for the quick fix! [8] * Trey finished up and deployed code to not use Generic language fallbacks in Mediawiki for Elasticsearch language analyzers, once we re-index various small wikis, this code will be in production. [9] [10] * The A/B test running on 18 wikis with > 1% of search traffic has been stopped and is ready for analysis [11] [12] === Analysis === * Mikhail finished up the Interleaved Search Results A/B test analysis [13] * Chelsy fixed an issue where the dashboard was counting events, not sessions, for mobile web events [14] [15] === Portal === * Continuing conversations on the handling of Chinese languages on the Wikipedia portal with the community [16] === Maps === * Paul finalized a few map styling issues: tunnels; implement all osm-bright.tm2 features in brighmed [17] [18] * The regeneration of map tiles in production finished up on Oct 10 [19] * Mapframe has been enabled on Spanish Wikipedia [20] [0] https://es.wikipedia.org/wiki/Wikipedia:Encuestas/2017/Mapas_din%C3%A1micos [1] https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Should_we_… [2] https://en.wikivoyage.org/wiki/Wikivoyage:Travellers%27_pub#.7B.7BMapframe.… [3] https://en.wikivoyage.org/wiki/Wikivoyage:Map [4] https://commons.wikimedia.org/wiki/Commons:Deletion_requests/Data_talk:Kual… [5] https://phabricator.wikimedia.org/T155290 [6] https://phabricator.wikimedia.org/T171215#3677637 [7] https://phabricator.wikimedia.org/T170378 [8] https://phabricator.wikimedia.org/T177659 [9] https://phabricator.wikimedia.org/T177871 [10] https://phabricator.wikimedia.org/T147959 [11] https://phabricator.wikimedia.org/T177490 [12] https://phabricator.wikimedia.org/T176493 [13] https://phabricator.wikimedia.org/T171215 [14] http://discovery.wmflabs.org/metrics/#mobile_events [15] https://phabricator.wikimedia.org/T176811 [16] https://phabricator.wikimedia.org/T171647#3679370 [17] https://phabricator.wikimedia.org/T154823 [18] https://phabricator.wikimedia.org/T153400 [19] https://phabricator.wikimedia.org/T176252 [20] https://phabricator.wikimedia.org/T177695 --- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Cheers! -- deb tankersley Product Manager, Discovery Wikimedia Foundation

1 0

Fwd: [Wikidata] weekly summary #282
by Deborah Tankersley 16 Oct '17

16 Oct '17

Forwarding from the Wikidata email list. Read about brand new stuff in the Wikidata Query Service by Stas, as noted in the weekly summary: - Slides of the presentation "Wikidata Query Service: The State of the Engine" <https://docs.google.com/presentation/d/e/2PACX-1vT6TxLmaz9jC5SkTVJsI2vMbtpK…> by Stas Malyshev Cheers, Deb -- deb tankersley Product Manager, Discovery Wikimedia Foundation ---------- Forwarded message ---------- From: Lydia Pintscher <lydia.pintscher(a)wikimedia.de> Date: Mon, Oct 16, 2017 at 3:39 PM Subject: [Wikidata] weekly summary #282 To: "Discussion list for the Wikidata project." < wikidata(a)lists.wikimedia.org> Hey everyone :) Here is your summary of what happened around Wikidata over the past week. Discussions - New request for comments: Association Football Matches <https://www.wikidata.org/wiki/Wikidata:Requests_for_comment/Association_Foo…> Events <https://www.wikidata.org/wiki/Special:MyLanguage/Wikidata:Events>/ Press/Blogs <https://www.wikidata.org/wiki/Special:MyLanguage/Wikidata:Press_coverage> - Upcoming: *WikidataCon <https://www.wikidata.org/wiki/Wikidata:WikidataCon_2017>*, Berlin, 28-29 October (sold out) - If you organize a local event for the Wikidata birthday, feel free to add it on the events page <https://www.wikidata.org/wiki/Wikidata:Events> and the birthday page <https://www.wikidata.org/wiki/Wikidata:Fifth_Birthday> - A blog post <http://zone47.com/dozo/textes-daffiches-des-films-a-lire-sur-gallica> describing Textes d'affiches <http://textesdaffiches.histoiredelart.fr/#%21/>, a tool using Wikidata to show movies and the works they are adapted from, developed during a hackathon at the national French library (by Shonagon, in French) - Slides of the presentation "Wikidata Query Service: The State of the Engine" <https://docs.google.com/presentation/d/e/2PACX-1vT6TxLmaz9jC5SkTVJsI2vMbtpK…> by Stas Malyshev - Slides of the presentation "Wikidata and structured data initiatives at Wikimedia. Current trends and priorities" <https://docs.google.com/presentation/d/e/2PACX-1vSspElTEbYSnusd-9ROd-M3p3jW…> by Dario Taborelli - Wikidata as a linking hub for knowledge organization systems? Integrating an authority mapping into Wikidata and learning lessons for KOS mappings by Joachim Neubert <http://ceur-ws.org/Vol-1937/paper2.pdf> Other Noteworthy Stuff - If you are interested in Structured Data on Commons <https://commons.wikimedia.org/wiki/Commons:Structured_data>, and helping out as a Wikidata contributor, please consider joining the new Structured Commons community focus group <https://commons.wikimedia.org/wiki/Commons:Structured_data/Get_involved/Com…> ! - Participate to the Global Legislative Openness Week <https://www.mysociety.org/2017/10/12/join-the-glow-run-a-wikidata-workshop-…> with MySociety and organize a Wikidata workshop in your country, from 20th to 30th November 2017 - Inventaire now has a News section <https://wiki.inventaire.io/wiki/News> where you can follow ongoing discussions and developments of the project - wikidata-cli <https://github.com/maxlath/wikidata-cli> added bot support <https://github.com/maxlath/wikidata-cli/blob/master/docs/config.md#bot> and new commands (wd search <https://github.com/maxlath/wikidata-cli/blob/master/docs/read_operations.md…>, wd aliases <https://github.com/maxlath/wikidata-cli/blob/master/docs/read_operations.md…>, wd add-alias <https://github.com/maxlath/wikidata-cli/blob/master/docs/read_operations.md…>, wd remove-alias <https://github.com/maxlath/wikidata-cli/blob/master/docs/read_operations.md…>, wd set-alias <https://github.com/maxlath/wikidata-cli/blob/master/docs/read_operations.md…> ) - wikidata-edit <https://github.com/maxlath/wikidata-edit> added bot edits support <https://github.com/maxlath/wikidata-edit/blob/master/docs/how_to.md#bot-edi…> and new functions (alias.add <https://github.com/maxlath/wikidata-edit/blob/master/docs/how_to.md#add-ali…>, alias.remove <https://github.com/maxlath/wikidata-edit/blob/master/docs/how_to.md#remove-…>, alias.set <https://github.com/maxlath/wikidata-edit/blob/master/docs/how_to.md#set-ali…> ) - HarvestTemplates <http://tools.wmflabs.org/pltools/harvesttemplates> has a new collaboration platform <http://tools.wmflabs.org/pltools/harvesttemplates/share.php> - You can now use reCh <http://tools.wmflabs.org/pltools/rech/> to patrol items from your PagePile Did you know? - Newest properties <https://www.wikidata.org/wiki/Special:ListProperties>: Women's Sports Foundation ID <https://www.wikidata.org/wiki/Property:P4377>, Oklahoma Sports Hall of Fame ID <https://www.wikidata.org/wiki/Property:P4376>, North Carolina Sports Hall of Fame ID <https://www.wikidata.org/wiki/Property:P4375>, New Mexico Sports Hall of Fame ID <https://www.wikidata.org/wiki/Property:P4374>, National Trust Collections ID <https://www.wikidata.org/wiki/Property:P4373>, Infopatrimônio ID <https://www.wikidata.org/wiki/Property:P4372>, KBO pitcher ID <https://www.wikidata.org/wiki/Property:P4371>, KBO hitter ID <https://www.wikidata.org/wiki/Property:P4370>, Cairn author ID <https://www.wikidata.org/wiki/Property:P4369>, Mississippi Sports Hall of Fame ID <https://www.wikidata.org/wiki/Property:P4368>, Louisiana Sports Hall of Fame ID <https://www.wikidata.org/wiki/Property:P4367>, Kansas Sports Hall of Fame ID <https://www.wikidata.org/wiki/Property:P4366>, Hawai‘i Sports Hall of Fame ID <https://www.wikidata.org/wiki/Property:P4365>, Georgia Sports Hall of Fame ID <https://www.wikidata.org/wiki/Property:P4364>, DSMHOF athlete ID <https://www.wikidata.org/wiki/Property:P4363>, ASHOF athlete ID <https://www.wikidata.org/wiki/Property:P4362>, ExecutedToday ID <https://www.wikidata.org/wiki/Property:P4361>, Monumentos de São Paulo ID <https://www.wikidata.org/wiki/Property:P4360>, gravsted.dk ID <https://www.wikidata.org/wiki/Property:P4359>, Dutch lost building register ID <https://www.wikidata.org/wiki/Property:P4358>, Musikverket person ID <https://www.wikidata.org/wiki/Property:P4357>, Megalithic Portal ID <https://www.wikidata.org/wiki/Property:P4356>, PROSITE documentation ID <https://www.wikidata.org/wiki/Property:P4355>, Search formatter URL <https://www.wikidata.org/wiki/Property:P4354>, nominated by <https://www.wikidata.org/wiki/Property:P4353>, BillionGraves cemetery ID <https://www.wikidata.org/wiki/Property:P4352>, Cravo Albin artist ID <https://www.wikidata.org/wiki/Property:P4351>, salinity <https://www.wikidata.org/wiki/Property:P4350>, Lotsawa House Indian author ID <https://www.wikidata.org/wiki/Property:P4349>, Lotsawa House Tibetan author ID <https://www.wikidata.org/wiki/Property:P4348>, Florida Sports Hall of Fame athlete ID <https://www.wikidata.org/wiki/Property:P4347>, T4T35 megalith ID <https://www.wikidata.org/wiki/Property:P4346>, excavation director <https://www.wikidata.org/wiki/Property:P4345>, QEdu ID <https://www.wikidata.org/wiki/Property:P4344>, WBPLN author ID <https://www.wikidata.org/wiki/Property:P4343>, Store norske leksikon ID <https://www.wikidata.org/wiki/Property:P4342>, synodic period <https://www.wikidata.org/wiki/Property:P4341>, PubsHistory pub ID <https://www.wikidata.org/wiki/Property:P4340>, Organization ID on Radio Radicale <https://www.wikidata.org/wiki/Property:P4339>, LOINC ID <https://www.wikidata.org/wiki/Property:P4338>, digitised page from Lloyd's Register of Ships from 1930 to 1945 <https://www.wikidata.org/wiki/Property:P4337>, Lloyd's Register Ship ID <https://www.wikidata.org/wiki/Property:P4336>, IDESCAT territorial code in Catalonia <https://www.wikidata.org/wiki/Property:P4335>, Heritage Conservation District of Ontario ID <https://www.wikidata.org/wiki/Property:P4334>, GenBank Assembly accession <https://www.wikidata.org/wiki/Property:P4333>, Deutsche Synchronkartei actor-ID <https://www.wikidata.org/wiki/Property:P4332>, Department of Education and Skills roll number <https://www.wikidata.org/wiki/Property:P4331>, contains <https://www.wikidata.org/wiki/Property:P4330>, Wikimedia template <https://www.wikidata.org/wiki/Property:P4329>, Brueckenweb id <https://www.wikidata.org/wiki/Property:P4328>, BHL bibliography ID <https://www.wikidata.org/wiki/Property:P4327>, BFI-Filmography person ID <https://www.wikidata.org/wiki/Property:P4326>, APA phoneme code <https://www.wikidata.org/wiki/Property:P4325>, combination classification <https://www.wikidata.org/wiki/Property:P4324>, young rider classification <https://www.wikidata.org/wiki/Property:P4323>, best sprinter classification <https://www.wikidata.org/wiki/Property:P4322>, best combative classification <https://www.wikidata.org/wiki/Property:P4321>, mountains classification <https://www.wikidata.org/wiki/Property:P4320>, Elite Prospects staff ID <https://www.wikidata.org/wiki/Property:P4319>, Scottish Sports HoF athlete ID <https://www.wikidata.org/wiki/Property:P4318>, GARD rare disease ID <https://www.wikidata.org/wiki/Property:P4317>, kinship equivalent in SPARQL at Wikidata <https://www.wikidata.org/wiki/Property:P4316>, SwimSwam ID <https://www.wikidata.org/wiki/Property:P4315>, SpeedskatingResults.com speed skater ID <https://www.wikidata.org/wiki/Property:P4314>, musée de Bretagne identifiant collections ID <https://www.wikidata.org/wiki/Property:P4313>, camera setup <https://www.wikidata.org/wiki/Property:P4312>, FOIH taxon ID <https://www.wikidata.org/wiki/Property:P4311>, FOIH decree types ID <https://www.wikidata.org/wiki/Property:P4310>, FOIH event types ID <https://www.wikidata.org/wiki/Property:P4309>, FOIH value types ID <https://www.wikidata.org/wiki/Property:P4308>, FOIH heritage types ID <https://www.wikidata.org/wiki/Property:P4307>, FOIH periods ID <https://www.wikidata.org/wiki/Property:P4306>, FOIH styles and cultures ID <https://www.wikidata.org/wiki/Property:P4305>, FOIH materials ID <https://www.wikidata.org/wiki/Property:P4304> - Query examples: - Flags with more than 5 stars <https://query.wikidata.org/#%23defaultView%3AImageGrid%0ASELECT%20DISTINCT%…> (source <https://twitter.com/PoulpyWP/status/918944978231054337>) - Timeline of programming languages <https://query.wikidata.org/#%23defaultView%3ATimeline%0Aselect%20%3Flanguag…> (source <https://twitter.com/SciHiBlog/status/919578602001391617>) - Female scientists sorted by number of citations of their work <https://query.wikidata.org/#SELECT%20%3Fcount%20%3Fauthor%20%3FauthorLabel%…> (source <https://twitter.com/fnielsen/status/917788771235528710>) - Newest WikiProjects <https://www.wikidata.org/wiki/Special:MyLanguage/Wikidata:WikiProjects>: Women <https://www.wikidata.org/wiki/Wikidata:WikiProject_Women> - Newest database reports: Jean Rochefort Filmography <https://www.wikidata.org/wiki/Wikidata:WikiProject_Movies/Filmography/Jean_…> Development - Continued working on persistently storing edits on Lexeme pages (specifically Forms) - Dealt with issues caused by too many edits being injected into recent changes on Commons and Russian Wikipedia - Constraint checks will also be done on qualifiers and references ( phabricator:T176863 <https://phabricator.wikimedia.org/T176863>) - Fixed a regression with the focus not automatically being set after a property was selected (phabricator:T177485 <https://phabricator.wikimedia.org/T177485>) - Full URIs for external identifiers will soon be in the RDF export ( phabricator:T121274 <https://phabricator.wikimedia.org/T121274>) - The secondary wb_entity_per_page table got removed now ( phabricator:T140890 <https://phabricator.wikimedia.org/T140890>) - The secondary wb_terms table does have a term_full_entity_id column now (phabricator:T167114 <https://phabricator.wikimedia.org/T167114>) - Released version 4.0.0 of the Wikibase DataModel JavaScript library <https://github.com/wmde/WikibaseDataModelJavaScript> as a pure NPM package with no MediaWiki dependency (phabricator:T177476 <https://phabricator.wikimedia.org/T177476>) - Released version 3.0.0 of the Wikibase Serialization JavaScript library <https://github.com/wmde/WikibaseSerializationJavaScript> as a pure NPM package with no MediaWiki dependency (phabricator:T177613 <https://phabricator.wikimedia.org/T177613>) - Released version 0.10.0 of the DataValues JavaScript library <https://github.com/wmde/DataValuesJavaScript> as a pure NPM package with no MediaWiki dependency (phabricator:T177232 <https://phabricator.wikimedia.org/T177232>) You can see all open tickets related to Wikidata here <https://phabricator.wikimedia.org/maniphest/query/4RotIcw5oINo/#R>. Monthly Tasks - Add labels, in your own language(s), for the new properties listed above. - Comment on property proposals: all open proposals <https://www.wikidata.org/wiki/Wikidata:Property_proposal/Overview> - Suggested and open tasks <https://www.wikidata.org/wiki/Wikidata:Contribute/Suggested_and_open_tasks> ! - Contribute to a Showcase item <https://www.wikidata.org/wiki/Special:MyLanguage/Wikidata:Showcase_items> . - Help translate <https://www.wikidata.org/wiki/Special:LanguageStats> or proofread the interface and documentation pages, in your own language! - Help merge identical items <https://www.wikidata.org/wiki/User:Pasleim/projectmerge> across Wikimedia projects. - Help write the next summary! <https://www.wikidata.org/wiki/Wikidata:Status_updates/Next> Anything to add? Please share! :) Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

1 0

Fwd: [Analytics] Heads up: mw.track client-side EventLogging mechanism "ignored" certain events
by Erik Bernhardson 12 Oct '17

12 Oct '17

This might explain the unexpectedly low results from the explore similar test. Not saying he fix would ensure usage, but it seems we lost quite a few events. This problem also effects autocompletd data we dashboard, although I don't think we've used it for tests recently. Full text search analytics should be unaffected as they don't use the mw.track functionality. ---------- Forwarded message ---------- From: "Sam Smith" <samsmith(a)wikimedia.org> Date: Oct 12, 2017 3:41 AM Subject: [Analytics] Heads up: mw.track client-side EventLogging mechanism "ignored" certain events To: "Analytics List" <analytics(a)lists.wikimedia.org> Cc: o/ Prior to Thursday, 28th September, if your client-side EventLogging instrumentation logged event via mw.track, then only events tracked during the first pageview of a user's session were logged. Now, technically, the events weren't ignored or dropped. Instead, the subscriber for the "event" topic was never subscribed when the module was loaded from the ResourceLoader's cache and so events published on that topic simply weren't received and logged. This bug was discovered while testing some instrumentation maintained by Readers Web [0] and independently by Timo Tijhof, who submitted the ideal fix [1] promptly. -Sam [0] https://phabricator.wikimedia.org/T175918 [1] https://gerrit.wikimedia.org/r/#/c/378804/ --- Engineering Manager Readers Timezone: BST (UTC+1) IRC (Freenode): phuedx _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

2 1

Discovery Weekly Update for the week starting 2017-10-02
by Chris Koerner 10 Oct '17

10 Oct '17

Hello all, Here is the Discovery weekly update for 2017-10-02. As always, feedback and questions are welcome. ==Highlights== * The week of October 9th, we will deploy code to disable the automatic use of UI message fallback languages for linguistic analysis. In most cases, the messaging fallback language is unrelated to the main language of a wiki (e.g., Wolof falls back to French). A fuller description is on MediaWiki, and you can join the discussion there or on the related Phab ticket. [0] [1] * The Analysis team had several meetings to scope out new work to establish a baseline of metrics for the Structured Data on Commons project. [2] ==Discussions== === Search === * Trey is investigating linguistic analysis software that could be adapted for use on wiki projects. If you know of any open-source linguistic analysis software for languages that don't already have it on their wiki projects, please drop a note on Phab ticket T171652. [3] * David was busy with various tasks: ** relaxing the default "AND" of the retrieval query filter with minimum_should_match [4] ** change loading of cross-wiki configuration to use cirrus-dump-config api call [5] ** finishing up the work to upgrade Elastica to support elastic 5.5.x (will be in production next week [6] ** and adding Special:Search—add additional namespaces to search results for Persian Wikibooks , Hebrew Wikisource, and English Wikibooks [7] [8] [9] * We also stopped the A/B test that was running on 18 wikis with > 1% of search traffic, analysis will follow [10] === Analysis === * Chelsy investigated the full-text search pattern on mobile web and found interesting things: "we think that since people use full-text instead of prefix search results when they have a in-depth information need, and this kind of need is often driven by work/school project, we see more full-text searches on weekdays than weekends on both desktop and mobile web." [11] === Maps === * Paul kicked off the process to regenerate the map tiles in production that will probably finish early next week. We had to do this manually because of a bug in tileshell/node.js that we're working to resolve. [12] [13] [0] https://www.mediawiki.org/wiki/Wikimedia_Discovery/Disabling_Messaging_Fall… [1] https://phabricator.wikimedia.org/T147959 [2] https://phabricator.wikimedia.org/T174519 [3] https://phabricator.wikimedia.org/T171652 [4] https://phabricator.wikimedia.org/T177302 [5] https://phabricator.wikimedia.org/T156497 [6] https://phabricator.wikimedia.org/T174654 [7] https://phabricator.wikimedia.org/T176908 [8] https://phabricator.wikimedia.org/T176907 [9] https://phabricator.wikimedia.org/T176906 [10] https://phabricator.wikimedia.org/T177490 [11] https://phabricator.wikimedia.org/T176815 [12] https://phabricator.wikimedia.org/T176252 [13] https://phabricator.wikimedia.org/T175123 --- Subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison Wikimedia Foundation

1 0

Discovery Weekly Update for the week starting 2017-09-25
by Chris Koerner 02 Oct '17

02 Oct '17

Queridísimos amigos, A slightly more digestible update this week. Particularly of note are the discussions on English and Spanish Wikipedia about enabling the <mapframe> feature. One last reminder: There is a new way to follow these weekly updates.You can subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. Subscribe to be notified! https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly ==Discussions== === Search === * Erik manually speed up the process of moving results (categories) in-between indexes in response to a bug filed for the same search result listed twice in the same query, however, while this type of move is underway, because there are tens of millions of changes, there will be duplicates that show up. [0] * Erik also fixed a bug where the <span> tags were being 'eaten' in the highlighted search preview output [1] * Gehel fixed an issue that was found during the last cluster restart, when the cluster failed to recover properly, by tuning the elasticsearch recovery settings [2] * David and Gehel fixed an issue where the api feature logs were not being sent to both eqiad and codfw clusters [3] * David finished up the large amount of work needed to be done to release the 1.0 version of the ltr (learn to rank) plugin [4] === Analysis === * Chelsy wrapped up the initial analysis of some concerns with the mobile search dashboard data and wrote up next steps [5] === Portal === * Jan and Deb are planning next steps to kicking off automating the Wikipedia.org portal updates [6] === Maps === * Paul has been investigating several issues for switching the geoline/geoshape to a new schema and if we should remove the old kartotherian/render repo [7] [8] * Discussions are continuing for enabling mapframe on English and Spanish Wikipedia. [9] [10] [0] https://phabricator.wikimedia.org/T176683 [1] https://phabricator.wikimedia.org/T173172 [2] https://phabricator.wikimedia.org/T176409 [3] https://phabricator.wikimedia.org/T176430 [4] https://phabricator.wikimedia.org/T175896 [5] https://phabricator.wikimedia.org/T174396#3637453 [6] https://phabricator.wikimedia.org/T140159 [7] https://phabricator.wikimedia.org/T175739 [8] https://phabricator.wikimedia.org/T177114 [9] https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Should_we_… [10] https://es.wikipedia.org/wiki/Wikipedia:Encuestas/2017/Mapas_dinámicos ---- The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Un fuerte abrazo, Chris Koerner Community Liaison Wikimedia Foundation

1 0

Discovery Weekly Update for the week starting 2017-09-18
by Chris Koerner 26 Sep '17

26 Sep '17

Hello, It's been a busy week in Discoveryland. Here are the updates from the Discovery team for last week. As always, feedback and questions are welcome. Reminder: There is a new way to follow these weekly updates.You can subscribe to receive on-wiki (or opt-in email) notifications of the Discovery weekly update. Subscribe to be notified! https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly ==Highlights== * The explore similar language links A/B test has been completed and analysis has been done. Unfortunately, we only had one clickthrough to an article written in a different language (which was displayed in the new language links) as the report documents. We will not be going forward with this feature. [0] ** However, if a user wants to have the language link script added to their logged-in account, please follow these instructions. [1] ** The full explore similar script (displays related articles, categories and language links) can also be enabled for logged-in users, see the instructions. [2] * This latest A/B test (as noted directly above) effectively closes out the additional features that the Discovery Department were exploring to possibly add to the search engine results page (SERP); additional details can be read online [3]; overall A/B testing details can be found on mw.org [4] and self-guided testing instructions as well. [5] ==Discussions== === Search === * After successfully testing and deploying the machine learning to rank model on English Wikipedia [6], we have deployed a new test out to 18 other wikis that have >1% of traffic this week. [7] * For the relevance survey, Erik developed backend infrastructure to support lots of queries and lots of results per query [8] and the third running of the test was turned off this week [9], analysis will be detailed in [10] * The Chinese wiki was re-indexed [11], allowing multi-hyphen tokens to be enabled in production [12] * The Hebrew language wikis were also re-indexed [13] and the HebMorph plugin was also deployed [14] * We updated Vagrant to include new language plugins (Polish, Ukrainian, Chinese and Hebrew) [15] * After some exhaustive investigation, we've resolved the recent load spikes on the elasticsearch cluster in eqiad [16] * Jan has nearly finished the first Selenium test re-written from Ruby to Node.js and has learned a lot in the process. This first test will help to pave the way forward for the rest of the tests that will need to be re-written [17] [18] * We've completed testing for adding support of interleaved search results [19] and currently wrapping up the analysis of the test [20] * Fixed an issue with using mixed versions of the ltr plugin being deployed on elastic1020 [21] * Erik created a few bash scripts to send from terbium when reindexing the default namespaces [xx] (which were moved from general to content indices); this will go into effect when we reindex the wikis again [22] * The second running of the explore similar A/B test for language links was completed on Thursday [23] and analysis is complete [24]; the report can be read online. [25] === Analysis === * Chelsy finalized her work of creating a (mostly) automated and parameterized report template for the Search Platform teams's A/B tests [26] * Chelsy also completed some additional API usage break out (internal vs external) on the metrics dashboard [27] [28] * Chelsy also finalized a new method to keep data longer (that isn't in a dashboard) by adding reports into golden (/srv/published-datasets/discovery) [29] * Mikhail created a dashboard to track the prevalence of sister project search results on fulltext search result pages on desktop, broken up by language. For example, it turns out that nearly 80% of fulltext searches show sister projects on enwiki. [30] === Portal === * Jan has been working on updating the Wikipedia portal, to adjust the languages used for Chinese translations [31] === Maps === * Gehel cleared up some vm space on Horizon by deleting 4 unused maps-team instances [32] * The map service has been upgraded to Node.js 6.11 [33] * Map traffic has been enabled for active / active service (serving map tiles from both data centers) [34] [0] https://analytics.wikimedia.org/datasets/discovery/reports/Explore_Similar_… [1] https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/explor… [2] https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/self-g… [3] https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements [4] https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/Testing [5] https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/self-g… [6] https://phabricator.wikimedia.org/T175772 [7] https://phabricator.wikimedia.org/T175771 [8] https://phabricator.wikimedia.org/T174387 [9] https://phabricator.wikimedia.org/T175047 [10] https://phabricator.wikimedia.org/T174106 [11] https://phabricator.wikimedia.org/T173464 [12] https://phabricator.wikimedia.org/T172653 [13] https://phabricator.wikimedia.org/T167058 [14] https://phabricator.wikimedia.org/T167057 [15] https://phabricator.wikimedia.org/T164367 [16] https://phabricator.wikimedia.org/T169498 [17] https://gerrit.wikimedia.org/r/#/c/378688/ [18] https://phabricator.wikimedia.org/T174103 [19] https://phabricator.wikimedia.org/T150032 [20] https://phabricator.wikimedia.org/T171215 [21] https://phabricator.wikimedia.org/T175951 [22] https://phabricator.wikimedia.org/T176397 [23] https://phabricator.wikimedia.org/T175649 [24] https://phabricator.wikimedia.org/T175650 [25] https://analytics.wikimedia.org/datasets/discovery/reports/Explore_Similar_… [26] https://phabricator.wikimedia.org/T131795 [27] http://discovery.wmflabs.org/metrics/#referer_breakdown [28] https://phabricator.wikimedia.org/T172452 [29] https://phabricator.wikimedia.org/T172453 [30] https://discovery.wmflabs.org/metrics/#sister_search_prevalence [31] https://phabricator.wikimedia.org/T171647 [32] https://phabricator.wikimedia.org/T175998 [33] https://phabricator.wikimedia.org/T171707 [34] https://phabricator.wikimedia.org/T162362 ---- The archive of all past updates can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as "Easy" or "Volunteer needed" in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R Yours, Chris Koerner Community Liaison Wikimedia Foundation

2 1

Logstash not collecting logs while the elasticsearch cirrus cluster is down
by Guillaume Lederrey 21 Sep '17

21 Sep '17

Hello! Related to my previous incident report [1], we also had an issue with logstash [2]. Logstash stops collecting logs while elasticsearch / cirrus is down. This is most probably related to API Feature logging, which are sent by logstash to the cirrus cluster. Sadly, there are no obvious fix at this point. It might be possible to tune the elasticsearch output plugin to fail fast, but that is not obvious from the documentation. [1] https://wikitech.wikimedia.org/wiki/Incident_documentation/20170920-Elastic… [2] https://wikitech.wikimedia.org/wiki/Incident_documentation/20170920-Logstash -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST

1 0

Failed restart of the elasticsearch eqiad cluster
by Guillaume Lederrey 21 Sep '17

21 Sep '17

Hello! TL;DR: Our recent elasticsearch cluster restart did not go as planned. Most important lesson learned: we did not understand the recovery settings correctly. Yesterday, we did a cold restart of the elasticsearch / cirrus eqiad cluster. This restart did not go as planned. It did not generate any user facing impact, since we moved all the traffic to codfw before the restart. It did impact logstash (more of that in a different report). Incident documentation: https://wikitech.wikimedia.org/wiki/Incident_documentation/20170920-Elastic… Have fun! Guillaume -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST

1 0

Mapping Hiragana and Katakana
by Trey Jones 19 Sep '17

19 Sep '17

We recently got a suggestion via Phabricator[1] to automatically map between hiragana and katakana when searching on English Wikipedia and other wiki projects. As an always-on feature, this isn't difficult to implement, but major commercial search engines (Google.jp, Bing, Yahoo Japan, DuckDuckGo, Goo) don't do that. They give different results when searching for hiragana/katakana forms (for example, オオカミ/おおかみ "wolf"). They also give different *numbers* of results, seeming to indicate that it's not just re-ordering the same results (say, so that results in the same script are ranked higher).[2] I want to know what they know that I don't! Does anyone have any thoughts on whether this would be useful (seems that it would) and whether it would cause any problems (it must, or otherwise all the other search engines would do it, right?). Any idea why it might be different between a Japanese-language wiki and a non-Japanese-language wiki? We often are more aggressive in matching between characters that are not native to a given language--for example, accents on Latin characters are generally ignored on English-language wikis. So it might make sense to merge hiragana and katakana on English-language wikis but not Japanese-language wikis. Thanks very much for any suggestions or information! —Trey [1] https://phabricator.wikimedia.org/T176197 [2] Details of my tests at https://phabricator.wikimedia.org/T173650#3580309 Trey Jones Sr. Software Engineer, Search Platform Wikimedia Foundation

1 0

← Newer
1
...
20
21
22
23
24
25
26
...
76
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery