Discovery

discovery@lists.wikimedia.org

1 participants
756 discussions

Instabilities in Wikidata Query Service
by Guillaume Lederrey 03 May '16

03 May '16

This is a bad time for Wikidata Query Service. We had a few issues last week to update WDQS and reload all data. During this data reload, we are running on a single server. And now this server is starting to behave erratically. I honestly do not know yet what is happening except that we see a lot of file (pipes actually) left open and that WDQS stops responding. Investigation is going on [1] and we'll let you know what we find. Sorry for the inconvenience and thank you all for your patience! [1] https://phabricator.wikimedia.org/T134238 -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

Instabilities of Elasticsearch codfw cluster
by Guillaume Lederrey 03 May '16

03 May '16

We have recently experienced multiple instabilities of our elasticsearch cluster in codfw. First one on 2016-04-27 identified around 10am UTC, second one on 2016-05-02 identified around 11pm UTC. In both cases the symptoms were similar: * a cluster restart was in progress to modify cluster discovery strategy (moving from multicast to unicast [1]). * cluster wide operations (get list of nodes, shards, changing cluster settings) were extremely slow * number of pending tasks (`curl -s localhost:9200/_cluster/health?pretty | jq .number_of_pending_tasks`) was high * most of those tasks were deletion of shards (`curl -s localhost:9200/_cat/pending_tasks | grep indices_store | wc -l`) * a few of them were deletion and creation of the "mediawiki_cirrussearch_frozen_indexes" index * response time of client requests did not seem to be affected Only "more like" traffic is going to codfw. On the 27, we switched this traffic to eqiad to buy time for investigation and recovery [2]. We found a copy/paste error in the cluster discovery configuration [3]. After a full cluster restart the situation stabilized. On the 2, we did further investigation by capturing traffic and looking for the cause of the deletion of the "mediawiki_cirrussearch_frozen_indexes" index. We saw a high number of deletion and creation requests for this index, coming from mediawiki job runners. Further investigation show that Elastica (the library used by CirrusSearch to communicate with Elasticsearch) does a recreate (deletion followed by creation) to create a new index [4]. This was fixed quickly by disabling index creation [5][6]. A more permanent fix is tracked [7]. We suspect that this issue was seen only on codfw because of the latency increasing the probability of a race condition between 2 re-create operations. Lessons learned: * index creation / deletion can bring the cluster to its knees * despite that, Elasticsearch is robust, client request do not seem to have been affected * the number of pending tasks raising seems to be a god indication of cluster issues [8] Big thanks to David and Erik for their support in this issue! [1] https://phabricator.wikimedia.org/T110236 [2] https://gerrit.wikimedia.org/r/#/c/285620/ [3] https://gerrit.wikimedia.org/r/#/c/285612/ [4] https://github.com/ruflin/Elastica/blob/master/lib/Elastica/Index.php#L238 - not entirely sure about this reference [5] https://gerrit.wikimedia.org/r/#/c/286541/ [6] https://gerrit.wikimedia.org/r/#/c/286542/ [7] https://phabricator.wikimedia.org/T133793 [8] https://phabricator.wikimedia.org/T134240 -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

How might interactive maps be used on Wikipedia?
by Chris Koerner 02 May '16

02 May '16

Hello, The Maps team at the Wikimedia Foundation is getting closer to make it possible to add interactive maps <https://www.mediawiki.org/wiki/Maps> to Wikipedia. If you've ever used services like Google Maps or Mapquest you may be familiar with interactive maps. We’d like to invite editors to have a conversation on how these maps might be used within articles. We've put together information on how these maps and their style works from a technical perspective <https://www.mediawiki.org/wiki/Maps/Conversation_about_interactive_map_use> – where the data comes from, how maps are styled, how to add an interactive map, and a few example use cases. In particular we would like to focus the discussion around three key questions (open discussion outside these questions is welcome too). * What types of articles would use interactive maps? * How do these articles differ in their requirements? * Are there any classes of articles whose map styling requirement is fundamentally in conflict with other article classes, thus requiring multiple styles? If you are interested, please visit https://www.mediawiki.org/wiki/Maps/Conversation_about_interactive_map_use to learn more and get involved. -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Understanding Maps
by Guillaume Lederrey 02 May '16

02 May '16

As promised, I started to dig in to the Maps documentation [1] and started to write some diagrams. The source of those diagram is at the end of this email, you can process that source online if you want the pretty pictures [2] (well, not that pretty). I should probably post those diagrams somewhere for discussion, but not sure where it make sense... [1] https://wikitech.wikimedia.org/wiki/Maps [2] http://www.planttext.com/planttext **Maps component diagram:** @startuml [Varnish] [Kartotherian] [Tilerator] [TileratorUI] [OSM] database Cassandra database Redis database Postgres OSM -> Postgres: import maps data Tilerator -left-> Redis: consume job queue Tilerator --> Postgres: get data to pre-generate\nvector tiles Tilerator -> Cassandra: stores pre-generated\nvector tiles TileratorUI ..> Redis: schedule jobs to\npre-generate tiles Kartotherian --> Cassandra: serve tiles\nto end user Varnish --> Kartotherian @enduml **Maps deployment diagram:** @startuml () "Maps\n(public)" as mapsP package codfw { package "Maps cache cluster" as cache { node cp2003 { [varnish-frontend] as vfe2003 [varnish-backend] as vbe2003 } node cp2009 { [varnish-frontend] as vfe2009 [varnish-backend] as vbe2009 } node cp2015 { [varnish-frontend] as vfe2015 [varnish-backend] as vbe2015 } node cp2021 { [varnish-frontend] as vfe2021 [varnish-backend] as vbe2021 } } () "Maps\n(internal)" as mapsI node "maps-test2001\n(master)" as maps2001 { [Kartotherian] as Kartotherian2001 [Tilerator] as Tilerator2001 [TileratorUI] as TileratorUI2001 database Cassandra as Cassandra2001 database Redis as Redis2001 database "Postgres\nmaster" as Postgres2001 Tilerator2001 -left-> Redis2001 Tilerator2001 --> Postgres2001 Tilerator2001 -> Cassandra2001 TileratorUI2001 --> Redis2001 Kartotherian2001 --> Cassandra2001 } node "maps-test2002-4\n(slaves)" as maps2002 { [Kartotherian] as Kartotherian20xx [Tilerator] as Tilerator20xx [TileratorUI] as TileratorUI20xx database Cassandra as Cassandra20xx database Redis as Redis20xx database "Postgres\nslaves" as Postgres20xx Tilerator20xx -left-> Redis20xx Tilerator20xx --> Postgres20xx Tilerator20xx --> Cassandra20xx TileratorUI20xx -> Redis20xx Kartotherian20xx --> Cassandra20xx } mapsI - Kartotherian2001 mapsI - Kartotherian20xx vbe2003 -> mapsI vbe2009 -> mapsI vbe2015 -> mapsI vbe2021 -> mapsI vfe2003 --> vbe2003 vfe2009 --> vbe2009 vfe2015 --> vbe2015 vfe2021 --> vbe2021 ' un comment the block below to have the mostly complete Varnish connections ' vfe2003 --> vbe2003 ' vfe2003 --> vbe2009 ' vfe2003 --> vbe2015 ' vfe2003 --> vbe2021 ' ' vfe2009 --> vbe2003 ' vfe2009 --> vbe2009 ' vfe2009 --> vbe2015 ' vfe2009 --> vbe2021 ' ' vfe2015 --> vbe2003 ' vfe2015 --> vbe2009 ' vfe2015 --> vbe2015 ' vfe2015 --> vbe2021 ' ' vfe2021 --> vbe2003 ' vfe2021 --> vbe2009 ' vfe2021 --> vbe2015 ' vfe2021 --> vbe2021 mapsP -- vfe2003 mapsP -- vfe2009 mapsP -- vfe2015 mapsP -- vfe2021 Postgres20xx <- Postgres2001 Cassandra20xx <-> Cassandra2001 Redis20xx <-> Redis2001 note right of vfe2003 interconnections between Varnish frontend and backend are more complex, not showing all this here. end note note right of mapsI Need to check if this is a LVS endpoint or if Varnish talks directly to Mapsf end note } note as n1 - unsure about what communication there is between maps-test nodes end note @enduml -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

Discovery Weekly Update for the week starting 2016-04-25
by Chris Koerner 29 Apr '16

29 Apr '16

Howdy, A few updates on the work of the Discovery team this week. Thanks to those who sent something in, and thank you for reading. * David, Erik, and Trey had a discussion with JustinO about improving recall and improving search in general. Semi-readable notes are in an etherpad <https://etherpad.wikimedia.org/p/Recall>. Additional thoughts and comments are welcome. * Deb and Moiz had discussions with Abbey, Daisy and Edward about upcoming surveys planned for the Wikipedia portal; awaiting Legal approval. * The updated Perl version of TextCat is now available on GitHub <https://github.com/Trey314159/TextCat>. Reminder: the PHP version <https://github.com/wikimedia/wikimedia-textcat> has been available for a while. * Analysis has been completed <https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/TextCat_Optimization…> on low-performing [search] queries (< 3 results) on French, Spanish, and Italian Wikipedias, to optimize performance on language identification with TextCat on those wikis. German is coming up next. * Analysis is complete on recent Wikipedia.org portal page A/B test: Wikipedia Portal Test of Language Detection and Primary Link Resorting <https://commons.wikimedia.org/wiki/File:Wikipedia_Portal_Test_of_Language_D…> * Maps servers now have 16 varnish servers instead of 2 in 4 different data centers instead of 1. * A new Wikipedia portal A/B test will be released next week - it will add in descriptive text to the sister project links at the bottom of the page. * Search for a new analyst (Oliver's replacement) is going along well. We got 169 applicants and some of them have done or are scheduled to do the take-home analysis task. ---- Feedback and suggestions on this weekly update are welcome. The full update, and archive of past updates, can be found on Mediawiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Fwd: [discovery-private] Fwd: Lets play Name That Thing!
by Dan Garry 29 Apr '16

29 Apr '16

Forwarding this email, which accidentally went to the internal Discovery list. We decided on "Discernatron". :-) Dan ---------- Forwarded message ---------- From: Erik Bernhardson <ebernhardson(a)wikimedia.org> Date: 21 April 2016 at 14:18 Subject: Re: [discovery-private] Fwd: [discovery] Lets play Name That Thing! To: Internal communications for WMF search and discovery team < discovery-private(a)lists.wikimedia.org> Seems like Discernatron is the winner! I've create the repo at https://gerrit.wikimedia.org/r/wikimedia/discovery/discernatron and pushed the current state of the code there. I'll update any references in the code base and get a new version (with lots of other small updates i made yesterday as well) up sometime today. We also have a filtered set of queries that me and Trey agreeded on, and OAuth credentials to use with meta.mediawiki.org for logins. One of the last sticking points for getting this pushed out is coming up with good instructions for users so they give us good information. Still debating :S I'll initially put this up today or tomorrow with a dozen queries we use for testing and ask people to try it out and let me know what can be fixed for a roll out + small announcement next week. -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation

2 4

Discovery standup today - cannot attend
by Guillaume Lederrey 28 Apr '16

28 Apr '16

Hello! I'll be in the "Writing a Self-Review" workshop today. You said so much good about it that I can't miss it. It is in conflict with our standup so here is my status: * codfw elasticsearch cluster has been behaving erratically yesterday. This is probably related to a copy/paste error (by myself) in the unicast configuration (I mixed up eqiad and codfw). It does not completely explain why we started seeing issues only after the change was deployed to all but one server for > 12 hours. In the end, a full cluster restart fixed the issue, but we still don't really understand it. * started the restart of eqiad elasticsearch cluster (please all cross your fingers!) * WDQS data reload: in progress. I did not check properly that the latest version was deployed before starting the data import, so we actually loaded it with the wrong version (so geo indexing not yet enabled). Another data reload will be required (already in progress on wdqs1002, will do it afterward on wdqs1001). * Spent quite some time going through all the phabricator issues I am subscribed to and gerrit changes, did some cleaning. * new elasticsearch servers are almost ready to be installed, I spent some time understanding how installing them actually works. Will probably start that tonight or tomorrow. -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

Portals....and portals
by Kevin Smith 27 Apr '16

27 Apr '16

I just encountered this page and concept: https://en.wikipedia.org/wiki/Portal:Contents/Portals As a result, I'm wondering if we should stop referring to https://wikipedia.org as "the portal"? Kevin Smith Agile Coach, Wikimedia Foundation

2 1

Discovery Weekly Update for the week starting 2016-04-11
by Chris Koerner 22 Apr '16

22 Apr '16

Hello, A few updates from the Discovery department this week. * Portal team brainstormed ideas for the future of the Portal work and beyond; read more here <https://www.mediawiki.org/wiki/Wikipedia.org_Portal_brainstorming_ideas_for…> . * The zero results rate <http://discovery.wmflabs.org/metrics/#kpi_zero_results> [for search] was updated to no longer count queries that it should've been excluding. This shows that the completion suggester had a bigger effect than previously thought, actually reducing the zero results rate from roughly 33% to roughly 22%. ---- Feedback and suggestions on this weekly update are welcome. The full update, and archive of past updates, can be found on Mediawiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Village pump discussion about Wikipedia search improvements
by Tilman Bayer 22 Apr '16

22 Apr '16

Might be of interest: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(proposals)#Watson_and… -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB

2 1

← Newer
1
...
43
44
45
46
47
48
49
...
76
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery