Discovery October 2016

discovery@lists.wikimedia.org

10 participants
16 discussions

Re: [discovery] [Ops] Using Wikidata Query Service in a production context
by Guillaume Lederrey 13 Oct '16

13 Oct '16

On Thu, Oct 13, 2016 at 11:36 AM, Jaime Crespo <jcrespo(a)wikimedia.org> wrote: > 3) Move the service to labs, not providing any firm guarantee of service > level ? > > Labs is not the place where bad services go to die. Production is the place > where only very stable services reach so they can be properly managed. > > "WDQS do not go through any critical systems" > "all direct clients of WDQS are well protected by circuit breakers" > > Why using the production network, then? > > I think there was one exception, which is services that needed a lot of > resources so they could not run on vms, but don't we have a prototype of > "labs on real hardware"? I'm not sure why WDQS is in the production network (this predates me joining WMF). It is probably there as you suggest for the real hardware needs. There was also probably a wish to make WDQS a production level service with all the availability garanties that goes with it, even if that goal is probably not possible with the current way WDQS works. > Letting user run arbitrary queries is a problem for security, but not in the > common sense (sql injection), but in terms of exactly the situation that you > are describing- running easily out of resources (DOS). Even quarry, which I > have publicly complained about in the past, for what you say, has a better > resource management than wqs (30-minute limit execution, concurrency > control, etc.). (I did not know about quarry, I need to have a look!) The main power of wdqs is that do allow users to write arbitrary queries synchronously. With that power comes the ability to break the service. Removing this ability greatly reduces the value of wdqs. We can (and should) work on putting in place constraints to protect the service, but there are limits to what is possible. I'm pretty sure that whatever we put in place, it will still be possible to break that service (unless we invest crazy amount of time, energy, ...). No, I don't know for sure... > I do not think maps is a problem, as after all it is static tile requests > mostly (the worst it could happen is having a lot of requests)- the only > complain there is that it is constantly creating noise on icinga. But > running an unstable service (wdqs) on top of another unstable service > (wikidata data handling) will never be stable. Everytime a bot starts > writing to wikidata 600 times per second, s5 dbs shake (that is why we are > creating s8) and wqs goes down. :-) I don't think this assumption is true. I have some experience running available services on top of unavailable services. At JOB^1, we did use quite a few external service which were not all that great in term of robustness. Payment processors and credit check services are good example of external notoriously flaky services once you put some load on them. There are strategies to make that work, and in the end taking into account that your dependencies can fail is a great way to build much more robust services. I would even go as far as saying that making sure that your dependencies fail often is a good way to ensure that your system is robust. No, wdqs is not robust enough, but it is something that can (and should) be fixed without changing the way we integrate. > I would suggest using wqs on labs (or anywhere, non-production) with regular > imports rather than real-time updates. Less headaches. I am literally aiming > for that for labsdbs, too. In the specific case of wdqs integration with wikidata, I don't think that the integration pattern itself is wrong (on the fly import of wikidata to wdqs). It does needs some work to improve robustness (https://phabricator.wikimedia.org/T139445 comes to mind). And it does fulfil one of the important use case of wdqs: quite a few wikidata editors use wdqs to live check edits / imports to wikidata. I agree with you that in its current state, WDQS is probably closer to a labs service than to a production service (as far as I understand the definition of labs and production here). The question I'm trying to ask is how do we start using wdqs in a production context. I fully understand that there is work to do here. This is not something that will happen in a few days. But there is value in this idea, so we should start looking at what path we want to take (or make sure that there is no path worth taking, this is a perfectly acceptable answer, as long we look hard enough first). My assumption is that it makes more sense to learn how to integrate low reliability services in a production context than it does to make sure wdqs becomes highly reliable. We should still work on improving wdqs reliability, but we should accept that by its nature it will be less reliable than most of the production services that we have. By the way, thanks Jaime for the great writing! It does help me to structure quite a bit the random thoughts I have between my two ears! > On Tue, Oct 11, 2016 at 10:37 PM, Guillaume Lederrey > <glederrey(a)wikimedia.org> wrote: >> >> Hello! >> >> There is some discussion of starting to use WDQS in conjunction with >> maps and graphs. Here are a few thoughts, just to put them out there >> and to start getting some feedback. This is an attempt to put some >> order in my thoughts, there are not complete yet... >> >> WDQS exposes a SPARQL endpoint to users. This can be compared to >> giving the ability to our users to write arbitrary SQL queries. This >> is fairly close to the concept of the labs replica databases. Giving >> direct access to a SPARQL endpoint is at the same time a wonderful >> idea (it allows users to use WDQS in ways we would never have imagine) >> and a very scary idea (users can write complex queries which will >> consume all resources on our servers - which does happen from time to >> time). >> >> At the moment, WDQS is used by researcher, bots and power users. Those >> users understand this constraint well, and the fluctuation of >> performance of WDQS is not a major issue. >> >> Making WDQS robust enough while letting user run arbitrary queries is >> most probably extremely hard. I think that we should instead >> investigate how to use an unstable service from a stable one. >> >> Ideas... >> >> 1) We can accept service degradation of specific functionalities. We >> accept that WDQS is down, or slow some times. In this case, we degrade >> user experience, graphs will not work, maps will not display data >> layers. In term of implementation, we need to ensure that data flows >> involving WDQS do not go through any critical systems, and that all >> direct clients of WDQS are well protected by circuit breakers. >> >> 2) We want to conserve user experience. We go fully async. Graphs and >> maps are pre-generated and updated regularly outside of user >> interaction. We probably still need synchronous access for editors, to >> allow them to test their edits. Refresh can be relatively low >> frequency (1/day or maybe less). We can probably optimize this based >> on how often a specific graph / map is viewed. I'm not sure how easy >> it would be to scale such an approach... >> >> 3) Something else? >> >> Time to get some sleep... >> >> MrG >> >> -- >> Guillaume Lederrey >> Operations Engineer, Discovery >> Wikimedia Foundation >> UTC+2 / CEST >> >> _______________________________________________ >> Ops mailing list >> Ops(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/ops > > > > > -- > Jaime Crespo > <http://wikimedia.org> -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST

1 0

Using Wikidata Query Service in a production context
by Guillaume Lederrey 11 Oct '16

11 Oct '16

Hello! There is some discussion of starting to use WDQS in conjunction with maps and graphs. Here are a few thoughts, just to put them out there and to start getting some feedback. This is an attempt to put some order in my thoughts, there are not complete yet... WDQS exposes a SPARQL endpoint to users. This can be compared to giving the ability to our users to write arbitrary SQL queries. This is fairly close to the concept of the labs replica databases. Giving direct access to a SPARQL endpoint is at the same time a wonderful idea (it allows users to use WDQS in ways we would never have imagine) and a very scary idea (users can write complex queries which will consume all resources on our servers - which does happen from time to time). At the moment, WDQS is used by researcher, bots and power users. Those users understand this constraint well, and the fluctuation of performance of WDQS is not a major issue. Making WDQS robust enough while letting user run arbitrary queries is most probably extremely hard. I think that we should instead investigate how to use an unstable service from a stable one. Ideas... 1) We can accept service degradation of specific functionalities. We accept that WDQS is down, or slow some times. In this case, we degrade user experience, graphs will not work, maps will not display data layers. In term of implementation, we need to ensure that data flows involving WDQS do not go through any critical systems, and that all direct clients of WDQS are well protected by circuit breakers. 2) We want to conserve user experience. We go fully async. Graphs and maps are pre-generated and updated regularly outside of user interaction. We probably still need synchronous access for editors, to allow them to test their edits. Refresh can be relatively low frequency (1/day or maybe less). We can probably optimize this based on how often a specific graph / map is viewed. I'm not sure how easy it would be to scale such an approach... 3) Something else? Time to get some sleep... MrG -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST

1 0

Discovery Weekly Update for the week starting 2016-10-03
by Chris Koerner 09 Oct '16

09 Oct '16

Howdy, This past week was full of updates from the Discovery department. = Discussions = == Search == * Elasticsearch doesn't offer language analyzers for every language that we have wikis in, so sometimes we fall back to a language analyzer of a (linguistically or geopolitically) related language. This can lead to unexpected effects—such as changes in the Russian language analyzer affecting Ukrainian; or changes needed for Ukrainian maybe having to be made in the Russian language analyzer, unless we special-case them by language (which could affect T146358). [1] If you have any thoughts on this or just want to see the fallback language list, head over to the Fallback Languages page and leave a comment, or reply to the Discovery list. [2] * Translate extension updated to allow searches in a specific language (without translations) [3] * Satisfaction schema update for checkin events on if page is visible [4] * Update the ascii folding to preserve original emits that are duplicated tokens for non ascii characters [5] * Update the search api interwiki to only query when srinterwiki=1 is passed [6] * Reduce the number of replicas for some title suggest indices [7] == Analysis == * Analysis of WDQS geographic breakdown of SPARQL queries [8] * Wikipedia.org Portal Dashboard: investigate spike in languages visited [9] * Wikipedia.org dashboard: minor updates [10] * Bug in current data retrieval script [11] * Add comment to search-related eventlogging in iOS and Android apps to inform Discovery of changes [12] == Portal == (works in progress, needing code review or waiting to be released into production) * Improve color palette [13] * Update for new translations [14] * Update articles by language statistics [15] * Test for adding app links to page [16] * Update WMF logo [17] * Fix where a language link broke on two lines [18] * Few other small updates... == Interactive == * GeoJSON can now contain un-encoded SPARQL (query and ids ExternalData properties): example [19] * There is now "geoshape" and "geoline" types of ExternalData - one is for polygons (e.g. cities, countries), and the other is for lines (roads, rivers). As always, make sure OpenStreetMap has WikidataID attached to be able to use them. * Many minor Kartographer bug fixes [20] = Events and News == * WikiConference North America 2016 is being held in San Diego this weekend and several of the Discovery team will be in attendance. = Other Noteworthy Stuff = * Deb has started integrating slowly into the Discovery Interactive Team as a quasi-Product Manager * Dan and Deb will split up PM duties of the Discovery Search Team in the coming weeks [1] https://phabricator.wikimedia.org/T146358 [2] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Fallback_Langauges [3] https://phabricator.wikimedia.org/T125944 [4] https://phabricator.wikimedia.org/T145102 [5] https://phabricator.wikimedia.org/T146886 [6] https://phabricator.wikimedia.org/T147003 [7] https://phabricator.wikimedia.org/T147192 [8] https://phabricator.wikimedia.org/T143762 [9] https://phabricator.wikimedia.org/T146216 [10] https://phabricator.wikimedia.org/T146215 [11] https://phabricator.wikimedia.org/T146422 [12] https://phabricator.wikimedia.org/T143753 [13] https://phabricator.wikimedia.org/T146231 [14] https://phabricator.wikimedia.org/T143338 [15] https://phabricator.wikimedia.org/T128546 [16] https://phabricator.wikimedia.org/T137495 [17] https://phabricator.wikimedia.org/T144834 [18] https://phabricator.wikimedia.org/T144007 [19] https://www.mediawiki.org/wiki/Help:Extension:Kartographer#GeoShapes_extern… [20] https://phabricator.wikimedia.org/maniphest/query/SYDosTLCk706 [21] https://wikiconference.org/wiki/2016/Main_Page ---- The full update, and archive of past updates, can be found on Mediawiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as Easy or volunteer needed in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R -- Take care, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Fwd: [Wikitech-l] Fwd: [Wikidata] Wikimedia Foundation directly funding Wikidata
by Kevin Smith 05 Oct '16

05 Oct '16

Forwarding. Kevin Smith Agile Coach, Wikimedia Foundation ---------- Forwarded message ---------- From: Pine W <wiki.pine(a)gmail.com> Date: Wed, Oct 5, 2016 at 3:15 PM Subject: [Wikitech-l] Fwd: [Wikidata] Wikimedia Foundation directly funding Wikidata To: wikitech-l(a)lists.wikimedia.org, Wikimedia Mailing List < Wikimedia-l(a)lists.wikimedia.org> Forwarding. Pine ---------- Forwarded message ---------- From: "Léa Lacroix" <lea.lacroix(a)wikimedia.de> Date: Oct 5, 2016 01:20 Subject: [Wikidata] Wikimedia Foundation directly funding Wikidata To: "Discussion list for the Wikidata project." < wikidata(a)lists.wikimedia.org> Cc: Hello folks, An information that you could find useful : since the beginning of Wikidata, the project and the development team (led by Wikimedia Germany) were funded by the Wikimedia Foundation, through the Funds Dissemination Committee, and by third parties. From now, the Wikimedia Foundation will directly fund expenses for Wikidata software development. The two organizations signed an agreement to have direct funding at least for the next 3 years. This is good news, showing the strong support of our project by the Foundation and will allow us more stability in the future. You can read the full blog post here: https://blog.wikimedia.org/ 2016/10/04/supporting-the-future-of-wikidata/ Thanks for making Wikidata more and more awesome every day :) -- Léa Lacroix Community Communication Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 0

Who is doing what in Discovery
by Guillaume Lederrey 05 Oct '16

05 Oct '16

Hello! There has been some questions lately about who is doing what in the Discovery team and who to contact for different kind of issues. We do have some documentation of who the team is [1] and what the different roles mean [2]. This documentation does not cover all use cases, but it is a good start if you are looking for someone in the Discovery team. Please let us know what is missing there and we'll try to improve! Have fun! MrG [1] https://www.mediawiki.org/wiki/Wikimedia_Discovery#The_team [2] https://www.mediawiki.org/wiki/Wikimedia_Discovery/Roles_and_responsibiliti… -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST

1 0

Discovery Weekly Update for the week starting 2016-09-26
by Chris Koerner 03 Oct '16

03 Oct '16

Hello, I hope your day is going well. Here is this week's update from the Discovery department. == Discussions == * Analysis team gave an eventlogging crash course * Chatted about how to help with an effort to create offline pages for Wikipedia; will use the portal page as a test case == Events and News == === Search === * Works in progress but not yet in production: ** File property search functionality merged and documented. As soon as deployment happens (next week), the reindexing will be performed. [1] [2] ** Translate extension updated to allow searches in a specific language (without translations) [3] ** Satisfaction schema update for checkin events on if page is visible [4] ** Update the ascii folding to preserve original emits that are duplicated tokens for non ascii characters [5] ** Update the search api interwiki to only query when srinterwiki=1 is passed [6] === Analysis === * Works in progress but not yet in production: ** Analysis of BM25 A/B test [7] ** Analysis of WDQS geographic breakdown of SPARQL queries [8] ** Expanding display of other countries in portal dashboard [9] ** Analysis of recent spike/upward trend on portal [10] === Portal === * Work in progress but not in production: ** Improve color palette [11] ** Update for new translations [12] ** Update articles by language statistics [13] ** Test for adding app links to page [14] ** Update WMF logo [15] ** Fix where a language link broke on two lines [16] ** Few other small updates [1] https://phabricator.wikimedia.org/T144447 [2] https://www.mediawiki.org/wiki/Help:CirrusSearch#File_properties_search [3] https://phabricator.wikimedia.org/T125944 [4] https://phabricator.wikimedia.org/T145102 [5] https://phabricator.wikimedia.org/T146886 [6] https://phabricator.wikimedia.org/T147003 [7] https://phabricator.wikimedia.org/T143589 [8] https://phabricator.wikimedia.org/T143762 [9] https://phabricator.wikimedia.org/T138107 [10] https://phabricator.wikimedia.org/T146214 [11] https://phabricator.wikimedia.org/T146231 [12] https://phabricator.wikimedia.org/T143338 [13] https://phabricator.wikimedia.org/T128546 [14] https://phabricator.wikimedia.org/T137495 [15] https://phabricator.wikimedia.org/T144834 [16] https://phabricator.wikimedia.org/T144007 == Other Noteworthy Stuff == * Note: we had a code deployment freeze this week due to Ops having an offsite. Much of the work completed above will be released the week of 3 October. ---- The full update, and archive of past updates, can be found on MediaWiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as Easy or Volunteer needed in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery October 2016