Discovery

discovery@lists.wikimedia.org

1 participants
756 discussions

wdqs1002 - reinstall
by Guillaume Lederrey 11 Apr '16

11 Apr '16

So wdqs1002 is not going well. Long story short: we're going to need to reinstall it. That's a first for me, so I'm going to need some help. I'll ping the Ops side to get the low level stuff, but I'm probably going to need a bit of time from Stas and a few pointers in the right direction. @Stas: expect to hear from me in your morning ... I'll give you all the details tomorrow, but right now I need some sleep... Good night. -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

Wikidata Query Service server not rebooting
by Guillaume Lederrey 11 Apr '16

11 Apr '16

WDQS servers were schedule for a reboot during the weekly deployment window. While the first server rebooted without issue, things did not go as well with the second one. We do not know yet what the issue is, but we are investigating [1]. This has no direct impact to the end user, we can run on a single server, as long as we don't loose it as well... [1] https://phabricator.wikimedia.org/T132387 -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

Discovery Weekly Update for the week starting 2016-04-04
by Chris Koerner 08 Apr '16

08 Apr '16

Hello again, Here is the Discovery department's weekly status update for the week starting 04 April. * CirrusSearch now uses HTTPS for communication between Mediawiki and the Elasticsearch backend. This both increases the privacy of Wikipedia users and helps ensure that we can continue to serve our content even with a major failure of our main datacenter. * April 6, 2016 CREDIT Showcase[edit | edit source] Watch the whole thing <https://www.youtube.com/watch?v=guwVwTRAVsQ> or jump to specific parts: ** Guillaume gives an overview of SonarQube <https://www.youtube.com/watch?v=guwVwTRAVsQ>. ** Max gives an overview if his work to compare diff algorithms in MediaWiki <https://www.youtube.com/watch?v=guwVwTRAVsQ&t=4m45s>. ** Trey gives an update on new functionality in Relevance Forge <https://www.youtube.com/watch?v=guwVwTRAVsQ&t=11m12s>. ** Yuri shows some work he did at the recent Hackathon to store structured data on Wiki <https://www.youtube.com/watch?v=guwVwTRAVsQ&t=19m35s>, and consuming it from Graphs. **More links and Q&A on the CREDIT etherpad <https://etherpad.wikimedia.org/p/CREDIT>. Feedback and suggestions on this weekly update are welcome. The full update, and archive of past updates, can be found on Mediawiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

My news for the week...
by Guillaume Lederrey 08 Apr '16

08 Apr '16

Hello! I just added a short paragraph about the switch of CirrusSearch to HTTPS to our weekly update page [1]. I kept it pretty short as it is mainly an invisible technical change, but still, that's my first real achievement here, so I'm happy to share it. I did not add that I caused a 5 minute outage while deploying this change. I'm not trying to hide (I did hide an incident report [2]), but I don't feel the weekly update is the right place for this kind of communication. I'm happy to add it as well if you think otherwise. Let me know... MrG [1] https://www.mediawiki.org/wiki/Wikimedia_Discovery#Updates [2] https://wikitech.wikimedia.org/wiki/Incident_documentation/20160407-Mediawi… -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

2 1

Re: [discovery] [Ops] CirrusSearch switch to codfw - lessons learned
by Giuseppe Lavagetto 08 Apr '16

08 Apr '16

On Thu, Apr 7, 2016 at 10:24 PM, Bryan Davis <bd808(a)wikimedia.org> wrote: > On Thu, Apr 7, 2016 at 11:23 AM, Guillaume Lederrey > <glederrey(a)wikimedia.org> wrote: >> * labswiki is running on Zend, not HHVM, I need to remember that and >> to try to understand why > > Mostly because we have never tested all of the MediaWiki stack used by > wikitech (labswiki) on HHVM. That wiki is special in several ways > (ldap auth, semantic mediawiki) from any other wiki in the wiki farm. > In theory it should be able to run on hhvm; in practice I wouldn't be > surprised to find out that some extensions it uses have issues on > hhvm. I think that Andrew has a test wiki for wikitech now, so we may > be able to try out hhvm without killing wikitech to do it. > I think that we specifically know that semantic mediawiki and OpenStackManager are not supported by HHVM. G. -- Giuseppe Lavagetto, Ph.d. Senior Technical Operations Engineer, Wikimedia Foundation

1 0

Online / Offline evaluation of Search Engines
by Erik Bernhardson 07 Apr '16

07 Apr '16

JustinO linked these in IRC, and i thought they were quite interesting: http://romip.ru/russir2015/pdf/kanoulas-onoff-evaluation-russir-part-I.pdf http://romip.ru/russir2015/pdf/kanoulas-onoff-evaluation-russir-part-II.pdf It's also nice sometimes to have the talk that goes with the paper. This isn't for the same paper, but is from the same presenter about offline evaluation of search engines: https://www.youtube.com/watch?v=SgrfLmqfR_0

1 0

Re: [discovery] [Ops] CirrusSearch switch to codfw - lessons learned
by Guillaume Lederrey 07 Apr '16

07 Apr '16

On Thu, Apr 7, 2016 at 10:24 PM, Bryan Davis <bd808(a)wikimedia.org> wrote: > On Thu, Apr 7, 2016 at 11:23 AM, Guillaume Lederrey > <glederrey(a)wikimedia.org> wrote: >> * labswiki is running on Zend, not HHVM, I need to remember that and >> to try to understand why > > Mostly because we have never tested all of the MediaWiki stack used by > wikitech (labswiki) on HHVM. That wiki is special in several ways > (ldap auth, semantic mediawiki) from any other wiki in the wiki farm. > In theory it should be able to run on hhvm; in practice I wouldn't be > surprised to find out that some extensions it uses have issues on > hhvm. I think that Andrew has a test wiki for wikitech now, so we may > be able to try out hhvm without killing wikitech to do it. Reasonable explanation. Thanks for enlightening me! > > Bryan > -- > Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> > [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA > irc: bd808 v:415.839.6885 x6855 -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

CirrusSearch switch to codfw - lessons learned
by Guillaume Lederrey 07 Apr '16

07 Apr '16

A few notes about the switch, in no particular order. I probably missed a few points, feel free to add your own. Timeline: 09:35: deploy sizing of HHVM curl named pools 12:22: activating HTTPS + connection pooling, but staying on eqiad 12:37: stop using HTTPS+pooling for labswiki 12:46: point the codfw label back to the codfw cluster 13:22: Fix TTMServer elastic config 13:50: switch CirrusSearch traffic to codfw 13:53: rollback 14:35: switch mw1017 to codfw for CirrusSearch 14:47: switch all CirrusSearch traffic to codfw Issues found and fixed: * labswiki does not run on HHVM >= 3.9.0 but on zend, search is broken. Fixed by an exception in configuration (https://gerrit.wikimedia.org/r/#/c/282145/). Making the CirrusSearch\Elastica\PooledHttp class more robust would be nice, but might not be possible - https://phabricator.wikimedia.org/T132075) * All traffic (including updates) sent to eqiad : copy/paste error in wmf-config/CirrusSearch-production.php, fixed by https://gerrit.wikimedia.org/r/#/c/282147/. We lost some updates, re-indexing in progress. * Issue with TTM configuration broken by the change in CirrusSearch config. Fixed by https://gerrit.wikimedia.org/r/#/c/282154/. * All wikis in error for ~5 minutes. Issue in handling of array in CirrusSearch configuration. Fixed by https://gerrit.wikimedia.org/r/#/c/282163/1. Issues discovered but not fixed: * TTM does not handle multi DC - writes are done only to eqiad - which implies ttmserver index does not exist in codfw - in particular, saves do not work, which makes fixing this a blocker for the switch - dcausse and Nikerabbit seem to have a quick fix in mind - phab task created: https://phabricator.wikimedia.org/T132076 General lessons: * Elasticsearch as measured from mediawiki has a response time increase of ~50[ms] for all query types, except MoreLike queries (which were already sent to codfw). MoreLike queries have a response time decrease of ~15[ms]. Those differences seem to be fairly constant (similar across all percentiles). With a completey biased and unscientific experiment of using wikipedia myself, I feel those differences the most on the autocompletion of the search box. * unit testing configuration is hard, testing it outside of prod is mostly impossible * testing first be deploying manually on our tests servers (mw1017, ...) definitely make sense for all non trivial changes * labswiki is running on Zend, not HHVM, I need to remember that and to try to understand why * I know remember why I like strongly statically typed languages -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

1 0

Re: [discovery] [Ops] Wikidata Query Service (WQDS) regular deployment window
by Greg Grossmeier 06 Apr '16

06 Apr '16

<quote name="Moritz Muehlenhoff" date="2016-03-22" time="22:11:58 +0100"> > On Tue, Mar 22, 2016 at 10:04:21PM +0100, Guillaume Lederrey wrote: > > Let me know if you have any question or if you know of another place > > where I should publicize this deployment window. > > We have a page on wikitech which tracks all deployment/maintenance > windows: https://wikitech.wikimedia.org/wiki/Deployments Yes, please add your window there. As I see you want a 7 hour window, overlapping with other windows shouldn't be a problem. But, if you ever plan to do work that might impact more than just your own systems, please do coordinate with me. Thanks, Greg -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

3 2

Hardware for maps
by Guillaume Lederrey 04 Apr '16

04 Apr '16

Hello! I'm trying to make sure the hardware requests needed for Maps are tracked correctly. What I see so far: * Maps back end hardware [1] - seems to miss a specification of what hardware is required. @Yuri, could you have a look? - should be moving forward as soon as we document what kind of hardware we need. * Maps hardware planning for FY16/17 [2] - fairly high level description of needs to FY16/17 AND FY15/16 * Set up proper edge Varnish caching for maps cluster [3] - describes requirements for proper caching (seems to require 4 machines in each 4 datacenter, so 16 machines) - it seems that it is possible to reuse mobile cache machines, but it is presented as a possibility, unclear if any action has been taken in this direction - I need to dig into this one to see if anything needs to be done Am I missing anything? Note: only [1] is tagged as [hardware-requests]. [1] https://phabricator.wikimedia.org/T131180 [2] https://phabricator.wikimedia.org/T125126 [3] https://phabricator.wikimedia.org/T109162 -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation

2 2

← Newer
1
...
45
46
47
48
49
50
51
...
76
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Discovery