On Tue, Nov 19, 2013 at 11:00 AM, Seb35 <seb35wikipedia@gmail.com> wrote:
Le Tue, 19 Nov 2013 14:27:52 +0100, Nikolas Everett <neverett@wikimedia.org> a écrit:

As some of you know we had a chain of CirrusSearch outages and ultimately
rolled all wikis back to lucene search.  We tracked down three out of four
major issues yesterday and I'm planning on spending the morning hunting
down the last one.  We've made a list of bugs we're going to have to take
care of before redeploying Cirrus (
https://bugzilla.wikimedia.org/buglist.cgi?list_id=251172&resolution=---&resolution=LATER&resolution=DUPLICATE&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=cirrus_reenable&product=MediaWiki&product=MediaWiki%20extensions&product=Wikimedia).
Looking at it, I've got at least a week of work there.  Given that WMF
isn't doing any non-emergency deployments next week we're looking at early
December before I bring CirrusSearch back.

Obviously you can't test it in the intervening time and all my "please find
problems by" deadlines are now defunct.  I'll send another update when I
have more information.

Sorry for all the trouble,

It’s better to get this type of problems now, before CirrusSearch is set as primary.


I think in the end we learned a lot from this and the silver lining is how much more documentation and utility scripts I've added over the past day.