On Tue, Nov 19, 2013 at 12:43 PM, Federico Leva (Nemo) <nemowiki@gmail.com> wrote:
Seb35, 19/11/2013 17:00:

Itís better to get this type of problems now, before CirrusSearch is set
as primary. Thanks for the very progressive deployment (secondary motor
and opt-in wikis).

I was surprised to see that CirrusSearch had apparently "swallowed" million-pages wikis without a glitch; it's nice to see issues were nailed down like that (though it's not nice for those who have to fix them).

We're back now!† We fixed the problem that caused the outage two weeks ago.† We've added code to prevent glitches in the underlying infrastructure form effecting editing.† We've verified that if the infrastructure is half crippled then we'll log warnings at WMF but we'll still serve as many search results as possible.† Hopefully we won't break things again like we did two weeks ago, but if we do then we're ready.

We've reenabled Cirrus as a secondary for all wikis to which it was deployed before the outage.† None have it as primary at this point.† Here is the table of deployments from https://www.mediawiki.org/wiki/Search#Wikis:
Wiki Secondary Date Primary Date

mediawikiwiki December 3 (first ???) Unscheduled (first September 11)
itwiktionary December 3 (first September 16) Unscheduled (first September 23)
disabled December 3 (first September 23) Unscheduled (first September 24)
cawiki December 3 (first October XX) Unscheduled (first September 23)
enwikisource December 3 (first October 15) Unscheduled (first September 23)
bnwiki December 3 (first October 23) Unscheduled
wikivoyages December 3 (first October 23) Unscheduled
sewikimedia December 3 (first November 5) Unscheduled
astwiki December 3 (first November 5) Unscheduled
guwiki December 3 (first November 5) Unscheduled
elwiki December 3 (first November 5) Unscheduled
frwikisource December 3 (first November 5) Unscheduled
nlwiki December 3 (first November 12) Unscheduled
itwiki December 3 Unscheduled
plwiktionary December 3 Unscheduled
wikimedias Unscheduled Unscheduled
wikimanias Unscheduled Unscheduled
wikisources Unscheduled Unscheduled
wiktionaries Unscheduled Unscheduled

All wikis are fully ready except itwiki, nlwiki, and enwikisource.† All of the content for those wikis is indexed but the incoming links are being counted as I sent this email.† Until they are counted the number of links is assumed to be 0 which will cause uncounted articles to potentially be sorted under counted articles.† This process should be finished in a few hours.

Otherwise everything is working well now.† I'd like to switch mediawiki, itwiktionary, all the disabled wikis, cawiki, and enwikisource back to running Cirrus as primary soon.† If you are related to any of those wikis can you recheck Cirrus by adding the url parameter (srbackend=CirrusSearch)?† I'd love that.

I'd like to restart that two week timer.† That means that I'll request release windows to switch everything from bnwiki to plwiktionary after December 17th.† Please give Cirrus a shot between now and then and let me know if you find anything that would make it worse than what the wiki is running now.† You can test it by performing a normal search and then adding &srbackend=CirrusSearch to the end of the URL.† At some point in the next few days we'll deploy Cirrus as a beta features option as well.

Finally, a peace offering:† https://www.mediawiki.org/wiki/Search/CirrusSearchFeatures#prefer-recent: