[WikiEN-l] Fwd: [Wikitech-l] Downtime this morning

[David Gerard]

---------- Forwarded message ----------
From: Andrew Garrett <agarrett at wikimedia.org>
Date: 2009/11/16
Subject: [Wikitech-l] Downtime this morning
To: Wikimedia developers <wikitech-l at lists.wikimedia.org>


Hi all,

There has been some downtime this morning (about 15 minutes) due to a
software update.

I pushed a software update, and immediately servers started crashing
according to nagios. Looking at ganglia, it looks like the issue was
the familiar issue where scap pushes a few 4-CPU apaches into swap,
which then crash and come back a few minutes later. This time,
however, obviously a key memcached node fell over, causing a database
overload, resulting in the site being mostly inaccessible for about
ten minutes.

I prepared to revert the software update, but determined that the
problem was not the software update, and a scap would exacerbate the
issue. The problem resolved itself spontaneously.

We need to fix things up so the scap script is less liable to push
machines into swap :)

--
Andrew Garrett
agarrett at wikimedia.org
http://werdn.us/


_______________________________________________
Wikitech-l mailing list
Wikitech-l at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l