Hehe. Yeah, I'm with Carcharoth. Not sure what any of that meant Andrew, but it sounds important. Glad it's back up. Thanks for keeping us informed. Just like there's a certain amount of (anti)prestige associated with being one of the admins who've managed to delete the mainpage http://en.wikipedia.org/w/index.php?title=Special:Log&page=Main_Page is there also a barnstar for being a techie who has unintentionally taken the whole site down? :-) As Brion says "The internet is burning!" -Liam [[witty lama]] wittylama.com/blog Peace, love & metadata On Tue, Nov 17, 2009 at 2:28 AM, Carcharoth <carcharothwp(a)googlemail.com>wrote;wrote:

stevertigo wrote: Actually neither of them are "caching managers" or have any direct role in caching. This isn't the forum to go into a detailed discussion of what they do mean, and a google search would do just as well to fill Carcharoth in, if he was actually interested, which he obviously isn't. This is not particularly accurate either. I think you mean LiveJournal. http://wikitech.wikimedia.org/view/Scap Liam Wyatt wrote: We don't make a big deal of it. Unlike deleting the main page, crashing the site is an easy mistake to make. -- Tim Starling

On Tue, Nov 17, 2009 at 7:25 AM, Carcharoth <carcharothwp(a)googlemail.com> wrote: No, but you didn't. The message was originally posted on wikitech-l, and wasn't meant for wikien-l. It was a brief technical summary of why the site crashed, directed toward people who know about the site architecture and would get useful information from the description. Explaining what all the terms mean wouldn't really serve the purpose of the original message, which was to inform other people knowledgeable about and/or responsible for the site's operation so that the problem could be kept in mind in case it happened again, etc. I don't know what the point was of forwarding it to wikien-l, since it doesn't contain any information that's useful to users. The technical details are, in something closer to laymen's terms: Andrew updated the software (scapped), and some servers crashed. Looking at various monitoring tools (Nagios/Ganglia), he figured out that some of the older, less powerful (4-CPU) application servers (Apaches) didn't have enough resources to handle the update properly (went into swap). Unfortunately, this somehow (I'm not clear on this part) drove an important caching server (memcached node) to crash also. The reduction in caching caused the database to overload, as requests that normally would have been cached had to go to the database. This made the site mostly inaccessible for about ten minutes, until the caches were repopulated enough to reduce database load to normal levels. As you can see, this doesn't really contain any info useful to anyone but server admins. Which is why it was originally posted to wikitech-l, not wikien-l.

Carcharoth wrote: The most important thing is to decide what we are going to cross-post to wikitech-l to induce equal bafflement. Something involving 57 different flavours of idiosyncratic interpretation of IAR, and who thinks that they could be safely ignored, might do. Charles

On Tue, Nov 17, 2009 at 11:11 AM, Charles Matthews <charles.r.matthews(a)ntlworld.com> wrote: The original message was not cross-posted. Andrew posted it on wikitech-l only.

Aryeh Gregor <Simetrical+wikilist(a)gmail.com> wrote: Well if a 17 minute site crash is "no big deal" as Tim said.. Then a little casual informative crossposting ain't worth frettin over, eh? -Stevertigo "You're hopelessly hopeless...