On Sat, May 16, 2015 at 4:02 AM, Maarten Dammers <maarten(a)mdammers.nl> wrote: It may not be correct to say "you" here :). If something is really important, it should be run in a way that can handle failure of the underlying hardware. Labs infrastructure wasn't designed for high-uptime of the underlying instances (purposely). If it's production-ish, it should likely either be moved to production or you should put a bit of effort into making it work across multiple instances. The ideal goal is for services to be stateless, with their state living in databases that are also split across instances. It's best to have the service config managed (ideally puppetized since it's what wikimedia uses) so that a loss of an instance is only a brief inconvenience. - Ryan