Brion Vibber wrote: The power failure was due to circuit breakers being tripped within the colocation facility; some of our servers have redundant power supplies but *both* circuits failed, causing all our machines and the network switch to unceremoniously shut down. Whether a problem in MySQL, with our server configurations, or with the hardware (or some combination thereof), most of our database servers managed to glitch the data on disk when they went down. (Yes, we use InnoDB tables. This ain't good enough, apparently.) The good news: one server maintained a good copy, which we've been copying to the others to get things back on track. We're now serving all wikis read-only. The bad news: that copy was a bit over a day behind synchronization (it was stopped to run maintenance jobs), so in addition to slogging around 170gb of data to each DB server we have to apply the last day's update logs before we can restore read/write service. I don't know when exactly we'll have everything editable again, but it should be within 12 hours. -- brion vibber (brion @ pobox.com)