[Foundation-l] Re: [Wikipedia-l] Wiki Problems?

[Brion Vibber]

Brion Vibber wrote:
> James R. Johnson wrote:
>>             Is there something wrong with the wikis?  I was trying to do
>> some writing on ang.wikibooks.org, and ang.wiktionary.org and they don't
>> work.  Are they down right now, or did something else happen?
>
> There was some sort of power failure at the colocation facility. We're
> in the process of rebooting and recovering machines.

The power failure was due to circuit breakers being tripped within the
colocation facility; some of our servers have redundant power supplies
but *both* circuits failed, causing all our machines and the network
switch to unceremoniously shut down.

Whether a problem in MySQL, with our server configurations, or with the
hardware (or some combination thereof), most of our database servers
managed to glitch the data on disk when they went down. (Yes, we use
InnoDB tables. This ain't good enough, apparently.)

The good news: one server maintained a good copy, which we've been
copying to the others to get things back on track. We're now serving all
wikis read-only.

The bad news: that copy was a bit over a day behind synchronization (it
was stopped to run maintenance jobs), so in addition to slogging around
170gb of data to each DB server we have to apply the last day's update
logs before we can restore read/write service.

I don't know when exactly we'll have everything editable again, but it
should be within 12 hours.

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
Url : http://lists.wikimedia.org/pipermail/foundation-l/attachments/20050222/3217db09/attachment-0001.pgp