Hi,
one of our database servers, db22, had a disk failure a little while
ago, and while this failed disk was to be replaced another RAID
problem appeared.
This caused downtime of db22 and users started reporting problems at
around 7 pm:
19:09 < malafaya> so, what's wrong?
was even before:
19:19 <+nagios-wm> PROBLEM - Host db22 is DOWN: PING CRITICAL - Packet
loss = 100%
Since this affected CentralAuth, users kept getting error messages
like: [db22: s4] 10.0.6.32
Database ops immediately started moving a database slave to be the new
master, while the hardware issue on db22 is still being investigated.
The current effect is that commons is read-only. The expected downtime
was at 10 minutes when writing this.
--
--
Daniel Zahn <dzahn(a)wikimedia.org>