A couple of days ago Ariel, the database master for en.wikipedia, crashed in
some way for a brief time. This was not logged in detail in the admin log, so I
don't really know what happened as I was out at the time.
Today, when applying a full PHP update and sync to the software, en.wikipedia
suddenly began to display a read-only message about being locked due to a server
crash. It was determined that the database configuration file had been edited,
but then not synchronized, during the earlier crisis. The read-only message was
removed and the file resynchronized, opening the wiki back for editing.
Unfortunately, there was a combination of two other problems:
1) The config file also had ariel commented out, presumably to avoid error
messages during the temporary crisis two days ago. As a result, the next server
in the list was considered to be the master by the software.
2) The database slaves for en.wikipedia were misconfigured. All slaves *MUST* be
kept in read_only mode or there is a high risk of data corruption in the case of
wiki configuration errors.
They were *not* set in read_only mode, so db4, one of the slaves, ended up
accepting edits for 40 minutes.
Further, edits went back on ariel for several minutes while figuring out what
had happened.
en.wikipedia is currently locked while we examine the databases to see whether
recovery of the last 40 minutes' work is feasible.
-- brion vibber (brion @
pobox.com)