Brion Vibber wrote:
A combination of a couple configuration errors during restoration of one of our database servers may have corrupted data on the French, Dutch, and Japanese wikipedias, sending updates to the wrong server during at least some portion of the last 24 hours.
I've temporarily locked these three wikis while we work it out.
About 11 hours of updates (from 11:20 to 22:08 UTC) were sent to the wrong server on these three wikis. It appears to have been consistent site-wide so there should be no data loss; we just need to move the data back to the master server and things should be back to normal.
I'm currently backing up all the data on these wikis in preparation for the data move; they should be back online within a few hours with no data lost.
Details:
MySQL has a 'read-only' mode which we enable on our replicated slave servers to protect against misconfigurations where data might be sent to the wrong server. In the rush to get one of our two main servers back online after a disk failure last week, this switch was left out when it was brought online.
A subsequent typo in configuration left three wikis sending updates to that server instead of the primary master, so there were strange inconsistencies depending on whether the false 'master' or one of the regular slave servers was read from. This was manifested as weird edit conflict problems, inconsistent history, etc.
-- brion vibber (brion @ pobox.com)