Slaves out of sync - Wikitech-l

22 Oct 2004

I've just heard that ariel, our master DB server, recently ran out of 
disk space. Writes to the binlogs stopped but it kept committing to its 
local disk, thereby putting all the slaves irreversibly out of sync. 
This is a nasty failure mode that we previously weren't aware was possible.

To resync the slaves, we will have to put the site into read-only mode 
for maybe 6-12 hours. I'm hoping this will be done as soon as possible, 
because in the meantime, we will have poor performance. The slaves used 
to share the read load. Also, if ariel dies, we'll be down to periodic 
backups, meaning data loss and days of downtime.

Search will be disabled until the slaves are resynced, and other 
emergency optimisation measures such as watchlist caching might also 
need to be used.

-- Tim Starling