Bryan Tong Minh:
Would it be possible to revoke the drop command
from the replication
user to prevent this from happening in the future?
Yes. I also considered modifying trainwreck (our replication tool) to
ignore DROP DATABASE commands; perhaps we could do both.
It would be easier if these commands wouldn't find their way into the
binlog in the first place, but mistakes happen. Unfortunately there's
no way to prevent every possible command that might break our database.
In the future I hope to have one MySQL instance per cluster, which would
more closely mirror Wikimedia's configuration and hopefully make errors
like this less common.
- river.
There are legitimate cases for dropping tables. I think that on getting
a DROP command trainwreck should send an email to the admins and halt
replication.