River Tarnell wrote:
Bryan Tong Minh:
Would it be possible to revoke the drop command from the replication user to prevent this from happening in the future?
Yes. I also considered modifying trainwreck (our replication tool) to ignore DROP DATABASE commands; perhaps we could do both.
It would be easier if these commands wouldn't find their way into the binlog in the first place, but mistakes happen. Unfortunately there's no way to prevent every possible command that might break our database.
In the future I hope to have one MySQL instance per cluster, which would more closely mirror Wikimedia's configuration and hopefully make errors like this less common.
- river.
There are legitimate cases for dropping tables. I think that on getting a DROP command trainwreck should send an email to the admins and halt replication.