Sean what came of your discussion with Coren about limiting time or memory of queries? I think we should totally start enforcing those kinds of limits as it seems any queries running longer than a few days are usually accidents.
That discussion (ongoing) pertains to labsdb replicas. These queries are on s1-analytics-slave, which I think would deserve a different, more flexible, approach? Certainly the mechanism to kill queries based on rules, exists.
Another option might be an icinga alert sent only to you guys? Or perhaps to analytics + otto + gage + me?