Ok, so the trouble was that a configuration file was exposed publicly by accident.  To fix the problem, the following steps were taken:

0. I stopped all wikimetrics services (queue and web)
1. Coren reset the labsdb password for my user, I copied and replaced it in the db_config.yaml file
2. I reset the wikimetrics user db password and replaced it in db_config.yaml
3. I reset the Flask secret key that guards sessions and replaced it in web_config.yaml
4. I reset the Google OAuth consumer credentials and replaced them in web_config.yaml
5. I did not reset the MediaWiki OAuth consumer credentials as these were not leaked
6. I restarted apache and celery, and wikimetrics started serving again

I'm fairly confident that a reset secret key just means all people who were logged in may have to login again.  But there may be something unforeseen that went wrong - just let me know.


On Tue, Dec 10, 2013 at 3:13 PM, Dan Andreescu <dandreescu@wikimedia.org> wrote:
Ok, it's back up.  Let me know if you have trouble.  I'll work on a post-mortem and send it out shortly.


On Tue, Dec 10, 2013 at 2:38 PM, Dan Andreescu <dandreescu@wikimedia.org> wrote:
I'm taking wikimetrics down for a bit, I have to reset some passwords that were accidentally leaked.  I don't suspect anything bad happened as we caught it within a few minutes.

Dan