Ok, so the trouble was that a configuration file was exposed publicly by accident. To fix the problem, the following steps were taken:
0. I stopped all wikimetrics services (queue and web) 1. Coren reset the labsdb password for my user, I copied and replaced it in the db_config.yaml file 2. I reset the wikimetrics user db password and replaced it in db_config.yaml 3. I reset the Flask secret key that guards sessions and replaced it in web_config.yaml 4. I reset the Google OAuth consumer credentials and replaced them in web_config.yaml 5. I did not reset the MediaWiki OAuth consumer credentials as these were not leaked 6. I restarted apache and celery, and wikimetrics started serving again
I'm fairly confident that a reset secret key just means all people who were logged in may have to login again. But there may be something unforeseen that went wrong - just let me know.
On Tue, Dec 10, 2013 at 3:13 PM, Dan Andreescu dandreescu@wikimedia.orgwrote:
Ok, it's back up. Let me know if you have trouble. I'll work on a post-mortem and send it out shortly.
On Tue, Dec 10, 2013 at 2:38 PM, Dan Andreescu dandreescu@wikimedia.orgwrote:
I'm taking wikimetrics down for a bit, I have to reset some passwords that were accidentally leaked. I don't suspect anything bad happened as we caught it within a few minutes.
Dan