Ok, so the trouble was that a configuration file was exposed publicly by
accident. To fix the problem, the following steps were taken:
0. I stopped all wikimetrics services (queue and web)
1. Coren reset the labsdb password for my user, I copied and replaced it in
the db_config.yaml file
2. I reset the wikimetrics user db password and replaced it in
3. I reset the Flask secret key that guards sessions and replaced it in
4. I reset the Google OAuth consumer credentials and replaced them in
5. I did not reset the MediaWiki OAuth consumer credentials as these were
6. I restarted apache and celery, and wikimetrics started serving again
I'm fairly confident that a reset secret key just means all people who were
logged in may have to login again. But there may be something unforeseen
that went wrong - just let me know.
On Tue, Dec 10, 2013 at 3:13 PM, Dan Andreescu <dandreescu(a)wikimedia.org>wrote;wrote:
Ok, it's back up. Let me know if you have
trouble. I'll work on a
post-mortem and send it out shortly.
On Tue, Dec 10, 2013 at 2:38 PM, Dan Andreescu <dandreescu(a)wikimedia.org>wrote;wrote:
I'm taking wikimetrics down for a bit, I have
to reset some passwords
that were accidentally leaked. I don't suspect anything bad happened as we
caught it within a few minutes.