Update on this. Stefan and I got it running again. Everything seems to be in order but please let us know right away if something's wrong.
As far as the lost data - "Get Data" won't work for past reports. They are in the log files and if you really need the old reports and can't re-run them, I can try to rescue them out of the service log files. I apologize for this nasty inconvenience. We will stress test redis and make sure it behaves well in the future.
Dan
On Wed, Oct 23, 2013 at 6:47 PM, Dan Andreescu dandreescu@wikimedia.orgwrote:
Hi,
I had to reboot the redis server that stores all of your report results today. This uncovered the very unpleasant truth that redis was never backing up properly. Unfortunately, any report results that you had saved are now lost. They can be rerun of course, but that gets me to the second piece of bad news.
Even more frustratingly, I can't seem to get redis to come back up again. We are working on this problem and will try to fix it ASAP. I will post here once it's fixed.
Dan