Does anybody know anything about this? Did anybody set, fix, or change a clock? Did somebody correct the reported database entries without telling anybody about it? If you did it, please give a whistle so I'll be able to sleep at night again. :)
;-) You know, I like torturing sleepless devs. Anyway, it was kind of our fault. After installing our new boxes, we did fix the clock, but didn't sync it to hardware clock (this operation is done by FreeBSD's crontab, but on Linux it is probably done by reboot/shutdown scripts, which were not invoked by crash). As servers were restarted I didn't catch up too fast with them and there we have, corrupted data. And yes, I didn't think of not synced hardware clock.
Domas
wikitech-l@lists.wikimedia.org