This morning a bug was filed about some pages showing timestamps in the
year 2025:
http://bugzilla.wikimedia.org/show_bug.cgi?id=2138
By the time I was awake and looked at them, there was no trace:
everything said 2005 as expected.
A few minutes ago somebody piped up on IRC about another one on
fr.wikinews, which I was able to confirm. I fixed that one item, and
recorded this fact in the server admin log
(
http://wp.wikidev.net/Server_admin_log#11_May)
There are basically two ways this could happen: either some machine had
its clock incorrectly set to 2025 instead of 2005, and was subsequently
corrected, or there's a horrible mysterious bug that corrupts one
particular digit of the timestamp sometimes.
I can't find any trace of a machine with its clock incorrectly set now.
A few of the earlier machines (srv2-srv4) had their hardware clocks set
in the year 2003, but the live system clocks were correct on current
time. (I've synchronized the hardware clocks in the hopes they will boot
up right next time, too). If there was a machine with its hardware clock
set to 2025, it's either not listed in the Apache node group -- in which
case we have a rogue machine we can't account for and can't control
through the regular means -- or it was corrected and nobody recorded
this fact.
If we had a machine with correct hardware clock time that somehow got
the wrong system time temporarily, well who knows...
Does anybody know anything about this? Did anybody set, fix, or change a
clock? Did somebody correct the reported database entries without
telling anybody about it? If you did it, please give a whistle so I'll
be able to sleep at night again. :)
-- brion vibber (brion @
pobox.com)