We started getting some weird errors from the database for a few
minutes; "lost connection from server" and "syntax errors" on
statements that looked fine.
Checking the mysql and system error logs on geoffrin I found that
mysqld threads were getting killed off due to out-of-memory conditions.
I've lowered the key and innodb buffer sizes (from 384M each to 256M)
and restarted it, hopefully this will be fine.
Also I noticed that the clock on geoffrin has drifted; it's 8 hours and
a little bit off. This may be due to the hardware clock not getting
updated properly after the OS being set to UTC. I haven't recorrected
it yet because I'm not sure if it'll affect anything in the database to
set the time _backwards_ several hours. Timestamps in the db all come
from the webservers, so this shouldn't affect anything on the wiki to
leave it for a bit.
-- brion vibber (brion @
pobox.com)