We had a serious MySQL crash on suda with associated data corruption today (6/6). There's a summary of the events leading up to the crash at http://openfacts.berlios.de/index-en.phtml?title=Wikipedia_plans
Whether it was the kill -9 that led to the corruption or whether the database was already corrupted and did therefore not respond we do not know; in any case, there seem to have been no alternatives to killing it (people on #mysql knew nothing either).
Shaihulud made a copy of the CUR tables from all wikis earlier today and imported it on Ariel. We've switched the live wikis to readonly from Ariel; readonly because Ariel doesn't have OLD and lots of other stuff, because it's not sufficiently tested, and because we'd like to prevent any data loss if possible.
Tim created a special "readonly" user on ariel for this purpose.
The following have been changed as long as we are in readonly mode: 1) Counters disabled on all wikis 2) Linkscc disabled 3) readonly file set to /home/wikipedia/common/readonly 4) user_newtalk disabled 5) $wgDatabaseServer and $wgDBuser changed to ariel
There will still be lots of error messages and because the OLD tables are not on Ariel revision histories are missing etc. This is *only* to make sure that people can read our articles.
The next step is to fix the data corruption on suda.
Regards,
Erik