On 6/3/07, Daniel Cannon &lt;cannon.danielc(a)gmail.com&gt; wrote: </melodrama> As an update, things do seem to be improving a bit. In any case, it's not a *critical* situation at this point. The databases are still lagging quite a bit--four hour lags on usercontributions of course the most noticeable. Server lag seems to have cleared itself up considerably. Still would be nice if a sysadmin could perchance restart the db servers or look at (and preferably fix :D) the problem. Thanks. -- Daniel Cannon (AmiDaniel) http://amidaniel.com cannon.danielc(a)gmail.com

Daniel Cannon wrote: I think we should have an escalation procedure for downtime that involves calling the Wikimedia office as a first step. Out of hours, a recorded message would give a special number to call which is diverted to, say, the mobile phone of a sysadmin. I was awake at the time, but busy with non-Internet stuff. I could have responded to a request by phone at any time. I fixed the problem as soon as I by chance turned on my monitor and saw the complaints on IRC. Brion arrived at the same time, presumably someone had called him. Last time I gave out my mobile number to a group of active Wikipedians, a member of that group turned troll and posted that number on Wikipedia in an attempt to get back at me for blocking him. Nobody ever called, but still, I'm slightly hesitant to do it again. Having a separate diverted number instead of distributing sysadmin contact details means we can implement call screening, time of day restrictions, etc. should the need arise. -- Tim Starling