[Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24

[Tim Starling]

On 25/05/11 22:27, Strainu wrote:
> What I understood from this thread is: if you have a planned
> maintenance windows between 13 and 14 GMT, it would be appreciated if
> you could:
> - create a simple page that says: "We are working on our servers
> between 13 and 14 GMT and Wikipedia might be unavailable during that
> time"
> - replace the usual error message with the newly created page as close
> as possible to 12:59
> - reinstate the usual error message at 14:01 (or whenever the maintenance ends)

There are dozens of places where error messages are generated. It's
not trivial to replace them all. Some of them are hard-coded in
compiled binaries, some are on the client side.

The error message in question comes from DBConnectionError in
Database.php in the MediaWiki source. It's hard-coded and the source
would have had to have been patched. Since no database problems were
anticipated, even if we had tried to implement your plan, we wouldn't
have thought to patch Database.php, and the result would have been the
same.

> Nobody (of the millions of anonymous users) really cares about whether
> a certain db server is down or up at 13:49, or some router is
> rebooting at 13:23. They just wanna know when they can come back to
> read about spark plugs (sic!).

There was no way to tell when the site was going to be back up, except
perhaps after the problem was isolated and the fix was halfway through
being implemented. But by that time there was only a few minutes of
downtime left. The maintenance window was 13:00 to 14:00, but after
things went wrong, there was no guarantee that all problems would be
fixed by 14:00.

Indeed, if it wasn't for Domas's help as a volunteer sysadmin, the
problem may have lasted much longer. Then there would have been plenty
of time for messaging and maybe we wouldn't be having this conversation.

-- Tim Starling