A few notes:
* We had a pretty, multilingual "down for maintenance" page all set up to be served for requests to the site during the downtime, but this was foiled for three or four hours because our offsite DNS in .nl hadn't been actually set up quite the way we thought it was, and our onsite DNS server in Florida was taken down earlier than planned by mistake.
We did get it mostly working for *.wikimedia.org for some people by partway through the downtime, but were not able to get the other domains (such as wikipedia.org) updated at the time. Once Zwinger was back online in the new rackspace, we had DNS again and the downtime page was visible for the remainder of the time, unless you were unlucky and it didn't work anyway.
* The downtime message was experimentally running Lighttpd+FastCGI instead of Apache. For no apparent reason it stopped understanding its 404 error handler page directive some time in the middle of things, so I switched it to Apache.
* The Paris squids were I think still sending requests to the offline Florida machines instead of the downtime page in .nl. Not totally sure what was the issue here.
* When bringing lots of web server machines online we have an issue with synchronization of time and configuration: the machines are set to automatically start the web server on boot, and the load balancers are set to automatically put work on them when they come up. But some machines have clock trouble and come up in the wrong time, and if the configuration has changed they'll have settings out of sync until changed. We need to resolve this; either by requiring a manual start or by some sort of sanity-check against the master clocks and config.
For massively wrong clocks (eg, BIOS reset to 2003) we can easily sanity check by comparing the current time against $wgCacheEpoch to make sure it's later. :)
* Things are very not happy booting without DNS or the LDAP server up. We should make bloody sure this is not as big of a problem; LDAP needs to be well-replicated, and important internal addresses should be resolvable without DNS.
-- brion vibber (brion @ pobox.com)