Brion Vibber wrote in gmane.science.linguistics.wikipedia.technical:
A few notes:
- We had a pretty, multilingual "down for maintenance" page all set up
to be served for requests to the site during the downtime, but this was foiled for three or four hours because our offsite DNS in .nl hadn't been actually set up quite the way we thought it was, and our onsite DNS server in Florida was taken down earlier than planned by mistake.
regarding the downtime page, i've put it in CVS as /tools/downtime/, because we've had such a notice at least twice now, and it seems wise to have a standard, translated message for use in the future.
[...]
- The Paris squids were I think still sending requests to the offline
Florida machines instead of the downtime page in .nl. Not totally sure what was the issue here.
this shouldn't have mattered, should it? the DNS for * was pointing at nl. ... well, in theory.
- When bringing lots of web server machines online we have an issue with
synchronization of time and configuration: the machines are set to automatically start the web server on boot, and the load balancers are set to automatically put work on them when they come up. But some machines have clock trouble and come up in the wrong time, and if the configuration has changed they'll have settings out of sync until changed. We need to resolve this; either by requiring a manual start or by some sort of sanity-check against the master clocks and config.
For massively wrong clocks (eg, BIOS reset to 2003) we can easily sanity check by comparing the current time against $wgCacheEpoch to make sure it's later. :)
i think it should be possible to have apaches do a scap when booting before starting apache, which would remove the problem with outdated PHP files.
-- brion vibber (brion @ pobox.com)
kate.