The Main Page on en: was vandalized yesterday, when a penis image remained on the page for many minutes. It was vandalized again today -- a goatse image remained there for almost /20 minutes/.
Today it happened during a particularly slow time of the morning, around 14:35 UTC, perhaps in combination with other use of the site that slowed it down. It was noticed quickly, but it took a good 17 minutes for it to be successfully deleted once the problem had been announced on IRC, by the seemingly-omniscient Jimmy Wales.
While everyone was fretting over the site's slowness, a few problems presented themselves: * There was no one-click way to remove or delete an image * There was no packaged way to shut down all access to the site in an emergency * There was no packaged way to quickly redirect all visitors (to en:, say) to another site or page * There was no way to bring the site[s] to (or restart the site in) an 'emergency mode' that only allowed limited access (say, by logged-in users) ** Even had there been such a way, there were few (only 1-2) people with shell access who would have been able to run shell scripts, and it took an extra minute or two to get someone's attention. * There were a limited number of ways to reach the collection of devs to let them know there was an emergency.
This was not the worst emergency in the world, so the last point in particular was not as big a deal as it might have been.
=========== Possible solutions:
1) Documentation: write down a standard way to quickly block all incoming requests / take down a site in an emergency / put up in its place a try-back-soon message or redirection to a static snapshot (see 3)
2) Code: add an 'emergency mode' that redirects all visitors to a static read-only snapshot of the site taken once a day
2.1) Code: add a text-only mode that only produces text 2.2) Code: add a one-click (js widget?) option [maybe 2 clicks with some kind of pop-up confirmation that doesn't require rendering another whole WP-page] so that even when the site is very slow, evil images can be deleted in under 15 minutes 2.3) More Code: add a different 'emergency mode' that only allows a limited set of users [logged-in users? users on a specific list?] to use the site.
3) Code + Image Policy: add an IMAGE REVIEW step that imposes a time delay (or requires user approval) before an image can be displayed live on a page [until then the image could still be linked to via an html link]
4) Offer pagers <s>and implantable homing devices</s> to devs who are going to be in the vicinity of computers anyway and are willing to be on-call for certain parts of the day; something more reliable than the blinking of an IRC window. ============
1), 2), and 3) seem important to me. 2) also has useful implications for periods of deep sloth, and for taking things down to make changes. 3) addresses many problems we are having, not just on the main page.
Please comment or suggest implementations.