Hi everyone,

I just posted a notehttp://blog.wikimedia.org/2011/11/18/nobody-notices-when-its-not-broken-new-database-servers-deployed/on the blog about our new external store but wanted to add a few details here.  The deploy went smoothly, and I'm very happy with how the project progressed overall.  There are plenty more details on the project itself on the project wiki pagehttp://wikitech.wikimedia.org/view/External_storage/Update_2011-08and hiding in RT.  there were a few followup things to come out of it, and I want to talk through those in hopes that someone either picks them up or has suggestions on what to do.

From what I've read on the blog, that sounds like a "Good Work

everyone" is in order :)

During the deploy there was a brief (about 10 minute) period during which article saves failed due to the external store databases being in read-only mode.  As expected, some folks showed up in IRC telling us of the 'problem'.  After the migration was complete we brainstormed a bit in IRC about good ways of informing editors of planned maintenance such as this migration.  The regular databases (s3, etc.) have a read-only mode flag so that the affected wikis show a reasonable error, but the external store databases are a little different.  Because of the way they're spread out, the outage of a specific database cluster does not affect specific language projects, but instead affects a specific time range for all wikis. Additionally, the currently writable external store database affects article edits on all wikis.

There were a few suggestions thrown around:

1. use central notice.  This would certainly have the effect of alerting

all wikis that there was some maintenance, but it has the disadvantage of telling all *readers* about the outage, rather than only the people that would actually be interested (those editing pages).

--^ Seems to be the most reasonable thing to do. Readers who don't care, probably don't read sitenotices that don't involve giant pictures of everyone's favourite Wikipedia founder, and even if they do read it, I'm sure they'd ignore it. A small maintenance notice is not likely to annoy people. Saves not working is likely to annoy people. Saves not working after you spent an hour revising whatever article you're writing will definitely frustrate people.

2. make mediawiki cache the change to conceal the outage from editors.  The

idea here is that mediawiki would notice that the backend database is currently in read-only mode and would cache the change and write it to the DB when it returns to read-write mode.  There are a number of technical challenges here, as well as the introduction of another system (the change cache), but it's an interesting way around the problem, since rather than addressing how to inform editors of impending maintenance it simply eliminates the necessity for that communication.

Sounds (very) needlessly complex. Not to mention if it partially fails and people edits go through, then un-go through, people won't be happy.

3. throw up a banner on the edit page itself.  The time when we want to

inform someone that there is going to be maintenance that will impede editing is when the user begins an edit.  (at the moment we inform them when they try to save the edit in the form of an error message.) [..]

This also sounds reasonable. I personally would just go with just a normal central notice, but this would also be fine imo.

4. don't make any change from what we do now.  The external store databases

rarely fail or undergo maintenance.  Increasing the complexity of the system to protect against their outage will be more likely cause harm than the outages themselves.  Instead, just announce it on the blog before and apologize to anybody affected afterwards.

I really feel that having a site/central notice for planned maintenance is important. At the very least stuff like this should probably be announced on various mailing lists or VP's. Our editors are important, we should make sure we avoid unnecessarily wasting any time/effort they put in to the wiki.

-bawolff