I totally agree with Thomas. On Tue, May 24, 2011 at 4:32 PM, Thomas Morton <morton.thomas(a)googlemail.com > So, just a quick thought for future reference - during maintenance is it > possible in future to update the error message to explain that maintenance > is ongoing? > > Seeing as how widely WMF projects are used by a non-technical project the > current MySQL connection error I am seeing on Commons is just going to > cause > confusion :) And the standard error page WIkipedia was showing a minute ago > is not particularly helpful/explanatory in this specific situation. > > Indeed, given the almost certainty of downtime from this maintenance, would > it not just be best to bite the bullet (in such cases) and take the > affected > sites off-line with a useful maintenance message? It's essentially the same > end point. > > What's the best way of addressing this suggestion to the right people? > (i.e. > the ops team?) > > Tom > > On 23 May 2011 21:19, Guillaume Paumier <gpaumier(a)wikimedia.org> > > Dear all, > > > > The Wikimedia Foundation will be performing network maintenance on > > Tuesday, May 24 between 13:00 and 14:00 (UTC) (see other timezones on > > timeanddate.com: http://ur1.ca/49cl2 ). > > > > During the maintenance period, you may experience intermittent > > connection issues to Wikimedia Foundation websites, including > > wikipedia.org. > > > > We have been experiencing router networking issues (and as a direct > > result, latency issues) since last week. After much investigation, and > > temporary fixes, the Operations team decided to update the router > > software and tune the configuration. > > > > We apologize for the inconvenience. > > > > -- > > Guillaume Paumier > > > > _______________________________________________ > > foundation-l mailing list > > foundation-l(a)lists.wikimedia.org > > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l > > > _______________________________________________ > foundation-l mailing list > foundation-l(a)lists.wikimedia.org > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l >

On 24/05/11 23:32, Thomas Morton wrote: Database connection errors were not an anticipated consequence of the scheduled router upgrades. The people who might have been able to change the error message were busy diagnosing and fixing the problem. When we have a lengthy period of downtime, more sysadmins arrive online, and a wider perspective on the problem develops, including attention to community impact and communication. But since the downtime in this case was only half an hour, there was not enough time for this to happen. -- Tim Starling

On 25/05/11 17:32, FT2 wrote: You mean replace the entire site with an error page? But only part of the site was down. More and more things became accessible as each database server was fixed. I'm not sure how this could work. Even if we did prepare an error message saying "Wikipedia will be down for 2 minutes while a router restarts", I don't think that could be called "explanatory" if it were displayed for half an hour. Writing informative error messages and displaying them in appropriate places is necessarily a low-priority task during downtime, the higher priority task being to get the site working again. Maybe at some time in the future, we will have enough 24/7 sysadmin manpower that we can respond to any unplanned downtime in the way you suggest. But we don't have that capability just yet. -- Tim Starling

On 25/05/11 18:14, Thomas Morton wrote: Maybe we can replace the IRC link in the Squid error message with a link to the WatchMouse page (status.wikimedia.org). That would reduce the IRC flood. -- Tim Starling

On 25/05/11 21:19, MZMcBride wrote: Maybe it's time to completely rewrite it. I noticed Austin joking about the awkward "probably temporary" phrasing, and the idea that we need to buy new hardware to avoid downtime is a bit dated. The source is in Subversion, at /trunk/debs/squid/debian/errors. Maybe if someone proposed some new text on meta.wikimedia.org, I could see that it gets included in the next Squid update. -- Tim Starling

On 05/25/2011 01:12 PM, Tim Starling wrote: Site notice for a week before the maintenance would be useful, too. We communicate with our users via web site, not via emails.

I'm pretty sure there was a site notice; I recall seeing one anyway :) Tom On 26 May 2011 09:09, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote:

On 05/26/2011 10:18 AM, Theo10011 wrote: So, then it should just last a bit longer (maybe three days if not a week?) and we would avoid the most of complains.

On 05/26/2011 10:09 AM, Federico Leva (Nemo) wrote: A small site notice? Not shown after dismissal? :) I mean, there are always ways to make site notices less intrusive. It is now common to get notice ~2 weeks before maintenance. And the most of our users are not getting foundation-l or announcement-l emails.

We already get spammed enough with notices, which is one of the reasons many people hide them permanently via css so they never intrude again, which would make them pointless for the more established users, also overkill for what was meant to be (from my understanding) only a few minutes of downtime. Just because we can send something via notices doesn't mean we should, it can and has been devaluing their importance to people. As for the earlier comments about changing the irc channel, how about we point it to the #wikimedia-status (or whatever its called) channel that is designed to hand out info in downtimes compared to the general #wikipedia which i believe it currently points to. -Peachey

In future can I have vanilla and strawberry with that? :) FT2 On Wed, May 25, 2011 at 9:16 AM, Domas Mituzas <midom.lists(a)gmail.com>wrote;wrote:

Austin, That's interesting, what was the wording for the maintenance message? I only ever saw the default "our servers are experiencing a technical problem" error page. Tom On 25 May 2011 10:53, Austin Hair <adhair(a)gmail.com> wrote:

assigned to updating the message in real time. No, definitely not what was being suggested. This is the error message that appeared for me (and apparently others): http://nomulous.com/blog/wp-content/uploads/2009/09/wikipedia_error.png As you can see it refers to some unknown error. In this case the maintentance was known and* pre-planned* for several days. A lot of people were confused by the outage and the error page was unhelpful to them. This could have been mitigated simply by editing that page temporarily to say "Our servers are undergoing scheduled maintenance, which has resulted in some downtime. This should be concluded by 14:00 UTC, please be patient whilst the maintenance progesses." And this is the extent of my suggestion to improve our communication with readers. Tom On 25 May 2011 11:02, Austin Hair <adhair(a)gmail.com> wrote:

Huh? The downtime was expected during 13:00 and 14:00 UTC, or at least there was an email warning of such things the day before... hardly unplanned or unknown. Tom On 25 May 2011 11:12, Domas Mituzas <midom.lists(a)gmail.com> wrote:

The maintenance was planned, downtime was noted as possible. An error message that reflects that seems, frankly, a good idea. The response to what I thought to be a helpful suggestion in improving communication with readership has been... incredibly disappointing. I wish I hadn't bothered. :( I was just passing on comments from people who came to IRC and basically said "oh, well why didn't the site just say that then". Of course; if we are ignoring our readers' concerns now, then fine. Tom On 25 May 2011 11:20, Domas Mituzas <midom.lists(a)gmail.com> wrote:

probably fix the problem beforehand. :-) That's... completely missing the point. Yes the specific errors faced were unexpected or unforseen, BUT they were a* direct result* of the maintenance between 13:00 and 14:00. I am simply passing on the feeling of our readership; which was that the situation was badly communicated to them. I am trying to share my experience here as a sysadmin and website operator; users hate downtime/maintenance, and will complain about it endlessly. Improving our communication of planned maintenance is definitely a good idea. unlike your negativism. I'm trying to be positive, but it seems to simply be dismissed (incorrectly) as "well we didn't know what was going to happen". Tom On 25 May 2011 11:33, Domas Mituzas <midom.lists(a)gmail.com> wrote: > Hi! > > > The maintenance was planned, downtime was noted as possible. An error > > message that reflects that seems, frankly, a good idea. > > There're lots of great ideas around the world, feeding the hungry and > curing the cancer among them. > > > The response to what I thought to be a helpful suggestion in improving > > communication with readership has been... incredibly disappointing. > > Well, you were complaining about confusion at first, probably we indeed > should not show any technical details about anything. > "Site is down, bye!" might be better choice, I guess. > > > I wish I hadn't bothered. :( I was just passing on comments from people > who came to > > IRC and basically said "oh, well why didn't the site just say that then". > > probably fix the problem beforehand. :-) > > > Of course; if we are ignoring our readers' concerns now, then fine. > > unlike your negativism. > > Cheers, > Domas > _______________________________________________ > foundation-l mailing list > foundation-l(a)lists.wikimedia.org > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l >

Maybe we can replace the IRC link in the Squid error message with a link to the WatchMouse page @Tim; that seems a good idea. @Domas, I'm afraid you don't seem to have understood the premise of my suggestion.. which is fine. But one fallacy is worth responding to: endlessly! The first rule of a website people use regularly is: "users will complain endlessly" One of my business mentors has a good maxim about this: "Just because you can't see them complaining, don't simply assume they are not. Because they are." Twitter, Facebook, IRC and all sorts of other websites had people complaining about the down time. That is just a fact of life :) To wit: If that static error page cannot easily be changed prior to a maintenance, then fine :) no worries Tom On 25 May 2011 12:10, Domas Mituzas <midom.lists(a)gmail.com> wrote: > Hi! > > > That's... completely missing the point. Yes the specific errors faced > were > > unexpected or unforseen, BUT they were a* direct result* of the > maintenance > > between 13:00 and 14:00. I am simply passing on the feeling of our > > readership; which was that the situation was badly communicated to them. > > As majority of our users are anons, who visit us once a day or two, we > should probably have started a communication campaign at least two months > before the maintenance. > We practice a lot during fundraisers :-) > > OTOH, if there's no downtime, maybe we're causing quite some frustration > with superfluous communication? :-) > > > I am trying to share my experience here as a sysadmin and website > operator; > > Oh, finally we got some sysadmins and website operators here. > As a sysadmin you sure understand that in larger distributed systems which > are not all built on a set of SPOFs there can be various failure modes, > happening at various layers and various fuzziness. > As a website operator you sure know that it is lots of effort to prepare > boilerplates for every possible situation :-) > > > users hate downtime/maintenance, and will complain about it endlessly. > > endlessly! > > > Improving our communication of planned maintenance is definitely a good > idea. > > So is curing cancer. > > Marcus Buck wrote: > > Domas, what are you trying to achieve with your comments on Tom's > > suggestions? > > > Put some clue in? > > > The sensible reaction (from a person who is involved in the maintenance) > would be: > > I know nobody likes this, but sensible reaction is to work on good > operation rather than standing in front of a mirror and trying five hundred > different "I'm sorry" phrases. > You look too much from that single position, that "communication is good", > without weighting costs or other options. > > Cheers, > Domas > _______________________________________________ > foundation-l mailing list > foundation-l(a)lists.wikimedia.org > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l >

Theo10011 wrote: Is it so much to ask that you read the mailing list thread before replying? Nobody's asking you to memorize every word, but having some general idea of what has been discussed would make your replies less redundant and/or seemingly obtuse. MZMcBride

On Wed, May 25, 2011 at 10:09 PM, Theo10011 <de10011(a)gmail.com> wrote: Where did Mz ever suggest to read Bz????... I only see mention to reading what was already suggested in this email thread. -Peachey

On 25/05/11 22:27, Strainu wrote: There are dozens of places where error messages are generated. It's not trivial to replace them all. Some of them are hard-coded in compiled binaries, some are on the client side. The error message in question comes from DBConnectionError in Database.php in the MediaWiki source. It's hard-coded and the source would have had to have been patched. Since no database problems were anticipated, even if we had tried to implement your plan, we wouldn't have thought to patch Database.php, and the result would have been the same. There was no way to tell when the site was going to be back up, except perhaps after the problem was isolated and the fix was halfway through being implemented. But by that time there was only a few minutes of downtime left. The maintenance window was 13:00 to 14:00, but after things went wrong, there was no guarantee that all problems would be fixed by 14:00. Indeed, if it wasn't for Domas's help as a volunteer sysadmin, the problem may have lasted much longer. Then there would have been plenty of time for messaging and maybe we wouldn't be having this conversation. -- Tim Starling

As a non-tech, don't all reads (at least) pass through the squids, so we can identify and report in a nice way a lot of connection errors at that point? </ignoreifnaive> FT2 On Wed, May 25, 2011 at 2:18 PM, Tim Starling &lt;tstarling(a)wikimedia.org&gt;wrote;wrote:

On 25/05/11 23:41, FT2 wrote: Maybe it would be possible to identify error messages by their HTTP response code, and replace the body with some other text, presumably with the original text embedded somehow for debugging purposes. But I don't think Squid has such a feature, and we have very little development time to spend on this sort of thing. -- Tim Starling

On 26/05/11 00:05, Thomas Morton wrote: No, it's private. I can give it to you if you want it, just give me your assurance offlist that it'll be worth my time to clean any private data out of it and tar it up. -- Tim Starling

Domas, what are you trying to achieve with your comments on Tom's suggestions? He just said that if we know that maintenance is done and could cause outages we should put up an error message that informs the reader about the maintenance work and tells him not to worry. That's obviously a good thing. The sensible reaction (from a person who is involved in the maintenance) would be: "Oh, sorry, we were so much occupied with making the maintenance work as smooth and uninterruptive as possible that we totally didn't think about that. We will integrate it into our flow charts so we won't forget it the next time we need to do maintenance that could cause outages." Everything else is not very goal-oriented. Marcus Buck User:Slomox

Zitat von MZMcBride &lt;z(a)mzmcbride.com&gt;om>: I'm not a native speaker of English and I don't know whether it was the right word. I meant a documentation that tells you which steps need to be taken before, while and after you touch a critical system. Marcus Buck User:Slomox

It might be more worthwhile to put downtime status updates on status.wikimedia.org as a logical page to display the status of the servers, and link to it from the default error messages. Given that status.wm.org is an external service, it would hopefully not be affected by any outages and the Watchmouse service probably should have the functionality to host informational messages like this and explanations for outages (like appstatus.google.com does) even if only after the fact when the ops team has time to write down what is happening. Best regards, Bence

WHOAH THERE IS QUITE SOME SELF ENTITLEMENT THERE. Microsoft revenue: $62B (though you should look at their internet division losses) Google revenue: $29B Apple revenue: $62B Wikimedia revenue: ??? Tech staffing and such is somewhat proportional :) Oh, by the way, I don't know where you look, but I somewhat missed communication about maintenance events ongoing in Google or Microsoft or Apple - you think they have none? Did you get lots of clarification why your gmail was unreachable? Did you get explanation/information why search index was outdated? Do they use site-wide sitenotices for that or what? *shrug*, would be interesting if anyone would actually explain policies of other website incident handling. Domas

Hi! I'm contrarian in this case :) Well, as with every downtime, there are way more issues* that end up uncovered and have to be looked at, and yet largest email threads are about nicer error messages :-) This will be my constructive contribution to the thread: FAIL WHALE! W W W W W W W '. W .-""-._ \ \.--| / "-..__) .-' | _ / \'-.__, .__.,' `'----'._\--' VVVVVVVVVVVVVVVVVVVVV Domas * buggy forcedeth behavior, european DNS server was hanging before maintenance started, loadbalancer likes to throw errors on first slave failure and doesn't go to others, no auto-fallback to read-only mode, too long connect timeouts, etc

The measure is "impact to users", not "revenue losses"... Access to Wikimedia is ubiquitous - it pops up in other tools, it's linked from other sites, it's used to track events of interest to the reader, it's used for work, leisure, projects, and social activities. Imagine people browsing google.com, apple.com, microsoft.com, and suddenly for an extended period getting "server not found" with nothing more of explanation, no details, no idea when they can use the site again (minutes? days?) or if it will repeat. The impact is similar for Wikimedia's readers. FT2 On Wed, May 25, 2011 at 2:50 PM, Domas Mituzas &lt;midom.lists(a)gmail.com&gt;wrote;wrote:

Tim, When I originally wrote: during maintenance is it possible in future to update the error message to explain that maintenance is ongoing? That was a bit of a silly moment from me :) I see how that implies in-maintenance updates. In fact my suggestion was to update the error message to mention the planned maintenance and the timeframes. Sorry for the confusion! Tom On 25 May 2011 08:15, Tim Starling &lt;tstarling(a)wikimedia.org&gt; wrote:

Hi, The maintenance was scheduled on Monday, for the day after that. We had only a few hours to plan for it and communicate about it, and I think we did a pretty good job given the time we had. The maintenance banner was up for a few hours (not a day) prior to the maintenance window to give readers & editors a heads-up. The notice was also posted to social media channels (identica, twitter, facebook) as well as on the most relevant lists. I think that amount of communication is reasonable for a planned maintenance operation that shouldn't result in long downtime. As it was already mentioned in this thread, database errors weren't expected during this network maintenance. It's always possible that unplanned issues arise, and this is why the error page shouldn't be too specific: if we plan for an issue and we end up encountering another one, the error page may display incorrect information about the cause or, more importantly, the severity of the issue. About more ways to communicate on outages: I have a few items on my todo list about this as well, so I'm glad that they were brought up in this thread. The status.wikimedia.org page could certainly be designed in a way that emphasizes the main information; I'm also investigating whether we can use an API to display information on other places, e.g. the Wikimedia blog (assuming the blog isn't down too). I also agree the "WMF error page" could be improved. As a matter of fact, I started thinking about how to improve it a few weeks ago. If you're interested in this, I would welcome your help. Thanks, -- Guillaume Paumier