On Mon, Oct 7, 2019 at 1:44 PM Maarten Dammers <maarten(a)mdammers.nl> wrote:
Andrew,
I got 30+ emails and counting for stale file handles, permission errors,
email bounces, etc. Not sure what definition of brief you're using for
the outage (in my line of work that's sometimes less than a second), but
all these emails seem to have been send in a two hour window based on
the timestamps. Start 17:42 end 19:55 (both Amsterdam time). Maybe you
can stop cron next time you're going to planned break toollabs? Or was
this caused by the dns + network outage?
The errors in Toolforge were unintended side effects of the OpenStack
upgrade. The network issues cascaded to cause a variety of issues
across Cloud VPS projects including Toolforge. We will be working on
an incident report and it will be shared with this list when it has
been prepared.
Those of us on the "root@" emails got around 400+ emails triggered by
various parts of the service interruption, so we have empathy for the
inbox problem this caused for others. :/
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808