Hi Rogol,
On 16/05/2017 15:44, Faidon Liambotis wrote:
This was a small outage, affecting only a small portion of European traffic (and tiny portion of our global traffic) and not what we'd call a "widespread" one -- very far from it.
+1 I think I have to apologize to the Operations team for the the wording I have chosen for the title of this thread. In hindsight, this issue was minor.
I was taken aback by the combination of the fact that we (at WM-IT) were contacted by a journalist (which, in hindsight, put on us some pressure about trying to understand the causes of the problem); we were in fact experiencing some problems (both Wikimedia Italia's office and myself at home use Fastweb as ISP); and, finally, I have misjudged the information contained in the status page.
This could have been remediated by either a) the intermediate provider fixing or working around the issue b) us disabling the use of that provider temporarily c) the affected ISPs (like Fastweb) disabling the use of that provider temporarily. In this case, we did not wait for (a) or (c) and worked around the issue ourselves, faster than Fastweb etc. did (and in less than an hour even since the initial report). I think the response from our side and our engineers (and especially Arzhel) was pretty stellar.
It was. Thank you, Faidon, and thanks to all the team for the amazing work you do.
Cristian