Hi,
At Wikimedia Italia have been contacted in the last few minutes because Wikipedia seems unreachable and very slow from Italy.
The status pages for servers and services signals several perfomance issues: https://status.wikimedia.org/
Does somebody know what's going on?
Thank you.
Cristian
Throughout last night (around 7pm WAT till I slept off), I couldn't access Wikipedia in Nigeria. I thought it was my service provider acting funny at first. I'll like to know what the problem is, as well.
Sam.
On Mon, May 15, 2017 at 10:03 AM, Cristian Consonni cristian@balist.es wrote:
On 15/05/2017 11:02, Cristian Consonni wrote:
Hi,
At Wikimedia Italia have been contacted in the last few minutes because Wikipedia seems unreachable and very slow from Italy.
(Ah, I forgot to mention. by a journalist from ANSA, the national news agency)
Cristian
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/ wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/ wiki/Wikimedia-l New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Hi Sam, if this will help, I think it was your provider. I was able to access Wikipedia at that same time last night. Wikipedia is also up as at this morning. Kayode Yussuf
On Monday, May 15, 2017 10:40 AM, Oloruntoba Oyeyele oloruntobaoyeyele@gmail.com wrote:
Throughout last night (around 7pm WAT till I slept off), I couldn't access Wikipedia in Nigeria. I thought it was my service provider acting funny at first. I'll like to know what the problem is, as well.
Sam.
On Mon, May 15, 2017 at 10:03 AM, Cristian Consonni cristian@balist.es wrote:
On 15/05/2017 11:02, Cristian Consonni wrote:
Hi,
At Wikimedia Italia have been contacted in the last few minutes because Wikipedia seems unreachable and very slow from Italy.
(Ah, I forgot to mention. by a journalist from ANSA, the national news agency)
Cristian
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/ wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/ wiki/Wikimedia-l New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
_______________________________________________ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Hi Cristian,
We've had some network connectivity issues with one of our ISPs in our European datacenter that were probably the source of your problems (FastWeb in Italy were among the affected ISPs). This was a localized issue -- only in the European datacenter and one out of six major network carriers and dozens of network connectivity partners.
The issue has been worked around as of 09:26 UTC and the situation is being monitored.
This issue is being tracked at: https://phabricator.wikimedia.org/T165288
Do let us know if this is not the case for you of if you're experiencing any trouble, here, off-list or directly on that Phabricator task.
Thanks, Faidon -- Faidon Liambotis Principal Operations Engineer Wikimedia Foundation
On Mon, May 15, 2017 at 11:02:39AM +0200, Cristian Consonni wrote:
Hi,
At Wikimedia Italia have been contacted in the last few minutes because Wikipedia seems unreachable and very slow from Italy.
The status pages for servers and services signals several perfomance issues: https://status.wikimedia.org/
Does somebody know what's going on?
Thank you.
Cristian
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Hi Faidon,
On 15/05/2017 13:02, Faidon Liambotis wrote:
Hi Cristian,
We've had some network connectivity issues with one of our ISPs in our European datacenter that were probably the source of your problems (FastWeb in Italy were among the affected ISPs). This was a localized issue -- only in the European datacenter and one out of six major network carriers and dozens of network connectivity partners.
The issue has been worked around as of 09:26 UTC and the situation is being monitored.
This issue is being tracked at: https://phabricator.wikimedia.org/T165288
Do let us know if this is not the case for you of if you're experiencing any trouble, here, off-list or directly on that Phabricator task.
Thanks for the update. Keep up the good work.
Ciao,
Cristian
Faidon
Do you believe that your recovery plan was adequate or will you be reviewing it in the light of this widespread outage?
"Rogol"
On Mon, May 15, 2017 at 12:02 PM, Faidon Liambotis faidon@wikimedia.org wrote:
Hi Cristian,
We've had some network connectivity issues with one of our ISPs in our European datacenter that were probably the source of your problems (FastWeb in Italy were among the affected ISPs). This was a localized issue -- only in the European datacenter and one out of six major network carriers and dozens of network connectivity partners.
The issue has been worked around as of 09:26 UTC and the situation is being monitored.
This issue is being tracked at: https://phabricator.wikimedia.org/T165288
Do let us know if this is not the case for you of if you're experiencing any trouble, here, off-list or directly on that Phabricator task.
Thanks, Faidon -- Faidon Liambotis Principal Operations Engineer Wikimedia Foundation
On Mon, May 15, 2017 at 11:02:39AM +0200, Cristian Consonni wrote:
Hi,
At Wikimedia Italia have been contacted in the last few minutes because Wikipedia seems unreachable and very slow from Italy.
The status pages for servers and services signals several perfomance
issues:
Does somebody know what's going on?
Thank you.
Cristian
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/ wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/ wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/ wiki/Wikimedia-l New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Hi Rogol,
I'm not sure what you're referring to when you're talking about a "recovery plan". These are all pretty standard day-to-day operations for us.
This was a small outage, affecting only a small portion of European traffic (and tiny portion of our global traffic) and not what we'd call a "widespread" one -- very far from it. It wasn't even noticeable in traffic graphs at all. This was the result of a networking issue in an intermediate provider between us and the ISPs affected, so nothing that was in our direct control.
This could have been remediated by either a) the intermediate provider fixing or working around the issue b) us disabling the use of that provider temporarily c) the affected ISPs (like Fastweb) disabling the use of that provider temporarily. In this case, we did not wait for (a) or (c) and worked around the issue ourselves, faster than Fastweb etc. did (and in less than an hour even since the initial report). I think the response from our side and our engineers (and especially Arzhel) was pretty stellar.
Faidon
On Mon, May 15, 2017 at 09:00:06PM +0100, Rogol Domedonfors wrote:
Faidon
Do you believe that your recovery plan was adequate or will you be reviewing it in the light of this widespread outage?
"Rogol"
On Mon, May 15, 2017 at 12:02 PM, Faidon Liambotis faidon@wikimedia.org wrote:
Hi Cristian,
We've had some network connectivity issues with one of our ISPs in our European datacenter that were probably the source of your problems (FastWeb in Italy were among the affected ISPs). This was a localized issue -- only in the European datacenter and one out of six major network carriers and dozens of network connectivity partners.
The issue has been worked around as of 09:26 UTC and the situation is being monitored.
This issue is being tracked at: https://phabricator.wikimedia.org/T165288
Do let us know if this is not the case for you of if you're experiencing any trouble, here, off-list or directly on that Phabricator task.
Thanks, Faidon -- Faidon Liambotis Principal Operations Engineer Wikimedia Foundation
On Mon, May 15, 2017 at 11:02:39AM +0200, Cristian Consonni wrote:
Hi,
At Wikimedia Italia have been contacted in the last few minutes because Wikipedia seems unreachable and very slow from Italy.
The status pages for servers and services signals several perfomance
issues:
Does somebody know what's going on?
Thank you.
Cristian
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/ wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/ wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/ wiki/Wikimedia-l New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Hi Rogol,
On 16/05/2017 15:44, Faidon Liambotis wrote:
This was a small outage, affecting only a small portion of European traffic (and tiny portion of our global traffic) and not what we'd call a "widespread" one -- very far from it.
+1 I think I have to apologize to the Operations team for the the wording I have chosen for the title of this thread. In hindsight, this issue was minor.
I was taken aback by the combination of the fact that we (at WM-IT) were contacted by a journalist (which, in hindsight, put on us some pressure about trying to understand the causes of the problem); we were in fact experiencing some problems (both Wikimedia Italia's office and myself at home use Fastweb as ISP); and, finally, I have misjudged the information contained in the status page.
This could have been remediated by either a) the intermediate provider fixing or working around the issue b) us disabling the use of that provider temporarily c) the affected ISPs (like Fastweb) disabling the use of that provider temporarily. In this case, we did not wait for (a) or (c) and worked around the issue ourselves, faster than Fastweb etc. did (and in less than an hour even since the initial report). I think the response from our side and our engineers (and especially Arzhel) was pretty stellar.
It was. Thank you, Faidon, and thanks to all the team for the amazing work you do.
Cristian
On Mon, May 15, 2017 at 11:02 AM, Cristian Consonni cristian@balist.es wrote:
At Wikimedia Italia have been contacted in the last few minutes because Wikipedia seems unreachable and very slow from Italy.
The status pages for servers and services signals several perfomance issues: https://status.wikimedia.org/
Does somebody know what's going on?
Maybe related to https://phabricator.wikimedia.org/T165252 (hardware problems in one of the caching servers)?
Hi,
Probably https://phabricator.wikimedia.org/T165288.
On Mon, May 15, 2017 at 2:04 PM, Gergő Tisza gtisza@gmail.com wrote:
On Mon, May 15, 2017 at 11:02 AM, Cristian Consonni cristian@balist.es wrote:
At Wikimedia Italia have been contacted in the last few minutes because Wikipedia seems unreachable and very slow from Italy.
The status pages for servers and services signals several perfomance issues: https://status.wikimedia.org/
Does somebody know what's going on?
Maybe related to https://phabricator.wikimedia.org/T165252 (hardware problems in one of the caching servers)? _______________________________________________ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
wikimedia-l@lists.wikimedia.org