Hi all,
We had a fibre cut of our connection to our Tampa DC this morning. ETA of a fix is still pending, but the cuts have been located and crews are being dispatched. Meanwhile public traffic is being rerouted via the public Internet, so most services should be reachable. Tampa is our secondary DC which we're decommissioning, so there was no impact on our main sites. Impacted temporarily were: inbound Wikimedia.org email, Bugzilla, Labs. Fundraising is still impacted, but shouldn't be (we're debugging).
Thanks to the ops team for their quick response.
Erik
Erik Moeller erik@wikimedia.org wrote:
We had a fibre cut of our connection to our Tampa DC this morning. ETA of a fix is still pending, but the cuts have been located and crews are being dispatched. Meanwhile public traffic is being rerouted via the public Internet, so most services should be reachable. Tampa is our secondary DC which we're decommissioning, so there was no impact on our main sites. Impacted temporarily were: inbound Wikimedia.org email, Bugzilla, Labs. Fundraising is still impacted, but shouldn't be (we're debugging).
Small correction: Labs is still impacted as neither the replica servers nor Wikipedia & Co. can be reached without the link so tools and bots relying on that are out of ser- vice at the moment.
Tim
I wrote:
We had a fibre cut of our connection to our Tampa DC this morning. ETA of a fix is still pending, but the cuts have been located and crews are being dispatched. Meanwhile public traffic is being rerouted via the public Internet, so most services should be reachable. Tampa is our secondary DC which we're decommissioning, so there was no impact on our main sites. Impacted temporarily were: inbound Wikimedia.org email, Bugzilla, Labs. Fundraising is still impacted, but shouldn't be (we're debugging).
Small correction: Labs is still impacted as neither the replica servers nor Wikipedia & Co. can be reached without the link so tools and bots relying on that are out of ser- vice at the moment.
... and when I hit C-c C-c, the link came back up. So now Labs should be fully working again.
Tim
Did that on purpose ;)
But the links are still recovering, we're on a single link and we're not sure of their stability, so be wary!
On Fri, Jan 17, 2014 at 2:26 PM, Tim Landscheidt tim@tim-landscheidt.de wrote:
I wrote:
We had a fibre cut of our connection to our Tampa DC this morning. ETA of a fix is still pending, but the cuts have been located and crews are being dispatched. Meanwhile public traffic is being rerouted via the public Internet, so most services should be reachable. Tampa is our secondary DC which we're decommissioning, so there was no impact on our main sites. Impacted temporarily were: inbound Wikimedia.org email, Bugzilla, Labs. Fundraising is still impacted, but shouldn't be (we're debugging).
Small correction: Labs is still impacted as neither the replica servers nor Wikipedia & Co. can be reached without the link so tools and bots relying on that are out of ser- vice at the moment.
... and when I hit C-c C-c, the link came back up. So now Labs should be fully working again.
Tim
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org