[Labs-l] All Clear: Labs network work Wednesday 2015-08-19 21:00 UTC
Andrew Bogott
abogott at wikimedia.org
Thu Aug 20 15:04:31 UTC 2015
On 8/19/15 5:04 PM, Andrew Bogott wrote:
> This is done and everything should be back to normal. Let me know if
> you encounter irregularities!
A few followup details:
- During the update window, there was a general network outage of about
15 minutes. This was because, predictably, nova-network didn't behave
as we expected.
- Due to a puppet bug (https://phabricator.wikimedia.org/T109711),
network performance was subpar for 18 hours or so after the switch. That
resulted in a lot of spurious Diamond alerts and some puppet failures.
This should be resolved now.
- The good news is: We're now running a slightly-more-modern (and more
upgradeable) network host. We also learned a lot during the switch so
should be able to arrange for a much shorter outage time during future
upgrades. In addition, we're a few days away from having a fully
functioning hot spare for our network node.
-Andrew
More information about the Labs-l
mailing list