tl;dr: a
random assortment of security group/firewall rules were briefly
broken which caused networking issues for some cloud-vps services. This is now resolved.
This outage
began at 12:51 PM UTC, was mostly
resolved at 13:25 UTC and full resolved at 13:35 UTC.
Full (still short) story:
As part of ongoing efforts to make all of cloud-vps
work with IPv6, we just now ran an automated script to expand
existing security group rules to ipv6 access. A bug in that script
effectively destroyed existing rules rather than updating them,
which caused many open doors to unceremoniously slam shut.
In order to recover from this as quickly as possible,
we restored a backup of the network database from earlier today
(04:28 UTC). This restored service but also wiped out any networking changes that might have been made during that time.
This means that if you happen to have changed your security
groups since then, you will need to repeat your changes.
- Andrew + WMCS staff, Slightly
embarassed