tl;dr: a random assortment of security group/firewall rules were briefly broken which caused networking issues for some cloud-vps services. This is now resolved.
This outage began at 12:51 PMUTC, was mostly resolved at 13:25 UTC and full resolved at 13:35UTC.
Full (still short) story:
As part of ongoing efforts to make all of cloud-vps work with IPv6, we just now ran an automated script to expand existing securitygroup rules to ipv6 access. A bug in that script effectively destroyed existing rules rather than updating them, which caused many open doors to unceremoniously slam shut.
In order to recover from this as quickly as possible, we restored a backup of the network database from earlier today (04:28 UTC). This restored service but also wiped out any networking changes that might have been made during that time. This means that if you happen to have changed your security groups since then, you will need to repeat your changes.
The still-in-progress incident report can be found here: https://wikitech.wikimedia.org/wiki/Incidents/2025-05-07_cloud-vps_security_...
- Andrew +WMCS staff, Slightly embarassed
_______________________________________________ Cloud-announce mailing list -- cloud-announce@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.o...