Toolforge NFS services are not accessible to the VMs. Something went wrong during the upgrade and the team is working to restore services as quickly as possible.

I have shut off the bastions to protect their local disks.

Brooke Storm
Staff SRE
Wikimedia Cloud Services
bstorm@wikimedia.org



On Apr 27, 2021, at 8:34 AM, Roy Smith <roy@panix.com> wrote:

I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ and https://spi-tools-dev.toolforge.org/.  Also:

ssh: connect to host dev.tools.wmflabs.org port 22: Network is unreachable

Could the VPS work be causing this?


On Apr 27, 2021, at 10:03 AM, Andrew Bogott <abogott@wikimedia.org> wrote:

Reminder: this is starting shortly.

On 4/27/21 12:48 AM, Andrew Bogott wrote:
We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time).

The total upgrade should take 60-90 minutes.  During the upgrade period Horizon will be disabled.  There may also be brief network interruptions as we restart router services.

-Andrew + the WMCS Team




_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud