Toolforge NFS services are not accessible to the VMs. Something went wrong during the upgrade and the team is working to restore services as quickly as possible.
I have shut off the bastions to protect their local disks.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org
On Apr 27, 2021, at 8:34 AM, Roy Smith roy@panix.com wrote:
I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ https://spi-tools.toolforge.org/ and https://spi-tools-dev.toolforge.org/ https://spi-tools-dev.toolforge.org/. Also:
ssh: connect to host dev.tools.wmflabs.org http://dev.tools.wmflabs.org/ port 22: Network is unreachable
Could the VPS work be causing this?
On Apr 27, 2021, at 10:03 AM, Andrew Bogott <abogott@wikimedia.org mailto:abogott@wikimedia.org> wrote:
Reminder: this is starting shortly.
On 4/27/21 12:48 AM, Andrew Bogott wrote:
We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time).
The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services.
-Andrew + the WMCS Team
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org mailto:Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org mailto:labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud