We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time).
The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services.
-Andrew + the WMCS Team
Reminder: this is starting shortly.
On 4/27/21 12:48 AM, Andrew Bogott wrote:
We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time).
The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services.
-Andrew + the WMCS Team
I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ https://spi-tools.toolforge.org/ and https://spi-tools-dev.toolforge.org/ https://spi-tools-dev.toolforge.org/. Also:
ssh: connect to host dev.tools.wmflabs.org port 22: Network is unreachable
Could the VPS work be causing this?
On Apr 27, 2021, at 10:03 AM, Andrew Bogott abogott@wikimedia.org wrote:
Reminder: this is starting shortly.
On 4/27/21 12:48 AM, Andrew Bogott wrote:
We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time).
The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services.
-Andrew + the WMCS Team
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Toolforge NFS services are not accessible to the VMs. Something went wrong during the upgrade and the team is working to restore services as quickly as possible.
I have shut off the bastions to protect their local disks.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org
On Apr 27, 2021, at 8:34 AM, Roy Smith roy@panix.com wrote:
I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ https://spi-tools.toolforge.org/ and https://spi-tools-dev.toolforge.org/ https://spi-tools-dev.toolforge.org/. Also:
ssh: connect to host dev.tools.wmflabs.org http://dev.tools.wmflabs.org/ port 22: Network is unreachable
Could the VPS work be causing this?
On Apr 27, 2021, at 10:03 AM, Andrew Bogott <abogott@wikimedia.org mailto:abogott@wikimedia.org> wrote:
Reminder: this is starting shortly.
On 4/27/21 12:48 AM, Andrew Bogott wrote:
We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time).
The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services.
-Andrew + the WMCS Team
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org mailto:Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org mailto:labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Yep, it's related to the upgrade, working on it, thanks for your patience!
Will update shortly.
On 04/27 11:34, Roy Smith wrote:
I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ https://spi-tools.toolforge.org/ and https://spi-tools-dev.toolforge.org/ https://spi-tools-dev.toolforge.org/. Also:
ssh: connect to host dev.tools.wmflabs.org port 22: Network is unreachable
Could the VPS work be causing this?
On Apr 27, 2021, at 10:03 AM, Andrew Bogott abogott@wikimedia.org wrote:
Reminder: this is starting shortly.
On 4/27/21 12:48 AM, Andrew Bogott wrote:
We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time).
The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services.
-Andrew + the WMCS Team
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
There's still a bit of upgrading happening but the recent network interruption should now be resolved. We're about to put Horizon back online so things should be back to normal shortly. As always, some tools may have failed to recover from the network interruption so you're encouraged to check on the state of things.
Sorry for the interruption!
-Andrew
On 4/27/21 10:37 AM, David Caro wrote:
Yep, it's related to the upgrade, working on it, thanks for your patience!
Will update shortly.
On 04/27 11:34, Roy Smith wrote:
I'm getting timeouts and 502's on both https://spi-tools.toolforge.org/ https://spi-tools.toolforge.org/ and https://spi-tools-dev.toolforge.org/ https://spi-tools-dev.toolforge.org/. Also:
ssh: connect to host dev.tools.wmflabs.org port 22: Network is unreachable
Could the VPS work be causing this?
On Apr 27, 2021, at 10:03 AM, Andrew Bogott abogott@wikimedia.org wrote:
Reminder: this is starting shortly.
On 4/27/21 12:48 AM, Andrew Bogott wrote:
We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time).
The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services.
-Andrew + the WMCS Team
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
cloud-announce@lists.wikimedia.org