Most of the upgrade is complete; however the upgrade has caused an unexpected partial network interruption. Work on this issue is ongoing, I'll send another email when the issue is resolved.
-Andrew
We'll be upgrading the cloud services OpenStack install on Monday, beginning at 06:00 UTC
The entire upgrade process may take an hour or two. Early on in the process, Horizon (and associated OpenStack APIs) will be disabled (probably for 20 to 30 minutes.) There may also be brief network interruptions during the upgrade.
Toolforge and existing VMs should be largely unaffected apart from possible network hiccups.
This is the second upgrade since October; we are trying to speed up our upgrade schedule in order to catch up with the upstream releases. With luck there will be further such upgrades in the coming months.
- Andrew + the WMCS team
_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
We are still chasing down stray issues (in particular, some of the dump and scratch mounts on toolforge or now wrong) but for almost all use cases things should be back to normal.
-Andrew
On 12/12/19 3:58 PM, Andrew Bogott wrote:
Most of the upgrade is complete; however the upgrade has caused an unexpected partial network interruption. Work on this issue is ongoing, I'll send another email when the issue is resolved.
-Andrew
We'll be upgrading the cloud services OpenStack install on Monday, beginning at 06:00 UTC
The entire upgrade process may take an hour or two. Early on in the process, Horizon (and associated OpenStack APIs) will be disabled (probably for 20 to 30 minutes.) There may also be brief network interruptions during the upgrade.
Toolforge and existing VMs should be largely unaffected apart from possible network hiccups.
This is the second upgrade since October; we are trying to speed up our upgrade schedule in order to catch up with the upstream releases. With luck there will be further such upgrades in the coming months.
- Andrew + the WMCS team
_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
On 12/12/19 11:50 AM, Andrew Bogott wrote:
We are still chasing down stray issues (in particular, some of the dump and scratch mounts on toolforge or now wrong) but for almost all use cases things should be back to normal.
-Andrew
We consider the operations finished. Everything has been done.
NFS, being one of the weakest components of our infra, suffered during today's operation. We were force to reboot most of Toolforge servers, so grid jobs and webservices in both the web grid and kubernetes have most likely been restarted and may present error log entries corresponding to the window of this operation.
Other CloudVPS projects users of NFS (dumps shares, maps, etc) might also require some checking. Please get in touch if you are a project admin of such project.
regards.