Hi!
There will be a general CloudVPS network maintenance on 20202-10-29, from 16:00 UTC to 17:00 UTC.
During the operation window, all cloud services might be intermittently down, inaccessible.
This operation affects all CloudVPS projects, including Toolforge, PAWS and Quarry. Services running in the cloud might fail to contact external entities, and connections to ToolsDB, NFS, wiki-replicas or LDAP might be affected as well.
In the best case scenario, the changes (and downtime) will be barely noticed. The maintenance consist on introducing new hardware equipment in to the CloudVPS edge network. You can find additional details in Phabricator [0].
regards.
[0] https://phabricator.wikimedia.org/T265288
On 2020-10-22 17:41, Arturo Borrero Gonzalez wrote:
Hi!
There will be a general CloudVPS network maintenance on 20202-10-29, from 16:00 UTC to 17:00 UTC.
During the operation window, all cloud services might be intermittently down, inaccessible.
This operation affects all CloudVPS projects, including Toolforge, PAWS and Quarry. Services running in the cloud might fail to contact external entities, and connections to ToolsDB, NFS, wiki-replicas or LDAP might be affected as well.
In the best case scenario, the changes (and downtime) will be barely noticed. The maintenance consist on introducing new hardware equipment in to the CloudVPS edge network. You can find additional details in Phabricator [0].
regards.
Reminder, this is happening now.
regards.
On 2020-10-29 16:59, Arturo Borrero Gonzalez wrote:
On 2020-10-22 17:41, Arturo Borrero Gonzalez wrote:
Hi!
There will be a general CloudVPS network maintenance on 20202-10-29, from 16:00 UTC to 17:00 UTC.
During the operation window, all cloud services might be intermittently down, inaccessible.
This operation affects all CloudVPS projects, including Toolforge, PAWS and Quarry. Services running in the cloud might fail to contact external entities, and connections to ToolsDB, NFS, wiki-replicas or LDAP might be affected as well.
In the best case scenario, the changes (and downtime) will be barely noticed. The maintenance consist on introducing new hardware equipment in to the CloudVPS edge network. You can find additional details in Phabricator [0].
regards.
Reminder, this is happening now.
The operation is now completed. There was a brief interruption of the service, but should be recovered now.
Let us know if you see anything weird matching the timing or somehow related to this operation window.
regards.
Hi Arturo,
On 29-10-2020 18:30, Arturo Borrero Gonzalez wrote:
Let us know if you see anything weird matching the timing or somehow related to this operation window.
This was announced as network maintenance, but the tools-sgebastion-08 rebooted at Thu Oct 29 17:15. Is this related or did the server happen to crash in the same window?
Maarten
Given the announcement that it was completed at 1730 UTC and the fact that I had connection issues on a cloud VPS instance at 1709 UTC (per logs), I think it's safe to say this was related to the maintenance.
AmandaNP English Wikipedia CheckUser & Oversighter UTRS & ACC Developer
On Thu, 29 Oct 2020 at 19:04, Maarten Dammers maarten@mdammers.nl wrote:
Hi Arturo,
On 29-10-2020 18:30, Arturo Borrero Gonzalez wrote:
Let us know if you see anything weird matching the timing or somehow
related to
this operation window.
This was announced as network maintenance, but the tools-sgebastion-08 rebooted at Thu Oct 29 17:15. Is this related or did the server happen to crash in the same window?
Maarten
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
On 2020-10-30 00:04, Maarten Dammers wrote:
Hi Arturo,
On 29-10-2020 18:30, Arturo Borrero Gonzalez wrote:
Let us know if you see anything weird matching the timing or somehow related to this operation window.
This was announced as network maintenance, but the tools-sgebastion-08 rebooted at Thu Oct 29 17:15. Is this related or did the server happen to crash in the same window?
Hi there,
due to the network maintenance, we had NFS issues on some VMs, particularly affecting our Grid service.
So indeed we rebooted a couple of servers (including tools-sgebastion-08) to get to a stable situation and make sure everything was working as expected.
thanks for double checking.
regards.