Tomorrow (June 11th) at 1600 UTC, we will be failing over the primary NFS server to do maintenance and upgrades on it. The secondary partner in the cluster is already upgraded and ready, and recent changes *should* make it a fairly straightforward failover with a brief period of high load. If it doesn't proceed smoothly, it will be a slightly longer period of high load and NFS lockup as failover completes (10-20 min or so). After maintenance it will be failed back, which will also, hopefully, be quick and painless.
This is starting in around 10 minutes.
On 6/10/20 2:03 PM, Brooke Storm wrote:
Tomorrow (June 11th) at 1600 UTC, we will be failing over the primary NFS server to do maintenance and upgrades on it. The secondary partner in the cluster is already upgraded and ready, and recent changes *should* make it a fairly straightforward failover with a brief period of high load. If it doesn't proceed smoothly, it will be a slightly longer period of high load and NFS lockup as failover completes (10-20 min or so). After maintenance it will be failed back, which will also, hopefully, be quick and painless. -- Brooke Storm SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
This is done! Sorry it was a bit more "freeze/hang" prone than hoped, but it was smoother than in the past and improvements will be made based on our findings.
On 6/11/20 8:52 AM, Brooke Storm wrote:
This is starting in around 10 minutes.
On 6/10/20 2:03 PM, Brooke Storm wrote:
Tomorrow (June 11th) at 1600 UTC, we will be failing over the primary NFS server to do maintenance and upgrades on it. The secondary partner in the cluster is already upgraded and ready, and recent changes *should* make it a fairly straightforward failover with a brief period of high load. If it doesn't proceed smoothly, it will be a slightly longer period of high load and NFS lockup as failover completes (10-20 min or so). After maintenance it will be failed back, which will also, hopefully, be quick and painless. -- Brooke Storm SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
cloud-announce@lists.wikimedia.org