Our maintenance is done, and it caused more impact than we’d hoped during the changes.  Sorry for the inconvenience.
We are observing the effects of the changes in case things need to be rolled back for the next few hours.

Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm@wikimedia.org
IRC: bstorm_




On Aug 29, 2018, at 11:06 AM, Brooke Storm <bstorm@wikimedia.org> wrote:

We are starting this maintenance.  NFS services (and thus Grid and Kubernetes) could see short disruptions.  As long as all goes well, this should be minimal and only one server failover should be necessary, however, NFS is central to the setup and could cause difficulties if problems ensue.

Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm@wikimedia.org
IRC: bstorm_




On Aug 28, 2018, at 1:51 PM, Brooke Storm <bstorm@wikimedia.org> wrote:

Just a reminder that we will be performing the maintenance mentioned below tomorrow at 1500 UTC.

Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm@wikimedia.org
IRC: bstorm_



On Aug 21, 2018, at 1:07 PM, Brooke Storm <bstorm@wikimedia.org> wrote:

We will be performing some updates (including reboots) on the tools NFS servers for Toolforge and CloudVPS instances that use project NFS (with the exception of dumps) at 1500 UTC on Wed Aug 29.  The maintenance window will be two hours and more than one NFS server failover is expected during that time.  This could cause some temporary impact to performance and load on the various connected servers during failovers.

Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm@wikimedia.org
IRC: bstorm_