Our maintenance is done, and it caused more impact than we’d hoped during the changes.
Sorry for the inconvenience.
We are observing the effects of the changes in case things need to be rolled back for the
next few hours.
Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm(a)wikimedia.org
IRC: bstorm_
On Aug 29, 2018, at 11:06 AM, Brooke Storm
<bstorm(a)wikimedia.org> wrote:
We are starting this maintenance. NFS services (and thus Grid and Kubernetes) could see
short disruptions. As long as all goes well, this should be minimal and only one server
failover should be necessary, however, NFS is central to the setup and could cause
difficulties if problems ensue.
Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org>
IRC: bstorm_
On Aug 28, 2018, at 1:51 PM, Brooke Storm
<bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org>> wrote:
Just a reminder that we will be performing the maintenance mentioned below tomorrow at
1500 UTC.
Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org>
IRC: bstorm_
On Aug 21, 2018, at 1:07 PM, Brooke Storm
<bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org>> wrote:
We will be performing some updates (including reboots) on the tools NFS servers for
Toolforge and CloudVPS instances that use project NFS (with the exception of dumps) at
1500 UTC on Wed Aug 29. The maintenance window will be two hours and more than one NFS
server failover is expected during that time. This could cause some temporary impact to
performance and load on the various connected servers during failovers.
Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org>
IRC: bstorm_