On Wed 26-Feb, we are making a large change to how NFS is mounted in Cloud Services https://gerrit.wikimedia.org/r/c/operations/puppet/+/571821. This will impact any Cloud VPS projects that mount NFS for home directories, project directories and scratch, including Toolforge. During this change, NFS will become unresponsive for a short time. Some NFS clients will recover on their own with little impact. Where needed, WMCS will reboot or remount NFS clients.
This change will improve future NFS management and hopefully reduce future disruptions from maintenance.
_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
This is just a reminder of this maintenance and that it is starting in 15 min. Interruptions will be likely for an hour or so.
Brooke Storm SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
On 2/20/20 3:14 PM, Brooke Storm wrote:
On Wed 26-Feb, we are making a large change to how NFS is mounted in Cloud Services https://gerrit.wikimedia.org/r/c/operations/puppet/+/571821. This will impact any Cloud VPS projects that mount NFS for home directories, project directories and scratch, including Toolforge. During this change, NFS will become unresponsive for a short time. Some NFS clients will recover on their own with little impact. Where needed, WMCS will reboot or remount NFS clients.
This change will improve future NFS management and hopefully reduce future disruptions from maintenance.
_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
The maintenance is finished. Cron jobs likely would have failed during the maintenance and other issues. Please report new issues as they show up.
Brooke Storm SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
On 2/26/20 10:47 AM, Brooke Storm wrote:
This is just a reminder of this maintenance and that it is starting in 15 min. Interruptions will be likely for an hour or so. Brooke Storm SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_ On 2/20/20 3:14 PM, Brooke Storm wrote:
On Wed 26-Feb, we are making a large change to how NFS is mounted in Cloud Services https://gerrit.wikimedia.org/r/c/operations/puppet/+/571821. This will impact any Cloud VPS projects that mount NFS for home directories, project directories and scratch, including Toolforge. During this change, NFS will become unresponsive for a short time. Some NFS clients will recover on their own with little impact. Where needed, WMCS will reboot or remount NFS clients.
This change will improve future NFS management and hopefully reduce future disruptions from maintenance.
_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Hi Brooke,
Did NFS just stop working on toollabs about 10 minutes ago? Is that related to this?
Maarten
On 26-02-20 22:17, Brooke Storm wrote:
The maintenance is finished. Cron jobs likely would have failed during the maintenance and other issues. Please report new issues as they show up. Brooke Storm SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_ On 2/26/20 10:47 AM, Brooke Storm wrote:
This is just a reminder of this maintenance and that it is starting in 15 min. Interruptions will be likely for an hour or so. Brooke Storm SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_ On 2/20/20 3:14 PM, Brooke Storm wrote:
On Wed 26-Feb, we are making a large change to how NFS is mounted in Cloud Services https://gerrit.wikimedia.org/r/c/operations/puppet/+/571821. This will impact any Cloud VPS projects that mount NFS for home directories, project directories and scratch, including Toolforge. During this change, NFS will become unresponsive for a short time. Some NFS clients will recover on their own with little impact. Where needed, WMCS will reboot or remount NFS clients.
This change will improve future NFS management and hopefully reduce future disruptions from maintenance.
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
On Thu, Feb 27, 2020 at 10:10 AM Maarten Dammers maarten@mdammers.nl wrote:
Hi Brooke,
Did NFS just stop working on toollabs about 10 minutes ago? Is that related to this?
It did, and it is semi-related. An iptables rule change has failed to work as intended. We are working on restoring service.
Bryan
NFS should be ok again. The iptables rules are fixed. Sorry for the noise and issues.
Brooke Storm SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
On 2/27/20 10:12 AM, Bryan Davis wrote:
On Thu, Feb 27, 2020 at 10:10 AM Maarten Dammers maarten@mdammers.nl wrote:
Hi Brooke,
Did NFS just stop working on toollabs about 10 minutes ago? Is that related to this?
It did, and it is semi-related. An iptables rule change has failed to work as intended. We are working on restoring service.
Bryan