Due to repeated recent outages in the past 30 days and a long history of previous outages due to log files filling up NFS for Toolforge, I’ve deployed a change to restrict the maximum file size that can be created from the Toolforge system of 50 GB.
When a process hits that limit, it will fail to continue writing to the file with the message that the “maximum file size” has been reached. There are files over this size in the environment now that are not going to be affected, but moving them within the same filesystem is likely to require help from someone with root access.
If the limit becomes a problem, it can be revisited. Please let us know on the cloud discussion list or on #wikimedia-cloud if problems arise from the change.
Thanks!
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org mailto:bstorm@wikimedia.org IRC: bstorm_
_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
It might help to mail tool users about their tools that have files with
45GB, say, once a week or month.
Cheers, Magnus
On Wed, Mar 13, 2019 at 7:25 PM Brooke Storm bstorm@wikimedia.org wrote:
Due to repeated recent outages in the past 30 days and a long history of previous outages due to log files filling up NFS for Toolforge, I’ve deployed a change to restrict the maximum file size that can be created from the Toolforge system of 50 GB.
When a process hits that limit, it will fail to continue writing to the file with the message that the “maximum file size” has been reached. There are files over this size in the environment now that are not going to be affected, but moving them within the same filesystem is likely to require help from someone with root access.
If the limit becomes a problem, it can be revisited. Please let us know on the cloud discussion list or on #wikimedia-cloud if problems arise from the change.
Thanks!
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
On Thu, Mar 14, 2019 at 1:57 AM Magnus Manske via Cloud < cloud@lists.wikimedia.org> wrote:
It might help to mail tool users about their tools that have files with
45GB, say, once a week or month.
Cheers, Magnus
Filed as https://phabricator.wikimedia.org/T218285; thank you for the recommendation.