[Labs-l] [Labs-announce] [notice] Tools NFS space conservation

[Damian Zaremba]

I might be a 'little' guilty here....
1.5T    logs
67G    mysql_backups

Adding some crons for cleanup :)

Do you have a good suggestion for rotating the log files written out by 
jsub? Not sure how the grid engine handles things in regards to 
hup/removing files/stale file handlers etc... would rather not have to 
restart the processes as they're very long running bots.

- Damian

On 09/02/2016 19:17, Chase Pettet wrote:
> Greetings,
>
> We have 8T allocated to the Tools project and are now getting space 
> warnings.  A log/temp/cache cleanup was coming regardless for capacity 
> planning purposes but this has moved it up the priority list.  We will 
> try to truncate in use logs, and generally be reasonable about cleanup.
>
> A few thoughts:
>
> * Primary copies of scripts and bots should be elsewhere.  Keep these 
> in some source control mechanism.
> * Please try to be mindful of logs and space usage over time. It is 
> not difficult to find >10G log files at the moment.
> * Be aware that access, error, and log files are excluded from backups
> * This situation is partly holistic and hopefully we can transition to 
> a central logging mechanism that is better for all.
>
> As always, we are around to discuss in #wikimedia-labs on irc
>
> Thanks,
>
> Chase Pettet
> Operations Engineer
>
>
> _______________________________________________
> Labs-announce mailing list
> Labs-announce at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-announce
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20160209/f7b90377/attachment.html>