I just ran scap and saw the following for a lot of hosts:
srv285: rsync: write failed on "/usr/local/apache/common-local/php-1.19/cache/l10n/l10n_cache-ab.cdb": No space left on device (28)
srv285: rsync error: error in file IO (code 11) at receiver.c(302) [receiver=3.0.7]
srv285: rsync: connection unexpectedly closed (2051 bytes received so far) [generator]
srv285: rsync error: error in rsync protocol data stream (code 12) at io.c(601) [generator=3.0.7]
Also, on configchange:
mw21: rsync: write failed on "/apache/common-local/wmf-config/CommonSettings.php": No space left on device (28)
mw21: rsync error: error in file IO (code 11) at receiver.c(302) [receiver=3.0.7]
mw21: rsync: connection unexpectedly closed (37 bytes received so far) [generator]
mw21: rsync error: error in rsync protocol data stream (code 12) at io.c(601) [generator=3.0.7]
Not sure if this is a problem and/or if others are aware/working on it, but thought I'd mention it.
It appears that there are a few issues
#1 the cleanup cron is only looking for php* and there are many older files named timeline*
On some machines /tmp is a separate partition, but / is still filled up. This appears to be due to a huge amount of php errors filling the logs
example below : Apr 2 20:32:46 mw21 apache2[17566]: PHP Warning: readdir() expects parameter 1 to be resource, boolean given in /usr/local/apache/common-local/php-1.19/extensions/ConfirmEdit/FancyCaptcha.class.php on line 100
Can someone check out the php errors?
On Mon, Apr 2, 2012 at 4:19 PM, Arthur Richards arichards@wikimedia.org wrote:
I just ran scap and saw the following for a lot of hosts:
srv285: rsync: write failed on "/usr/local/apache/common-local/php-1.19/cache/l10n/l10n_cache-ab.cdb": No space left on device (28)
srv285: rsync error: error in file IO (code 11) at receiver.c(302) [receiver=3.0.7]
srv285: rsync: connection unexpectedly closed (2051 bytes received so far) [generator]
srv285: rsync error: error in rsync protocol data stream (code 12) at io.c(601) [generator=3.0.7]
Also, on configchange:
mw21: rsync: write failed on "/apache/common-local/wmf-config/CommonSettings.php": No space left on device (28)
mw21: rsync error: error in file IO (code 11) at receiver.c(302) [receiver=3.0.7]
mw21: rsync: connection unexpectedly closed (37 bytes received so far) [generator]
mw21: rsync error: error in rsync protocol data stream (code 12) at io.c(601) [generator=3.0.7]
Not sure if this is a problem and/or if others are aware/working on it, but thought I'd mention it.
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Le 03/04/12 01:30, Leslie Carr a écrit :
On some machines /tmp is a separate partition, but / is still filled up. This appears to be due to a huge amount of php errors filling the logs
Either we should stop Apache logging entirely (since logs are already sent on a central host anyway) or at least have /var/log/apache mounted on a dedicated partition.
On 03/04/12 15:58, Antoine Musso wrote:
Le 03/04/12 01:30, Leslie Carr a écrit :
On some machines /tmp is a separate partition, but / is still filled up. This appears to be due to a huge amount of php errors filling the logs
Either we should stop Apache logging entirely (since logs are already sent on a central host anyway) or at least have /var/log/apache mounted on a dedicated partition.
I thought so until I read about the output channel feature in rsyslog, then I decided to try using that instead. I deployed an rsyslog configuration change which splits off the apache log to a separate file from /var/log/syslog and limits it to 500MB including archives. Remote logging will continue to work as normal.
https://gerrit.wikimedia.org/r/#change,4149
-- Tim Starling
Le 03/04/12 08:22, Tim Starling a écrit : <snip>
I thought so until I read about the output channel feature in rsyslog, then I decided to try using that instead. I deployed an rsyslog configuration change which splits off the apache log to a separate file from /var/log/syslog and limits it to 500MB including archives. Remote logging will continue to work as normal.
I did not know that rsyslog could track file size. I thought about using logrotate instead, but it is probably not going to guarantee the log files are under a given threshold.
*gives yet another kudos to Tim*
wikitech-l@lists.wikimedia.org