[Labs-l] Tools webproxy outage (was Re: output truncated - webproxy?)

[Andrew Bogott]

On 5/23/14 12:04 AM, Hedonil wrote:
> Another effect since ~today.
> Output of longer lists get truncated in the middle of output.
>
>
/var filled up on the tools web proxy, which was causing the proxy to 
misbehave in a variety of ways, mainly because nginx was unable to write 
to /var/log/nginx/access.log and flipped out as a result.

I applied the role::labs::lvm::biglogs class to the proxy instance, 
which designates 8g of space for logging, then rebooted the box, and 
then scrambled a bit to recreate all of the /var/log/<whatever> dirs 
that services expected to find but were too stupid to create.  After 
that the proxy service appears to have resumed normal behavior.

8g is a fair bit of space, and the nginx logs are already set to rotate 
out every 2-3 days, so there /should/ be plenty of room now. 
Nevertheless, there's some chance that the log overflow problem was a 
secondary symptom of a different problem that caused extra logging in 
the first place, so I'm going to investigate that a bit more and also 
keep an eye out.

Sorry for the service interruption!

-Andrew