Hi all,

we had an outage of Wikitech and Labsconsole / Labs.

Apache on virt0 was broken.

It was reported by S Page on IRC, the symptons were wikitech and labsconsole being down via HTTP(S) but pingable.

We also had Icinga monitoring telling us about it.

I went to virt0 and saw Apache running, shortly after restarted it and that fixed the issue.

The root cause i found to be a bug in Phusion Passenger module that is used on virt0, as opposed to cluster Apache.

from /usr/lib/phusion_passenger/passenger-spawn-server:61

*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe) (process 23592):

I made a copy of the full log and for now it is on virt0 in /root/apache2_error_log_20131219.log if you would like to see it all.

So far i just blame https://www.phusionpassenger.com/ respectively libapache2-mod-passenger and didn't take any further action. I did tell the labs IRC channel but did not post on labs-l.

Daniel