Hi all,
we had an outage of Wikitech and Labsconsole / Labs.
Apache on virt0 was broken.
It was reported by S Page on IRC, the symptons were wikitech and labsconsole being down via HTTP(S) but pingable.
We also had Icinga monitoring telling us about it.
I went to virt0 and saw Apache running, shortly after restarted it and that fixed the issue.
The root cause i found to be a bug in Phusion Passenger module that is used on virt0, as opposed to cluster Apache.
from /usr/lib/phusion_passenger/passenger-spawn-server:61
*** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe) (process 23592):
I made a copy of the full log and for now it is on virt0 in /root/apache2_error_log_20131219.log if you would like to see it all.
So far i just blame https://www.phusionpassenger.com/ respectively libapache2-mod-passenger and didn't take any further action. I did tell the labs IRC channel but did not post on labs-l.
Daniel