Hi!
Going multithread is really easy for a socket listener.
Really? :)
However, not so much in the LogProcessors. If they are shared accross threads, you may end up with all threads blocked in the fwrite and if they aren't shared, the files may easily corrupt (depends on what you are exactly doing with them).
I don't really understand what you say ;-) Do you mean lost data as 'corrupt'?
Since the problem is that the socket buffer fills, it surprised me that the server didn't increase SO_RCVBUF. That's not a solution but should help (already set in /proc/sys/net/core/rmem_default ?).
It is long term CPU saturation issue - mux process isn't fast enough to handle 16 output streams. Do note, there're quite a few events a second :)
The real issue is: what are you placing on your pipes that are so slow to read from them? Optimizing those scripts could be a simpler solution.
No, those scripts are not the bottleneck, there's plenty of CPU available, and they are not blocking (for too long, everything is blocking for a certain amount of time ;-)
Wouldn't be hard to make the pipe writes non-blocking, properly blaming the slow pipes that couldn't be written
There are no slow pipes. Bottleneck is udp2log step.
Domas