Hi!
Going multithread is really easy for a socket
listener.
Really? :)
However, not so
much in the LogProcessors. If they are shared accross threads, you may
end up with all threads blocked in the fwrite and if they aren't shared,
the files may easily corrupt (depends on what you are exactly doing with
them).
I don't really understand what you say ;-) Do you mean lost data as
'corrupt'?
Since the problem is that the socket buffer fills, it
surprised me that
the server didn't increase SO_RCVBUF. That's not a solution but should
help (already set in /proc/sys/net/core/rmem_default ?).
It is long term CPU saturation issue - mux process isn't fast enough to handle 16
output streams.
Do note, there're quite a few events a second :)
The real issue is: what are you placing on your pipes
that are so slow
to read from them?
Optimizing those scripts could be a simpler solution.
No, those scripts are not the bottleneck, there's plenty of CPU available, and they
are not blocking (for too long, everything is blocking for a certain amount of time ;-)
Wouldn't be hard to make the pipe writes
non-blocking, properly blaming
the slow pipes that couldn't be written
There are no slow pipes. Bottleneck is udp2log step.
Domas