Domas Mituzas wrote:
Hi!
Going multithread is really easy for a socket listener.
Really? :)
Sure. Make each thread call accept and let the kernel give incoming sockets to one of them. There you have the listener done :) Solaris used to need an explicit locking, but it is now fixed there, too.
However, not so much in the LogProcessors. If they are shared accross threads, you may end up with all threads blocked in the fwrite and if they aren't shared, the files may easily corrupt (depends on what you are exactly doing with them).
I don't really understand what you say ;-) Do you mean lost data as 'corrupt'?
Given the following incomint events: udp2log has problems jeluf created a new wiki domas fixed the server
I call corrupted this: jeluf domas udp2log has fixed the server problems created a new wiki
Since the problem is that the socket buffer fills, it surprised me that the server didn't increase SO_RCVBUF. That's not a solution but should help (already set in /proc/sys/net/core/rmem_default ?).
It is long term CPU saturation issue - mux process isn't fast enough to handle 16 output streams. Do note, there're quite a few events a second :)
The real issue is: what are you placing on your pipes that are so slow to read from them? Optimizing those scripts could be a simpler solution.
No, those scripts are not the bottleneck, there's plenty of CPU available, and they are not blocking (for too long, everything is blocking for a certain amount of time ;-)
Wouldn't be hard to make the pipe writes non-blocking, properly blaming the slow pipes that couldn't be written
There are no slow pipes. Bottleneck is udp2log step.
Domas
I don't get it. What is slow on it?
What it does is: 1) Get socket data 2) Split line into pieces 3) fwrite each line in 16 fds 4) Go to 1
If there's plenty of CPU, the pipes doesn't fill, the fwrite doesn't block... Why isn't it coping with it? Too much time lost in context changes?