Platonides wrote:
Tim Starling wrote:
But it's not going to happen unless someone
gets around to writing a
program which:
* Accepts URLs on stdin, separated by line breaks
Seems simple.
* Identifies plain page views
I assume you
mean any /wiki/XXXX url, with no '?'. Quite easy, too.
* Breaks them down into per-page counts as
described
And do it really fast... If wgArticleId was also sent, sorting and using
the hashtable, would be easier.
* Provides a TCP query interface
I'd
share the memory hashtable between process, and simply add a
'reader' one. We can live with race conditions, too.
There's only one log stream so I would think there would be only one
process. The task is log analysis, not log collection.
* Does all
that for 30k req/s using less than 10% CPU and 2GB memory
You mean 10% of the
*cluster CPU*, isn't? ;)
10% of one processor. Maybe we could relax it if that proves to be
impossible, but there will only be one log host for now, and there might
be lots of log analysis tools, so we don't want any CPU hogs. Think C++,
not perl.
Impossible?
We could start by profiling. Read data for 5 minutes, compute it for 25.
This would low the rate to 1k req/s.
I could make a log snippet available for optimisation purposes, but
ultimately, it will have to work on the full stream. Sampling would give
you an unacceptable noise floor for the majority of those 25 million
articles.
(to nobody in particular) Thinking about the pipe buffer issues we had the
other day, it might make sense to recompile the kernel on henbane to have
a larger pipe buffer, to cut down on context switches. At 30k req/s, it
would fill every 1.2ms.
-- Tim Starling