[Toolserver-l] fwd: wikicnt_daemon.pl

[Gregory Maxwell]

On 7/7/06, Leon Weber <leon.weber at leonweber.de> wrote:
> Yeah. I wanted to create these stats for dewiki, so just like
> duesentrieb said, let the js just call the page with a probability of
> somethin maybe less than 1/1000. But NullC gave me a great idea now:
> I'll just let the js call an empty text file and collect the stats from
> the apachelogs. There's no more efficient way.

To clarify, IFF you disabled logging and you wrote an apache module to
maintain a counter in shared memory using an efficient datastructure
(perhaps a Judy array on page titles) and worked out the locking
issues... it would be faster.

However, apache logging is async and append only.  It's the simplest
form of writing that could happen, and if we moved the logs into tmpfs
it would likely be darn close to optimal.

Although toolserver is disk bound, what is killing us is random seeks
(see iostat, we are constantly pegged at 350-400 TPS but moving less
than 6MB/sec)... so I would not expect much problems from an async and
append only writer because all of it's activity will be mostly
sequential.