We now have squid logs again, after a long absence. We're using the UDP logging patch I describe here:

http://www.squid-cache.org/mail-archive/squid-dev/200701/0042.html

The log collection program is running on henbane. The log format is our own custom format described here:

https://wikitech.leuksman.com/view/Squid_log_format

Only a sampled log will be stored, the full log will only be available as a real-time stream.

I'd like to invite submissions at this stage for log analysis programs, which can process either the full stream or a sampled stream, aggregate the data and present statistics on the web. There are lots of log analysis packages already available, but I haven't found one yet that is designed to work at such a high request rate.

Currently we only have a raw log, which contains duplicates due to requests forwarded between squids. These should probably be filtered out by the collection daemon before the log is handed out to the analysers.

-- Tim Starling