We now have squid logs again, after a long absence. We're using the UDP logging patch I describe here:
http://www.squid-cache.org/mail-archive/squid-dev/200701/0042.html
The log collection program is running on henbane. The log format is our own custom format described here:
https://wikitech.leuksman.com/view/Squid_log_format
Only a sampled log will be stored, the full log will only be available as a real-time stream.
I'd like to invite submissions at this stage for log analysis programs, which can process either the full stream or a sampled stream, aggregate the data and present statistics on the web. There are lots of log analysis packages already available, but I haven't found one yet that is designed to work at such a high request rate.
Currently we only have a raw log, which contains duplicates due to requests forwarded between squids. These should probably be filtered out by the collection daemon before the log is handed out to the analysers.
-- Tim Starling