Hi all,

Over the years we've had several serious issues with huge underreporting on page view data due to message loss on udp2log.

There are now several diagnostic tools: alerts are sent and there is real-time monitoring http://tinyurl.com/kqmtfss

But none of those help to quantify total monthly loss.

I upgraded an existing csv file to html report, to be updated monthly.

http://stats.wikimedia.org/wikimedia/squids/SquidDataMonthlyPerSquidSet.htm

This reports show total monthly message loss as a percentage, plus a breakdown of message loss and traffic volume by server role and location.

Basic idea behind the report is that as we use 1:1000 sampling, for each squid server we should find sequence numbers between logged messages to be 1000 apart, on average.

If we actually find they are 1050 apart that translates into 4.7% data loss.

On how this is calculated see http://stats.wikimedia.org/wikimedia/squids/SquidDataMonthlyPerSquidSet.htm#calc

I use a weighted average for calculating total percentage data loss, taking into account data volume per server cluster, and ignoring servers where the sequence number mechanism is still broken (ssl servers).

Role and implementation of udp2log are in flux. But in any setup it would be good to have such overall assessment of loss.

Cheers,

Erik