> Do we process squid logs?

All Wikistats traffic reports are based on 1:1000 sampled squid logs.

From: analytics-bounces@lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Toby Negrin
Sent: Monday, August 26, 2013 9:33 PM
To: A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics.
Subject: Re: [Analytics] FYI: Inconsistent cache log lines

Do we process squid logs?

On Aug 26, 2013, at 12:31 PM, Diederik van Liere <dvanliere@wikimedia.org> wrote:

On Mon, Aug 26, 2013 at 3:23 PM, Matthew Walker <mwalker@wikimedia.org> wrote:
Whist investigating an orthogonal logging issue I encountered a couple of differences between squid/varnish which I didn't know about:

* Varnish does not give subsecond request time information
Yes that's correct
@ottomata: maybe we can add subsecond request time information to varnish requests?
* Varnish does give subsecond request processing time info
Yes that's correct.
* Varnish calls it 'hit/200' or 'miss/302' instead of 'TCP_MEM_HIT/200' or 'TCP_MISS/302'
Yes that's correct.
* Varnish does not URL encode the user agent field
We never URL encoded the user agent field as far as I know, we did replace space with %20 but have stopped doing that as we are using the tab as separator now. Maybe that patch has not been removed from the squids?

Example log lines:
amssq41.esams.wikimedia.org
1013692039
2013-07-31T23:00:02.331
0
XXX
TCP_MEM_HIT/200
614
GET
http://meta.wikimedia.org/XXX
NONE/-
image/png
http://en.wikipedia.org/XXX
-
Mozilla/5.0%20(Wind...
en-US
en;q=0.8
-
cp1006.eqiad.wmnet
1442176851
2013-07-31T23:00:02.452
0
XXX
TCP_MISS/302
406
GET
http://meta.wikimedia.org/XXX
NONE/-
-
http://en.m.wikipedia.org/XXX
-
Mozilla/5.0%20(i....
en-us
-
cp3012.esams.wikimedia.org
823553992
2013-07-31T23:00:02
0.000119448
XXX
hit/200
20
GET
http://meta.m.wikimedia.org/XXX
-
image/png
http://de.m.wikipedia.org/XXX
XXX
Mozilla/5.0 (iPho...
de-de
-

~Matt Walker
Wikimedia Foundation
Fundraising Technology Team

_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics

_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics