Hi all!
Just an FYI here that this has been done, yay! Varnish, Nginx, and Squid frontends are now all logging with tab as the field delimiter.
For those who would notice, for the time being, we have started outputting logs to new filenames with .tab. in the name, so as to differentiate the format. We will most likely change the file names back to their original names in a month or so.
Thanks all! -Andrew Otto
On Jan 28, 2013, at 11:33 AM, Matthew Flaschen mflaschen@wikimedia.org wrote:
On 01/27/2013 08:07 AM, Erik Zachte wrote:
The code to change existing tabs into some less obnoxious character is dead trivial, hardly any overhead. At worst one field will then be affected, not the whole record, which makes it easier to spot and debug the anomaly when it happens.
Scanning an input record for tabs and raising a counter is also very efficient. Sending one alert hourly based on this counter should make us aware soon enough when this issue needs follow-up, yet without causing bottle necks.
Doing both of those would be pretty robust. However, if that isn't workable, a simple option is just to strip tab characters before Varnish/Squid/etc. writes the line.
That means downstream code doesn't have to do anything special, and it shouldn't affect many actual requests.
Matt Flaschen
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
We will most likely change the file names back to their original names in a month or so
Please don't. It'll serve as a visible marker for the future for when we go back and look at the files and do a WTF.
~Matt Walker
Yes let's not change the filenames D
Sent from my iPhone
On 2013-01-31, at 18:45, Matthew Walker mwalker@wikimedia.org wrote:
We will most likely change the file names back to their original names in a month or so
Please don't. It'll serve as a visible marker for the future for when we go back and look at the files and do a WTF.
~Matt Walker _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Ah, no I mean change the future ones back to their original names. We'd leave the ones that are being generated as '.tab.' now as they are. We could see these filenames in the archives.
But! If everybody loves '.tab.', forever, that's fine with me too!
On Jan 31, 2013, at 6:51 PM, Diederik van Liere dvanliere@wikimedia.org wrote:
Yes let's not change the filenames D
Sent from my iPhone
On 2013-01-31, at 18:45, Matthew Walker mwalker@wikimedia.org wrote:
We will most likely change the file names back to their original names in a month or so
Please don't. It'll serve as a visible marker for the future for when we go back and look at the files and do a WTF.
~Matt Walker _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Thu, Jan 31, 2013 at 7:12 PM, Andrew Otto otto@wikimedia.org wrote:
Ah, no I mean change the future ones back to their original names. We'd leave the ones that are being generated as '.tab.' now as they are. We could see these filenames in the archives.
But! If everybody loves '.tab.', forever, that's fine with me too!
I really don't like .tab.
-Chad
.tsv - tab separated values?
On Thu, Jan 31, 2013 at 7:23 PM, Chad innocentkiller@gmail.com wrote:
On Thu, Jan 31, 2013 at 7:12 PM, Andrew Otto otto@wikimedia.org wrote:
Ah, no I mean change the future ones back to their original names. We'd
leave the ones that are being generated as '.tab.' now as they are. We could see these filenames in the archives.
But! If everybody loves '.tab.', forever, that's fine with me too!
I really don't like .tab.
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
.tsv doh!
That I like! That woulda been so much cleaner and obvious and I wouldn't mind leaving it forever.
Hokay!
On Jan 31, 2013, at 7:25 PM, Yuri Astrakhan yuriastrakhan@gmail.com wrote:
.tsv - tab separated values?
On Thu, Jan 31, 2013 at 7:23 PM, Chad innocentkiller@gmail.com wrote:
On Thu, Jan 31, 2013 at 7:12 PM, Andrew Otto otto@wikimedia.org wrote:
Ah, no I mean change the future ones back to their original names. We'd
leave the ones that are being generated as '.tab.' now as they are. We could see these filenames in the archives.
But! If everybody loves '.tab.', forever, that's fine with me too!
I really don't like .tab.
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org