----- Forwarded message from Greg Grossmeier greg@wikimedia.org -----
Date: Sun, 8 Jun 2014 20:07:41 -0700 From: Greg Grossmeier greg@wikimedia.org To: Operations Engineers ops@lists.wikimedia.org Subject: Re: Varnishkafka Delivery Errors
<quote name="Greg Grossmeier" date="2014-06-08" time="15:31:54 -0700"> > greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors" \#wikimedia-operations.06-08.log | wc -l > 1354
Last three days (by Eastern US timezone):
greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors" #wikimedia-operations.06-06.log | wc -l 0
greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors" #wikimedia-operations.06-07.log | wc -l 80
greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors" #wikimedia-operations.06-08.log | wc -l 1833
Do we care? If not, holy cow does this need to be silenced.
(Also, get this in Logstash :) )
Greg
-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |
----- End forwarded message -----
We do care! I generally don’t check my email on weekends, and I didn’t get any SMS texts about this, hmmmmMMMM!
This is a known occasional problem, see the last reply in this ticket: https://rt.wikimedia.org/Ticket/Display.html?id=6877
This problem is hard to reproduce. I am waiting for a procurement to be finalized before I start doing some failover tests to troubleshoot this further.
In the meantime, if this happens, it is easily fixable. In general, you just need to log into analytics1021 or analytics1022 and run kafka preferred-replica-election
, which I just did.
See: https://wikitech.wikimedia.org/wiki/Analytics/Kraken/Kafka/Administration#Re... for more info.
Thanks Greg, -Ao
On Jun 8, 2014, at 11:08 PM, Greg Grossmeier greg@wikimedia.org wrote:
----- Forwarded message from Greg Grossmeier greg@wikimedia.org -----
Date: Sun, 8 Jun 2014 20:07:41 -0700 From: Greg Grossmeier greg@wikimedia.org To: Operations Engineers ops@lists.wikimedia.org Subject: Re: Varnishkafka Delivery Errors
<quote name="Greg Grossmeier" date="2014-06-08" time="15:31:54 -0700"> > greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors" \#wikimedia-operations.06-08.log | wc -l > 1354
Last three days (by Eastern US timezone):
greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors" #wikimedia-operations.06-06.log | wc -l 0
greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors" #wikimedia-operations.06-07.log | wc -l 80
greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors" #wikimedia-operations.06-08.log | wc -l 1833
Do we care? If not, holy cow does this need to be silenced.
(Also, get this in Logstash :) )
Greg
-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |
----- End forwarded message -----
-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D | _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics