We do care! I generally don’t check my email on weekends, and I didn’t get any SMS texts
about this, hmmmmMMMM!
This is a known occasional problem, see the last reply in this ticket:
https://rt.wikimedia.org/Ticket/Display.html?id=6877
This problem is hard to reproduce. I am waiting for a procurement to be finalized before
I start doing some failover tests to troubleshoot this further.
In the meantime, if this happens, it is easily fixable. In general, you just need to log
into analytics1021 or analytics1022 and run
kafka preferred-replica-election
, which I just did.
See:
https://wikitech.wikimedia.org/wiki/Analytics/Kraken/Kafka/Administration#R…
for more info.
Thanks Greg,
-Ao
On Jun 8, 2014, at 11:08 PM, Greg Grossmeier <greg(a)wikimedia.org> wrote:
----- Forwarded message from Greg Grossmeier
<greg(a)wikimedia.org> -----
Date: Sun, 8 Jun 2014 20:07:41 -0700
From: Greg Grossmeier <greg(a)wikimedia.org>
To: Operations Engineers <ops(a)lists.wikimedia.org>
Subject: Re: Varnishkafka Delivery Errors
<quote name="Greg Grossmeier" date="2014-06-08"
time="15:31:54 -0700">
greg@rose:~/logs/irclogs/2014/Freenode$ grep
"Varnishkafka Delivery Errors" \#wikimedia-operations.06-08.log | wc -l
1354
Last three days (by Eastern US timezone):
greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors"
\#wikimedia-operations.06-06.log | wc -l
0
greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors"
\#wikimedia-operations.06-07.log | wc -l
80
greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors"
\#wikimedia-operations.06-08.log | wc -l
1833
Do we care? If not, holy cow does this need to be silenced.
(Also, get this in Logstash :) )
Greg
--
| Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |
----- End forwarded message -----
--
| Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics