Team,
We had an spike on EL yesterday nite that was caught by our alarms. The spike can be seen here:
http://graphite.wikimedia.org/render/?width=588&height=311&_salt=140...
As far as I can see it caused no service issues so there is no need to notify users. I am cc-ing the Media team on this e-mail cause from a brief inspection of events it looks like they started to log at a higher rate on the 25th.
Thanks,
Nuria
---------- Forwarded message ---------- From: icinga@neon.wikimedia.org Date: Wed, Jun 25, 2014 at 4:46 AM Subject: ** PROBLEM alert - tungsten/Throughput of event logging events is CRITICAL ** To: nuria@wikimedia.org
❤❤❤❤❤ Icinga ❤❤❤❤❤
Notification Type: PROBLEM
Service: Throughput of event logging events Host: tungsten Address: 10.64.0.18 State: CRITICAL
Date/Time: Wed Jun 25 02:46:22 UTC 2014
Additional Info:
CRITICAL: 7.14% of data exceeded the critical threshold [500.0] Love, Icinga
On Wed, Jun 25, 2014 at 01:36:14PM +0200, Nuria Ruiz wrote:
As far as I can see it caused no service issues so there is no need to notify users. I am cc-ing the Media team on this e-mail cause from a brief inspection of events it looks like they started to log at a higher rate on the 25th.
2014-06-25 - 07:00:53 <marktraceur> nuria: Still looking like our events are the problem? 2014-06-25 - 07:02:47 <nuria> no, marktraceur, my bad
So, it looks like we're off the hook. Thanks for the ping, nuria.
Apologies to multimedia team (bcc-ed) as they are not the cause of the logging spike we saw last nite. Thanks for the fast response in IRC.
On Wed, Jun 25, 2014 at 1:36 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Team,
We had an spike on EL yesterday nite that was caught by our alarms. The spike can be seen here:
http://graphite.wikimedia.org/render/?width=588&height=311&_salt=140...
As far as I can see it caused no service issues so there is no need to notify users. I am cc-ing the Media team on this e-mail cause from a brief inspection of events it looks like they started to log at a higher rate on the 25th.
Thanks,
Nuria
---------- Forwarded message ---------- From: icinga@neon.wikimedia.org Date: Wed, Jun 25, 2014 at 4:46 AM Subject: ** PROBLEM alert - tungsten/Throughput of event logging events is CRITICAL ** To: nuria@wikimedia.org
❤❤❤❤❤ Icinga ❤❤❤❤❤
Notification Type: PROBLEM
Service: Throughput of event logging events Host: tungsten Address: 10.64.0.18 State: CRITICAL
Date/Time: Wed Jun 25 02:46:22 UTC 2014
Additional Info:
CRITICAL: 7.14% of data exceeded the critical threshold [500.0] Love, Icinga
multimedia@lists.wikimedia.org