Team,

We had an spike on EL yesterday nite that was caught by our alarms. The spike can be seen here:

http://graphite.wikimedia.org/render/?width=588&height=311&_salt=1403681192.775&target=eventlogging.overall.valid.rate&target=eventlogging.overall.raw.rate

We look at the data for a little while and we can see the schema 'UniversalLanguageSelector-tofu' logging at a higher rate than it normally does. 

Can you guys look into what there might have been going on? Looks like the "higher than normal logging" might have been triggered by a Localization update that happened yesterday.

The event "02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 25 02:48:53 UTC 2014 (duration 48m 52s)" from https://wikitech.wikimedia.org/wiki/Server_Admin_Log

happened about at the same time we saw the spike. 

Thanks,

Nuria


---------- Forwarded message ----------
From: <icinga@neon.wikimedia.org>
Date: Wed, Jun 25, 2014 at 4:46 AM
Subject: ** PROBLEM alert - tungsten/Throughput of event logging events is CRITICAL **

To: nuria@wikimedia.org


❤❤❤❤❤ Icinga ❤❤❤❤❤

Notification Type: PROBLEM

Service: Throughput of event logging events
Host: tungsten
Address: 10.64.0.18
State: CRITICAL

Date/Time: Wed Jun 25 02:46:22 UTC 2014

Additional Info:

CRITICAL: 7.14% of data exceeded the critical threshold [500.0]
Love, Icinga