[I'm checking this with Nuria now.]

-- Amir Elisha Aharoni‏ ። אָמִיר אֱלִישָׁע אַהֲרוֹנִי Language Engineering‏ ። הַנְדָּסָה לְשׁוֹנִית Wikimedia Foundation‏ ። קֶרֶן וִיקִימֶדְיָה

2014-06-25 18:12 GMT+03:00 Nuria Ruiz nuria@wikimedia.org:

Team,

We had an spike on EL yesterday nite that was caught by our alarms. The spike can be seen here:

http://graphite.wikimedia.org/render/?width=588&height=311&_salt=140...

We look at the data for a little while and we can see the schema 'UniversalLanguageSelector-tofu' logging at a higher rate than it normally does.

Can you guys look into what there might have been going on? Looks like the "higher than normal logging" might have been triggered by a Localization update that happened yesterday.

The event "02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 25 02:48:53 UTC 2014 (duration 48m 52s)" from https://wikitech.wikimedia.org/wiki/Server_Admin_Log happened about at the same time we saw the spike.

Thanks,

Nuria

---------- Forwarded message ---------- From: icinga@neon.wikimedia.org Date: Wed, Jun 25, 2014 at 4:46 AM Subject: ** PROBLEM alert - tungsten/Throughput of event logging events is CRITICAL ** To: nuria@wikimedia.org

❤❤❤❤❤ Icinga ❤❤❤❤❤

Notification Type: PROBLEM

Service: Throughput of event logging events Host: tungsten Address: 10.64.0.18 State: CRITICAL

Date/Time: Wed Jun 25 02:46:22 UTC 2014

Additional Info:

CRITICAL: 7.14% of data exceeded the critical threshold [500.0] Love, Icinga