[I'm checking this with Nuria now.]
--
Amir Elisha Aharoni ። אָמִיר אֱלִישָׁע אַהֲרוֹנִי
Language Engineering ። הַנְדָּסָה לְשׁוֹנִית
Wikimedia Foundation ። קֶרֶן וִיקִימֶדְיָה
2014-06-25 18:12 GMT+03:00 Nuria Ruiz <nuria(a)wikimedia.org>rg>:
Team,
We had an spike on EL yesterday nite that was caught by our alarms. The
spike can be seen here:
http://graphite.wikimedia.org/render/?width=588&height=311&_salt=14…
We look at the data for a little while and we can see the schema
'UniversalLanguageSelector-tofu' logging at a higher rate than it normally
does.
Can you guys look into what there might have been going on? Looks like the
"higher than normal logging" might have been triggered by a Localization
update that happened yesterday.
The event "02:50 logmsgbot: LocalisationUpdate ResourceLoader cache
refresh completed at Wed Jun 25 02:48:53 UTC 2014 (duration 48m 52s)" from
https://wikitech.wikimedia.org/wiki/Server_Admin_Log
happened about at the same time we saw the spike.
Thanks,
Nuria
---------- Forwarded message ----------
From: <icinga(a)neon.wikimedia.org>
Date: Wed, Jun 25, 2014 at 4:46 AM
Subject: ** PROBLEM alert - tungsten/Throughput of event logging events
is CRITICAL **
To: nuria(a)wikimedia.org
❤❤❤❤❤ Icinga ❤❤❤❤❤
Notification Type: PROBLEM
Service: Throughput of event logging events
Host: tungsten
Address: 10.64.0.18
State: CRITICAL
Date/Time: Wed Jun 25 02:46:22 UTC 2014
Additional Info:
CRITICAL: 7.14% of data exceeded the critical threshold [500.0]
Love, Icinga