Hi,
on 2014-01-05 ~03:39 UTC gadolinium went down.
Gadolinium ( https://wikitech.wikimedia.org/wiki/Gadolinium ) acts as
* udp2log relay for anything but emery [1]. * eventlogging relay to vanadium.
As gadolinium is a SPOF for those services, they stopped receiving data when the machine went down.
Gadolinium has been brought up again, and since 2014-01-06 ~17:45 UTC, services should be getting data again.
Webstatscollector is producing good hourly files again on dumps.wikimedia.org.
Data for the tsvs gets collected again. The gzipped tsvs should show up again tomorrow on the usual places.
For eventlogging, I am still awaiting confirmation that is was affected and that it is seeing good data again.
Best regards, Christian
[1] So 5xx tsvs mobile-sampled-100 tsvs zero tsvs edits tsvs [...]
are affected, while only the
sampled-1000 tsvs api-usage tsvs glam-nara tsvs teahouse tsvs arabic-banner tsvs missing-wiki tsvs
are not affected.
Thanks for the report Christian!
On Jan 6, 2014, at 3:57 PM, Christian Aistleitner christian@quelltextlich.at wrote:
Hi,
on 2014-01-05 ~03:39 UTC gadolinium went down.
Gadolinium ( https://wikitech.wikimedia.org/wiki/Gadolinium ) acts as
- udp2log relay for anything but emery [1].
- eventlogging relay to vanadium.
As gadolinium is a SPOF for those services, they stopped receiving data when the machine went down.
Gadolinium has been brought up again, and since 2014-01-06 ~17:45 UTC, services should be getting data again.
Webstatscollector is producing good hourly files again on dumps.wikimedia.org.
Data for the tsvs gets collected again. The gzipped tsvs should show up again tomorrow on the usual places.
For eventlogging, I am still awaiting confirmation that is was affected and that it is seeing good data again.
Best regards, Christian
[1] So 5xx tsvs mobile-sampled-100 tsvs zero tsvs edits tsvs [...]
are affected, while only the
sampled-1000 tsvs api-usage tsvs glam-nara tsvs teahouse tsvs arabic-banner tsvs missing-wiki tsvs
are not affected.
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian@quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/
OpenPGP key transition from 0xEF78CCDE to 0x13C1072F: http://quelltextlich.at/openpgp-transition-0xEF78CCDE-to-0x13C1072F.txt _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hi,
On Mon, Jan 06, 2014 at 09:57:48PM +0100, Christian Aistleitner wrote:
For eventlogging, I am still awaiting confirmation that is was affected and that it is seeing good data again.
Ori helped out (Thanks!) and, yes: EventLogging was affected as well.
The esams part of EventLogging did not get through while gadolinium was down. (EventLogging from other data centers was not affected)
Just as with the other services, EventLogging is back to normal again, and we see esams part making it into EventLogging again.
Best regards, Christian
Well done folks -- thanks for getting these systems back up quickly (and for the excellent communications)
-Toby
On Mon, Jan 6, 2014 at 2:39 PM, Christian Aistleitner < christian@quelltextlich.at> wrote:
Hi,
On Mon, Jan 06, 2014 at 09:57:48PM +0100, Christian Aistleitner wrote:
For eventlogging, I am still awaiting confirmation that is was affected and that it is seeing good data again.
Ori helped out (Thanks!) and, yes: EventLogging was affected as well.
The esams part of EventLogging did not get through while gadolinium was down. (EventLogging from other data centers was not affected)
Just as with the other services, EventLogging is back to normal again, and we see esams part making it into EventLogging again.
Best regards, Christian
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian@quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/
OpenPGP key transition from 0xEF78CCDE to 0x13C1072F: http://quelltextlich.at/openpgp-transition-0xEF78CCDE-to-0x13C1072F.txt
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics