alright, that’s very unfortunate – thanks Christian for catching this. All these slaves are critical for a variety of scripts that populate dashboards and ad-hoc analysis outside of enwiki and dewiki. 

I’ll immediately file an RT ticket.

Dario

On Apr 21, 2014, at 5:27 PM, Toby Negrin <tnegrin@wikimedia.org> wrote:

Hi Christian --

Thanks for the heads-up. I've verbally notified Dario and the Research and Data team. They will follow up with tech-ops.

-Toby


On Mon, Apr 21, 2014 at 3:33 PM, Christian Aistleitner <christian@quelltextlich.at> wrote:
Hi,

one might be tempted to think that the pmtpa data center having its
servers shut down today should not affect the analytics database
slaves, as they come with “eqiad” in their name:

  s[1-7]-analytics-slave.eqiad.wmnet

.

But it seems at least our s7 slave went down while pmtpa machines are
taken down.

As that looked suspicious, I tried to only look at the first
indirection of the slaves' hostnames, which unmasked all our slaves
(except s1, and s5) as being just CNAMEs for pmtpa machines:

* s2-analytics-slave.eqiad.wmnet (db69.pmtpa.wmnet)
* s3-analytics-slave.eqiad.wmnet (db71.pmtpa.wmnet)
* s4-analytics-slave.eqiad.wmnet (db72.pmtpa.wmnet)
* s6-analytics-slave.eqiad.wmnet (db74.pmtpa.wmnet)
* s7-analytics-slave.eqiad.wmnet (db68.pmtpa.wmnet)

The s7 slave is already being unreachable at the time of writing. I
guess due to the pmpta servers being taken offline. And I expect the
remaining ones in the above list to go down soon as well.

I filed RT #7330 about it:
  https://rt.wikimedia.org/Ticket/Display.html?id=7330

Not clear when replacements will be available or how to
proceed. Waiting for a response from Ops.

Best regards,
Christian


--
---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ----
                           Companies' registry: 360296y in Linz
Christian Aistleitner
Gruendbergstrasze 65a        Email:  christian@quelltextlich.at
4040 Linz, Austria           Phone:          +43 732 / 26 95 63
                             Fax:            +43 732 / 26 95 63
                             Homepage: http://quelltextlich.at/
---------------------------------------------------------------

_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics


_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics