TL;DR:
* c1.labsdb (labsdb1001.eqiad.wmnet) is down due to hardware issues
* *.labsdb are pointing to c3.labsdb (labsdb1003.eqiad.wmnet)
The physical server behind c1.labsdb (labsdb1001.eqiad.wmnet)
experienced a hard drive failure around 2017-11-01T03:30 UTC. This
failure is preventing the MySQL service on that host from starting.
The *.labsdb service names that were pointed at that server have been
updated to point to c3.labsdb (labsdb1003.eqiad.wmnet) instead.
See <https://phabricator.wikimedia.org/T179464> for more information
and additional updates.
Expect slower than normal performance as all traffic is handled by a
single server. Now would be a great time to update the configuration
for your tools to use the new database cluster [0][1].
[0]:
https://phabricator.wikimedia.org/phame/post/view/70/new_wiki_replica_serveā¦
[1]:
https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA
irc: bd808 v:415.839.6885 x6855