The labsdb1003.eqiad.wmnet (aka c3.labsdb) server is no longer serving
*.labsdb requests.
The c3.labsdb service name will continue to point to the
labsdb1003.eqiad.wmnet server for the near future, but replication
will soon stop there and all tables will be made read-only.
User databases on c1.labsdb and c3.labsdb listed at
https://tools.wmflabs.org/tool-db-usage/ will be going away on
2018-01-03. You will need to migrate these to
tools.db.svc.eqiad.wmflabs if you need to save the data.
TL;DR
* Change your tools and scripts to use:
- "*.web.db.svc.eqiad.wmflabs" (real-time response needed)
- "*.analytics.db.svc.eqiad.wmflabs" (batch jobs; long queries)
* Replace "*" with either a shard name (e.g. s1) or a wikidb name
(e.g. enwiki).
* The new servers do not support user created databases/tables because
replication can't be guaranteed. See T156869 and below for more
information.
* Migrate your user created tables to tools.db.svc.eqiad.wmflabs
(also known as tools.labsdb) and JOIN via application space logic
rather than in-process in the database.
What is changing?
* Wednesday 2017-12-13
** "*.labsdb" service names switched to point at
"*.web.db.svc.eqiad.wmflabs" equivalents.
** User created tables will not be allowed on the new servers.
** "c3.labsdb" still points at labsdb1003.eqiad.wmnet
* Thursday 2017-12-14
** DBAs will stop replication from production hosts to labsdb1003.eqiad.wmnet
** DBAs will make databases on labsdb1003.eqiad.wmnet read-only for all users
* Wednesday 2018-01-03
** labsdb1001.eqiad.wmnet (aka c1.labsdb) will be shutdown permanently
** labsdb1003.eqiad.wmnet (aka c3.labsdb) will be shutdown permanently
Why are we doing this?
See <https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown>
and <https://phabricator.wikimedia.org/T142807> for a more complete
description of the reasons for these changes.
Bryan (on behalf of the Wikimedia Cloud Services and DBA teams)
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA
irc: bd808 v:415.839.6885 x6855
Hello all,
Some tools running on the Toolforge Kubernetes cluster are currently
suffering from network failures. It's not yet fully diagnosed, although
we have some ideas as to how to at least reduce the impact. The
tracking bug is https://phabricator.wikimedia.org/T182722.
We'll send another update when we have more information and/or when
things are resolved; in the meantime no action is required on your part
as we'll most likely restart affected tools and services ourselves as
part of fixing the problem.
Sorry for the downtime!
-Andrew + the WMCS team