The labsdb1003.eqiad.wmnet (aka c3.labsdb) server is no longer serving *.labsdb requests.
The c3.labsdb service name will continue to point to the labsdb1003.eqiad.wmnet server for the near future, but replication will soon stop there and all tables will be made read-only.
User databases on c1.labsdb and c3.labsdb listed at https://tools.wmflabs.org/tool-db-usage/ will be going away on 2018-01-03. You will need to migrate these to tools.db.svc.eqiad.wmflabs if you need to save the data.
TL;DR
* Change your tools and scripts to use: - "*.web.db.svc.eqiad.wmflabs" (real-time response needed) - "*.analytics.db.svc.eqiad.wmflabs" (batch jobs; long queries) * Replace "*" with either a shard name (e.g. s1) or a wikidb name (e.g. enwiki). * The new servers do not support user created databases/tables because replication can't be guaranteed. See T156869 and below for more information. * Migrate your user created tables to tools.db.svc.eqiad.wmflabs (also known as tools.labsdb) and JOIN via application space logic rather than in-process in the database.
What is changing?
* Wednesday 2017-12-13 ** "*.labsdb" service names switched to point at "*.web.db.svc.eqiad.wmflabs" equivalents. ** User created tables will not be allowed on the new servers. ** "c3.labsdb" still points at labsdb1003.eqiad.wmnet
* Thursday 2017-12-14 ** DBAs will stop replication from production hosts to labsdb1003.eqiad.wmnet ** DBAs will make databases on labsdb1003.eqiad.wmnet read-only for all users
* Wednesday 2018-01-03 ** labsdb1001.eqiad.wmnet (aka c1.labsdb) will be shutdown permanently ** labsdb1003.eqiad.wmnet (aka c3.labsdb) will be shutdown permanently
Why are we doing this?
See https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown and https://phabricator.wikimedia.org/T142807 for a more complete description of the reasons for these changes.
Bryan (on behalf of the Wikimedia Cloud Services and DBA teams)
I sometimes get questions about the difference between "web" and "analytics" databases "which one should I use?", I would like to explain in depth the original idea:
Right now there is no difference between them, except they are served by different set of servers; however, if at some point we run out of resources (not happening yet), web databases will be limited in query time (so ok for multiple connections with fast queries) and optimized for no lag, while analytics databases will be limited in the number of simultaneous connections (it is expected that, as queries can take a long time, you only do 1 or 2 long running queries at a time so other can do the same).
Of course, this is the initial idea to try to cover better both kind of usage, feel free to provide feedback on how to use them better.
On Thu, Dec 14, 2017 at 12:58 AM, Bryan Davis bd808@wikimedia.org wrote:
The labsdb1003.eqiad.wmnet (aka c3.labsdb) server is no longer serving *.labsdb requests.
The c3.labsdb service name will continue to point to the labsdb1003.eqiad.wmnet server for the near future, but replication will soon stop there and all tables will be made read-only.
User databases on c1.labsdb and c3.labsdb listed at https://tools.wmflabs.org/tool-db-usage/ will be going away on 2018-01-03. You will need to migrate these to tools.db.svc.eqiad.wmflabs if you need to save the data.
TL;DR
- Change your tools and scripts to use:
- "*.web.db.svc.eqiad.wmflabs" (real-time response needed)
- "*.analytics.db.svc.eqiad.wmflabs" (batch jobs; long queries)
- Replace "*" with either a shard name (e.g. s1) or a wikidb name (e.g. enwiki).
- The new servers do not support user created databases/tables because replication can't be guaranteed. See T156869 and below for more information.
- Migrate your user created tables to tools.db.svc.eqiad.wmflabs (also known as tools.labsdb) and JOIN via application space logic rather than in-process in the database.
What is changing?
- Wednesday 2017-12-13
** "*.labsdb" service names switched to point at "*.web.db.svc.eqiad.wmflabs" equivalents. ** User created tables will not be allowed on the new servers. ** "c3.labsdb" still points at labsdb1003.eqiad.wmnet
- Thursday 2017-12-14
** DBAs will stop replication from production hosts to labsdb1003.eqiad.wmnet ** DBAs will make databases on labsdb1003.eqiad.wmnet read-only for all users
- Wednesday 2018-01-03
** labsdb1001.eqiad.wmnet (aka c1.labsdb) will be shutdown permanently ** labsdb1003.eqiad.wmnet (aka c3.labsdb) will be shutdown permanently
Why are we doing this?
See https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown and https://phabricator.wikimedia.org/T142807 for a more complete description of the reasons for these changes.
Bryan (on behalf of the Wikimedia Cloud Services and DBA teams)
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Some issues have been found with drivers/connectors that try to auto-negotiate TLS on the new hosts- also happened for toolsdb after its upgrade (_so far only observed on C#/mono applications_): https://phabricator.wikimedia.org/T182892#3839690
We have enabled experimentally TLS on all new servers thinking we may be able to offer it on the future (so far only actually used for administration/monitoring and replication), but for now we only support non-TLS/non-SSL public *client* connections to the database. If you connect to the wikireplicas or toolsdb databases, use mono and get a TLS/SSL error, turn if off for now(;SslMode=none) Most connectors (non-mono: python, php, C, java, node) will be able to connect without TLS, as before, by default. No actions needed for now.
We can explore offering TLS for client connections in the future once we solve the practical and technical issues it requires (non-host based certificates, and client key and certificate distribution, observed TLSv1.2 limited client compatibility, proxy support).
Sorry for the inconveniences,
On Thu, Dec 14, 2017 at 9:48 AM, Jaime Crespo jcrespo@wikimedia.org wrote:
I sometimes get questions about the difference between "web" and "analytics" databases "which one should I use?", I would like to explain in depth the original idea:
Right now there is no difference between them, except they are served by different set of servers; however, if at some point we run out of resources (not happening yet), web databases will be limited in query time (so ok for multiple connections with fast queries) and optimized for no lag, while analytics databases will be limited in the number of simultaneous connections (it is expected that, as queries can take a long time, you only do 1 or 2 long running queries at a time so other can do the same).
Of course, this is the initial idea to try to cover better both kind of usage, feel free to provide feedback on how to use them better.
On Thu, Dec 14, 2017 at 12:58 AM, Bryan Davis bd808@wikimedia.org wrote:
The labsdb1003.eqiad.wmnet (aka c3.labsdb) server is no longer serving *.labsdb requests.
The c3.labsdb service name will continue to point to the labsdb1003.eqiad.wmnet server for the near future, but replication will soon stop there and all tables will be made read-only.
User databases on c1.labsdb and c3.labsdb listed at https://tools.wmflabs.org/tool-db-usage/ will be going away on 2018-01-03. You will need to migrate these to tools.db.svc.eqiad.wmflabs if you need to save the data.
TL;DR
- Change your tools and scripts to use:
- "*.web.db.svc.eqiad.wmflabs" (real-time response needed)
- "*.analytics.db.svc.eqiad.wmflabs" (batch jobs; long queries)
- Replace "*" with either a shard name (e.g. s1) or a wikidb name (e.g. enwiki).
- The new servers do not support user created databases/tables because replication can't be guaranteed. See T156869 and below for more information.
- Migrate your user created tables to tools.db.svc.eqiad.wmflabs (also known as tools.labsdb) and JOIN via application space logic rather than in-process in the database.
What is changing?
- Wednesday 2017-12-13
** "*.labsdb" service names switched to point at "*.web.db.svc.eqiad.wmflabs" equivalents. ** User created tables will not be allowed on the new servers. ** "c3.labsdb" still points at labsdb1003.eqiad.wmnet
- Thursday 2017-12-14
** DBAs will stop replication from production hosts to labsdb1003.eqiad.wmnet ** DBAs will make databases on labsdb1003.eqiad.wmnet read-only for all users
- Wednesday 2018-01-03
** labsdb1001.eqiad.wmnet (aka c1.labsdb) will be shutdown permanently ** labsdb1003.eqiad.wmnet (aka c3.labsdb) will be shutdown permanently
Why are we doing this?
See https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown and https://phabricator.wikimedia.org/T142807 for a more complete description of the reasons for these changes.
Bryan (on behalf of the Wikimedia Cloud Services and DBA teams)
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
-- Jaime Crespo http://wikimedia.org