This is a reminder of the timeline dates shared on March 12,
In two weeks -April 15- the old cluster will migrate to utilize new
replication hosts, at which point replication may stop.
In ~four weeks -April 28- the old hostnames will be redirected to the new
cluster.
If you can, please test your code works with the new cluster before the
redirects are in place.
- New host names
<https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#New_ho…>
- Help:Toolforge/Database
<https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database>
For questions and support reach out to #wikimedia-cloud on freenode IRC or
email cloud(a)lists.wikimedia.org
--
Joaquin Oltra Hernandez
Developer Advocate - Wikimedia Foundation
TLDR:
- PAWS can now connect to the new replicas, see News/Wiki Replicas 2020
Redesign#How should I connect to databases in PAWS?
<https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_sh…>
for
more info.
- Report issues here: T276284 Establish a working setup for PAWS with
multi-instance wikireplicas <https://phabricator.wikimedia.org/T276284>
Hi!
PAWS is now capable of connecting and using the new replicas.
Here are some resources you can check:
- News/Wiki Replicas 2020 Redesign#How should I connect to databases in
PAWS?
<https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_sh…>
- Accessing the new replicas, changes from the previous cluster
<https://public.paws.wmcloud.org/User:JHernandez_(WMF)/Accessing%20the%20new…>
- Using Wikireplicas from PAWS with Python
<https://public.paws.wmcloud.org/User:JHernandez_(WMF)/Accessing%20Wikirepli…>
In summary, due to issues with mysql-proxy and the new architecture,
connecting to the replicas will be more in line with the Toolforge approach.
There is a credentials file in $HOME/.my.cnf that you can use when
connecting, instead of the environment variables. For the host name, you
can use the same ones you would use when connecting from Toolforge ("
{wiki}.{analytics,web}.db.svc.wikimedia.cloud").
To update a notebook, here is an example of the couple of changes when
connecting:
- import os
import pymysql
conn = pymysql.connect(
- host = os.environ['MYSQL_HOST'],
+ host = "eswiki.analytics.db.svc.wikimedia.cloud",
- user = os.environ['MYSQL_USERNAME'],
- password = os.environ['MYSQL_PASSWORD'],
+ read_default_file = ".my.cnf",
database = "eswiki_p"
)
Note you have to connect to the host name of the DB you are going to query
against.
Existing notebooks remain readable with the output cached, and we are
working on updating the documentation.
In two weeks -April 15- the old cluster will migrate the old cluster to
utilize new replication hosts, at which point replication may stop and
running PAWS notebooks connecting to the old cluster may get stale results.
In ~four weeks -April 28- the old hostnames will be redirected to the new
cluster, and running notebooks connecting to MYSQL_HOST will not work and
will need updating the credentials and DB host name.
If you find any issues or problems or need help, please reach out via IRC,
mailing list, or in the phabricator task T276284 Establish a working setup
for PAWS with multi-instance wikireplicas
<https://phabricator.wikimedia.org/T276284>
--
Joaquin Oltra Hernandez
Developer Advocate - Wikimedia Foundation
Hi!
The 2020 survey collected feedback from Toolforge project members and Cloud
VPS project administrators on how the services offered can be improved to
help their development and maintenance needs. It was active between
2021-01-14 and 2021-02-05, and had 137 participants.
The summarized results have been published on meta:
https://meta.wikimedia.org/wiki/Research:Cloud_Services_Annual_Survey/2020
Thanks to everyone who participated and provided input and comments. They
are very useful and instrumental in shaping the future and improvements for
the cloud services.
Have a nice day!
--
Joaquin Oltra Hernandez
Developer Advocate - Wikimedia Foundation
With Cinder storage[0] now available and well-tested on cloud-vps, it's
time to move forward to widespread adoption. As of today, use of lvm
(e.g. via role::labs::lvm::srv) is deprecated in favor of cinder volumes
for all local storage outside of the initial 20GB allocated for for
standard OS functions.
In support of this move we'll be making several technical changes to our
infrastructure over the coming days and weeks:
- Default per-project cinder usage quotas will be increased from 10GB to
80GB
- Instance flavors with variable disk sizes will be removed. Going
forward all flavors will have a default local storage size of 20GB
regardless of associated cores or RAM.
- The old lvm puppet roles will continue to function as before on legacy
VMs. On new VMs the roles will detect the lack of available space for
partitioning and report failures[1].
- Instance resizing will be enabled in the Horizon UI. This will allow
you to adjust the allocated cores and RAM allocated to a given VM
without rebuilding.
Manual setup of Cinder volumes is documented here:
https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_ins…
If your workflow involves puppet roles that depend on labs_lvm, you will
need to alter both your puppet code and your manual processes. I've
provided some helper roles to assist. The simplest[2] is the new
'role::labs::cindermount::srv' which in most cases can be used as a
drop-in replacement for role::labs::lvm::srv: once a cinder volume is
attached to a VM that role will detect, format, and mount it as /srv.
The base class cinderutils::ensure[3] can be reused for other variations
on this theme. Please don't hesitate to contact me or other WMCS staff
for assistance with puppet patches!
In some cases the default quota of 80GB will be insufficient for a full
migration to cinder from the old flavor model, especially if you have
made use of the old 'large' or 'xlarge' sizes. When it comes time to to
rebuild these larger VMs, please open a quota request ticket and detail
what VMs (of what flavors) will be deleted and we will try to be quick
and generous with quota expansions to assist in migration.
[0] https://techblog.wikimedia.org/2021/02/05/cinder-on-cloud-vps/
[1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668567
[2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/669958
[3] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668757
TLDR:
- Instead of `*.db.svc.eqiad.wmflabs` use `*.db.svc.wikimedia.cloud` to use
the new replicas
- Quarry will migrate March 23 to use the new cluster
- In a ~month (April 15) the old cluster will start retiring. See
https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeli…
for more details
Hi everyone,
I'm happy to announce that the new replicas cluster is available for use
after a few weeks open for testing.
To use the new cluster, you will only need to change the hostname when
connecting to the databases. Instead of `*.db.svc.eqiad.wmflabs` you can
use `*.db.svc.wikimedia.cloud`. See:
-
https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#New_ho…
-
https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_ca…
- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database
This brings us to the timeline updates:
https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeli…
- March 2021
- Announce new cluster domains for all users - March 12
- Migrate Quarry to use the new cluster - March 23
- April 2021
- PAWS migration - Estimated first week of April
- Migrate the old cluster to utilize new replication hosts. Replication
may stop. - April 15
- Redirect old hostnames to the new cluster - April 28
Quarry is ready so it will transition first to use the new replicas. This
will happen on March 23, at which point we will deploy the changes and
update the docs. We are publishing a note on Tech news but if there are
other venues where this information would be useful please help us spread
the word.
PAWS is in the process of being migrated. We estimate it could be ready to
use the new cluster at the beginning of April, and will publish more
details as soon as we know.
On April 15 the old cluster will be moved under the new replication hosts,
at which point there may be side effects and replication may stop.
We recommend you update your code earlier to use the new hostnames to
verify things are working normally.
Not long after, the old hostnames (`*.db.svc.eqiad.wmflabs`) will be
redirected to the new ones (`*.db.svc.wikimedia.cloud`), at which point the
old cluster will effectively be inaccessible.
If you have questions or need help please reply to cloud(a)lists.wikimedia.org,
join the #wikimedia-cloud IRC channel, or open a ticket in Phabricator with
the Data-Services tag.
Thanks,
Greetings!
The foundation has long since stopped using Debian Stretch for new
projects, and will start actively removing it from the data centers this
summer.
In the meantime, I don't want any one who is on the fence to choose
Stretch when they could choose Buster, so I'm removing it by default
from the image options in Horizon. If you still need to use it for
legacy reasons, feel free to contact a cloud admin or open a phabricator
ticket; restoring access to a specific project is simple. I'm also
leaving it enabled for the deployment-prep and integration projects as
I've no doubt they'll still need to create Stretch VMs until it's fully
phased out in production.
-Andrew