Cloud-announce March 2021

cloud-announce@lists.wikimedia.org

2 participants
6 discussions

Wikireplicas: old cluster migrations start in 2 weeks. Please test your code with the new cluster
by Joaquin Oltra Hernandez 30 Mar '21

30 Mar '21

This is a reminder of the timeline dates shared on March 12, In two weeks -April 15- the old cluster will migrate to utilize new replication hosts, at which point replication may stop. In ~four weeks -April 28- the old hostnames will be redirected to the new cluster. If you can, please test your code works with the new cluster before the redirects are in place. - New host names <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#New_ho…> - Help:Toolforge/Database <https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database> For questions and support reach out to #wikimedia-cloud on freenode IRC or email cloud(a)lists.wikimedia.org -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation

1 0

PAWS ready to use the new DB replicas
by Joaquin Oltra Hernandez 30 Mar '21

30 Mar '21

TLDR: - PAWS can now connect to the new replicas, see News/Wiki Replicas 2020 Redesign#How should I connect to databases in PAWS? <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_sh…> for more info. - Report issues here: T276284 Establish a working setup for PAWS with multi-instance wikireplicas <https://phabricator.wikimedia.org/T276284> Hi! PAWS is now capable of connecting and using the new replicas. Here are some resources you can check: - News/Wiki Replicas 2020 Redesign#How should I connect to databases in PAWS? <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_sh…> - Accessing the new replicas, changes from the previous cluster <https://public.paws.wmcloud.org/User:JHernandez_(WMF)/Accessing%20the%20new…> - Using Wikireplicas from PAWS with Python <https://public.paws.wmcloud.org/User:JHernandez_(WMF)/Accessing%20Wikirepli…> In summary, due to issues with mysql-proxy and the new architecture, connecting to the replicas will be more in line with the Toolforge approach. There is a credentials file in $HOME/.my.cnf that you can use when connecting, instead of the environment variables. For the host name, you can use the same ones you would use when connecting from Toolforge (" {wiki}.{analytics,web}.db.svc.wikimedia.cloud"). To update a notebook, here is an example of the couple of changes when connecting: - import os import pymysql conn = pymysql.connect( - host = os.environ['MYSQL_HOST'], + host = "eswiki.analytics.db.svc.wikimedia.cloud", - user = os.environ['MYSQL_USERNAME'], - password = os.environ['MYSQL_PASSWORD'], + read_default_file = ".my.cnf", database = "eswiki_p" ) Note you have to connect to the host name of the DB you are going to query against. Existing notebooks remain readable with the output cached, and we are working on updating the documentation. In two weeks -April 15- the old cluster will migrate the old cluster to utilize new replication hosts, at which point replication may stop and running PAWS notebooks connecting to the old cluster may get stale results. In ~four weeks -April 28- the old hostnames will be redirected to the new cluster, and running notebooks connecting to MYSQL_HOST will not work and will need updating the credentials and DB host name. If you find any issues or problems or need help, please reach out via IRC, mailing list, or in the phabricator task T276284 Establish a working setup for PAWS with multi-instance wikireplicas <https://phabricator.wikimedia.org/T276284> -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation

1 0

Cloud Services Survey 2020 results published
by Joaquin Oltra Hernandez 26 Mar '21

26 Mar '21

Hi! The 2020 survey collected feedback from Toolforge project members and Cloud VPS project administrators on how the services offered can be improved to help their development and maintenance needs. It was active between 2021-01-14 and 2021-02-05, and had 137 participants. The summarized results have been published on meta: https://meta.wikimedia.org/wiki/Research:Cloud_Services_Annual_Survey/2020 Thanks to everyone who participated and provided input and comments. They are very useful and instrumental in shaping the future and improvements for the cloud services. Have a nice day! -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation

1 0

The future of storage and /srv on cloud-vps
by Andrew Bogott 17 Mar '21

17 Mar '21

With Cinder storage[0] now available and well-tested on cloud-vps, it's time to move forward to widespread adoption. As of today, use of lvm (e.g. via role::labs::lvm::srv) is deprecated in favor of cinder volumes for all local storage outside of the initial 20GB allocated for for standard OS functions. In support of this move we'll be making several technical changes to our infrastructure over the coming days and weeks: - Default per-project cinder usage quotas will be increased from 10GB to 80GB - Instance flavors with variable disk sizes will be removed. Going forward all flavors will have a default local storage size of 20GB regardless of associated cores or RAM. - The old lvm puppet roles will continue to function as before on legacy VMs. On new VMs the roles will detect the lack of available space for partitioning and report failures[1]. - Instance resizing will be enabled in the Horizon UI. This will allow you to adjust the allocated cores and RAM allocated to a given VM without rebuilding. Manual setup of Cinder volumes is documented here: https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_ins… If your workflow involves puppet roles that depend on labs_lvm, you will need to alter both your puppet code and your manual processes. I've provided some helper roles to assist. The simplest[2] is the new 'role::labs::cindermount::srv' which in most cases can be used as a drop-in replacement for role::labs::lvm::srv: once a cinder volume is attached to a VM that role will detect, format, and mount it as /srv. The base class cinderutils::ensure[3] can be reused for other variations on this theme. Please don't hesitate to contact me or other WMCS staff for assistance with puppet patches! In some cases the default quota of 80GB will be insufficient for a full migration to cinder from the old flavor model, especially if you have made use of the old 'large' or 'xlarge' sizes. When it comes time to to rebuild these larger VMs, please open a quota request ticket and detail what VMs (of what flavors) will be deleted and we will try to be quick and generous with quota expansions to assist in migration. [0] https://techblog.wikimedia.org/2021/02/05/cinder-on-cloud-vps/ [1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668567 [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/669958 [3] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668757

1 0

New Wikireplicas available, timeline update, and Quarry migration
by Joaquin Oltra Hernandez 12 Mar '21

12 Mar '21

TLDR: - Instead of `*.db.svc.eqiad.wmflabs` use `*.db.svc.wikimedia.cloud` to use the new replicas - Quarry will migrate March 23 to use the new cluster - In a ~month (April 15) the old cluster will start retiring. See https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeli… for more details Hi everyone, I'm happy to announce that the new replicas cluster is available for use after a few weeks open for testing. To use the new cluster, you will only need to change the hostname when connecting to the databases. Instead of `*.db.svc.eqiad.wmflabs` you can use `*.db.svc.wikimedia.cloud`. See: - https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#New_ho… - https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_ca… - https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database This brings us to the timeline updates: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeli… - March 2021 - Announce new cluster domains for all users - March 12 - Migrate Quarry to use the new cluster - March 23 - April 2021 - PAWS migration - Estimated first week of April - Migrate the old cluster to utilize new replication hosts. Replication may stop. - April 15 - Redirect old hostnames to the new cluster - April 28 Quarry is ready so it will transition first to use the new replicas. This will happen on March 23, at which point we will deploy the changes and update the docs. We are publishing a note on Tech news but if there are other venues where this information would be useful please help us spread the word. PAWS is in the process of being migrated. We estimate it could be ready to use the new cluster at the beginning of April, and will publish more details as soon as we know. On April 15 the old cluster will be moved under the new replication hosts, at which point there may be side effects and replication may stop. We recommend you update your code earlier to use the new hostnames to verify things are working normally. Not long after, the old hostnames (`*.db.svc.eqiad.wmflabs`) will be redirected to the new ones (`*.db.svc.wikimedia.cloud`), at which point the old cluster will effectively be inaccessible. If you have questions or need help please reply to cloud(a)lists.wikimedia.org, join the #wikimedia-cloud IRC channel, or open a ticket in Phabricator with the Data-Services tag. Thanks,

1 0

Use of Debian Stretch now discouraged
by Andrew Bogott 12 Mar '21

12 Mar '21

Greetings! The foundation has long since stopped using Debian Stretch for new projects, and will start actively removing it from the data centers this summer. In the meantime, I don't want any one who is on the fence to choose Stretch when they could choose Buster, so I'm removing it by default from the image options in Horizon. If you still need to use it for legacy reasons, feel free to contact a cloud admin or open a phabricator ticket; restoring access to a specific project is simple. I'm also leaving it enabled for the deployment-prep and integration projects as I've no doubt they'll still need to create Stretch VMs until it's fully phased out in production. -Andrew

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce March 2021