March 2021 - Cloud - lists.wikimedia.org

[Cloud-announce] Wikireplicas: old cluster migrations start in 2 weeks. Please test your code with the new cluster
by Joaquin Oltra Hernandez 06 Apr '21

06 Apr '21

This is a reminder of the timeline dates shared on March 12, In two weeks -April 15- the old cluster will migrate to utilize new replication hosts, at which point replication may stop. In ~four weeks -April 28- the old hostnames will be redirected to the new cluster. If you can, please test your code works with the new cluster before the redirects are in place. - New host names <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#New_ho…> - Help:Toolforge/Database <https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database> For questions and support reach out to #wikimedia-cloud on freenode IRC or email cloud(a)lists.wikimedia.org -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

4 5

[Cloud-announce] New Wikireplicas available, timeline update, and Quarry migration
by Joaquin Oltra Hernandez 05 Apr '21

05 Apr '21

TLDR: - Instead of `*.db.svc.eqiad.wmflabs` use `*.db.svc.wikimedia.cloud` to use the new replicas - Quarry will migrate March 23 to use the new cluster - In a ~month (April 15) the old cluster will start retiring. See https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeli… for more details Hi everyone, I'm happy to announce that the new replicas cluster is available for use after a few weeks open for testing. To use the new cluster, you will only need to change the hostname when connecting to the databases. Instead of `*.db.svc.eqiad.wmflabs` you can use `*.db.svc.wikimedia.cloud`. See: - https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#New_ho… - https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_ca… - https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database This brings us to the timeline updates: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeli… - March 2021 - Announce new cluster domains for all users - March 12 - Migrate Quarry to use the new cluster - March 23 - April 2021 - PAWS migration - Estimated first week of April - Migrate the old cluster to utilize new replication hosts. Replication may stop. - April 15 - Redirect old hostnames to the new cluster - April 28 Quarry is ready so it will transition first to use the new replicas. This will happen on March 23, at which point we will deploy the changes and update the docs. We are publishing a note on Tech news but if there are other venues where this information would be useful please help us spread the word. PAWS is in the process of being migrated. We estimate it could be ready to use the new cluster at the beginning of April, and will publish more details as soon as we know. On April 15 the old cluster will be moved under the new replication hosts, at which point there may be side effects and replication may stop. We recommend you update your code earlier to use the new hostnames to verify things are working normally. Not long after, the old hostnames (`*.db.svc.eqiad.wmflabs`) will be redirected to the new ones (`*.db.svc.wikimedia.cloud`), at which point the old cluster will effectively be inaccessible. If you have questions or need help please reply to cloud(a)lists.wikimedia.org, join the #wikimedia-cloud IRC channel, or open a ticket in Phabricator with the Data-Services tag. Thanks, _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

12 20

Wikimedia remote hackathon on May 22/23!
by Birgit Müller 31 Mar '21

31 Mar '21

Hello All, Many of us were hopeful that we would be able to organise an onsite hackathon and meet in person in 2021. While this is sadly not the case, we still wanted to offer the opportunity for the technical community to get together virtually, work together on various projects, and discuss new ideas. This is why we’re happy to invite you all to join the ‘21 Wikimedia remote hackathon on May 22/23 <https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021> - save the date, and bring your projects & ideas! Building on last year’s edition, this event is organized in a light mode, offering a lot of space for spontaneity, experimentation, and all kinds of projects. The ‘21 Wikimedia remote hackathon is built by and for its participants, and coordinated by a team of volunteers and staff - Amir, Birgit, Joaquin, Léa, Mohammed, Neslihan, Pavritha. On this page <https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021> you will find all the relevant information. The planned framework consists of one main track of sessions that participants can follow, open rooms for informal discussions, workshops and social events, and to work together on projects. We will send out more information on how to schedule a session in the program soon! You can also add yourself to the participants list <https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021/Participants>, and mention if you would like to help with tasks such as facilitation, or welcoming newcomers. We hope that this event can be a moment of fun, reconnecting with long-time-no-see Wikimedians as well as onboarding new people into the technical community. If you have any questions, feel free to reach out to us on the communication channels of the event <https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021/Discussions>. Hope to see you there! The ‘21 remote hack coordination team -- Birgit Müller (she/her) Director of Technical Engagement Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

[Cloud-announce] PAWS ready to use the new DB replicas
by Joaquin Oltra Hernandez 30 Mar '21

30 Mar '21

TLDR: - PAWS can now connect to the new replicas, see News/Wiki Replicas 2020 Redesign#How should I connect to databases in PAWS? <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_sh…> for more info. - Report issues here: T276284 Establish a working setup for PAWS with multi-instance wikireplicas <https://phabricator.wikimedia.org/T276284> Hi! PAWS is now capable of connecting and using the new replicas. Here are some resources you can check: - News/Wiki Replicas 2020 Redesign#How should I connect to databases in PAWS? <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_sh…> - Accessing the new replicas, changes from the previous cluster <https://public.paws.wmcloud.org/User:JHernandez_(WMF)/Accessing%20the%20new…> - Using Wikireplicas from PAWS with Python <https://public.paws.wmcloud.org/User:JHernandez_(WMF)/Accessing%20Wikirepli…> In summary, due to issues with mysql-proxy and the new architecture, connecting to the replicas will be more in line with the Toolforge approach. There is a credentials file in $HOME/.my.cnf that you can use when connecting, instead of the environment variables. For the host name, you can use the same ones you would use when connecting from Toolforge (" {wiki}.{analytics,web}.db.svc.wikimedia.cloud"). To update a notebook, here is an example of the couple of changes when connecting: - import os import pymysql conn = pymysql.connect( - host = os.environ['MYSQL_HOST'], + host = "eswiki.analytics.db.svc.wikimedia.cloud", - user = os.environ['MYSQL_USERNAME'], - password = os.environ['MYSQL_PASSWORD'], + read_default_file = ".my.cnf", database = "eswiki_p" ) Note you have to connect to the host name of the DB you are going to query against. Existing notebooks remain readable with the output cached, and we are working on updating the documentation. In two weeks -April 15- the old cluster will migrate the old cluster to utilize new replication hosts, at which point replication may stop and running PAWS notebooks connecting to the old cluster may get stale results. In ~four weeks -April 28- the old hostnames will be redirected to the new cluster, and running notebooks connecting to MYSQL_HOST will not work and will need updating the credentials and DB host name. If you find any issues or problems or need help, please reach out via IRC, mailing list, or in the phabricator task T276284 Establish a working setup for PAWS with multi-instance wikireplicas <https://phabricator.wikimedia.org/T276284> -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

[Cloud-announce] Cloud Services Survey 2020 results published
by Joaquin Oltra Hernandez 25 Mar '21

25 Mar '21

Hi! The 2020 survey collected feedback from Toolforge project members and Cloud VPS project administrators on how the services offered can be improved to help their development and maintenance needs. It was active between 2021-01-14 and 2021-02-05, and had 137 participants. The summarized results have been published on meta: https://meta.wikimedia.org/wiki/Research:Cloud_Services_Annual_Survey/2020 Thanks to everyone who participated and provided input and comments. They are very useful and instrumental in shaping the future and improvements for the cloud services. Have a nice day! -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

Install emacs 25 on bastion hosts?
by Roy Smith 22 Mar '21

22 Mar '21

Not sure if this is the right place to ask, but would it be possible to install emacs-25 on the bastion hosts? Apparently, the python mode in emacs-25 knows about the new python-3.5 asyncio keywords (async def, etc), but emacs-24 doesn't. One of these looks like the right thing: emacs25-lucid/oldstable,oldstable 25.1+1-4+deb9u1 amd64 GNU Emacs editor (with Lucid GUI support) emacs25-nox/oldstable,oldstable 25.1+1-4+deb9u1 amd64 GNU Emacs editor (without GUI support)

3 3

[Cloud-announce] The future of storage and /srv on cloud-vps
by Andrew Bogott 17 Mar '21

17 Mar '21

With Cinder storage[0] now available and well-tested on cloud-vps, it's time to move forward to widespread adoption. As of today, use of lvm (e.g. via role::labs::lvm::srv) is deprecated in favor of cinder volumes for all local storage outside of the initial 20GB allocated for for standard OS functions. In support of this move we'll be making several technical changes to our infrastructure over the coming days and weeks: - Default per-project cinder usage quotas will be increased from 10GB to 80GB - Instance flavors with variable disk sizes will be removed. Going forward all flavors will have a default local storage size of 20GB regardless of associated cores or RAM. - The old lvm puppet roles will continue to function as before on legacy VMs. On new VMs the roles will detect the lack of available space for partitioning and report failures[1]. - Instance resizing will be enabled in the Horizon UI. This will allow you to adjust the allocated cores and RAM allocated to a given VM without rebuilding. Manual setup of Cinder volumes is documented here: https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_ins… If your workflow involves puppet roles that depend on labs_lvm, you will need to alter both your puppet code and your manual processes. I've provided some helper roles to assist. The simplest[2] is the new 'role::labs::cindermount::srv' which in most cases can be used as a drop-in replacement for role::labs::lvm::srv: once a cinder volume is attached to a VM that role will detect, format, and mount it as /srv. The base class cinderutils::ensure[3] can be reused for other variations on this theme. Please don't hesitate to contact me or other WMCS staff for assistance with puppet patches! In some cases the default quota of 80GB will be insufficient for a full migration to cinder from the old flavor model, especially if you have made use of the old 'large' or 'xlarge' sizes. When it comes time to to rebuild these larger VMs, please open a quota request ticket and detail what VMs (of what flavors) will be deleted and we will try to be quick and generous with quota expansions to assist in migration. [0] https://techblog.wikimedia.org/2021/02/05/cinder-on-cloud-vps/ [1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668567 [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/669958 [3] https://gerrit.wikimedia.org/r/c/operations/puppet/+/668757 _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

[Cloud-announce] Use of Debian Stretch now discouraged
by Andrew Bogott 11 Mar '21

11 Mar '21

Greetings! The foundation has long since stopped using Debian Stretch for new projects, and will start actively removing it from the data centers this summer. In the meantime, I don't want any one who is on the fence to choose Stretch when they could choose Buster, so I'm removing it by default from the image options in Horizon. If you still need to use it for legacy reasons, feel free to contact a cloud admin or open a phabricator ticket; restoring access to a specific project is simple. I'm also leaving it enabled for the deployment-prep and integration projects as I've no doubt they'll still need to create Stretch VMs until it's fully phased out in production. -Andrew _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

Problem on linking Wikimedia account and LDAP account
by Kirk Lu 05 Mar '21

05 Mar '21

Hello, I am KirkLU (with the same username in wikimedia, phab and wikitech). I find a problem while I am trying to link my Wikimedia account and LDAP account, and cannot resolve it. Everytime I try to *login using OAuth* in my setting of Toolforge admin console, I have been noticed that "*Wikimedia account 'KirkLU' is already attached to another LDAP account*". However, I cannot remember which LDAP account I have linked to. I tried to find it in the User List, but failed. I am wondering if there is a way to find the old account, unlink the old connection and attach my Wikimedia account (KirkLU) to the LDAP account (KirkLU). Additionally, my phab account<https://phabricator.wikimedia.org/p/KirkLU/> has been linked with both my LDAP account (KirkLU) and Wikimedia account (KirkLU). Really hope someone can help me resolve this issue. And thank you very much for your attention. Best regards, Kirk -- User:KirkLU | Chinese Wikimedian admin@zhwp, mOTRS@info-zh Email: kirklu.wm(a)gmail.com https://meta.wikimedia.org/wiki/User:KirkLU

3 6

2024

2023

2022

2021

2020

2019

2018

2017

Cloud March 2021