- Cloud-announce - lists.wikimedia.org

Mounts for Scratch NFS changing a bit tomorrow (2021-05-27 around 20:00 UTC)
by Brooke Storm 26 May '21

26 May '21

Software that uses /data/scratch may see some disruption tomorrow when I merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/695447/ <https://gerrit.wikimedia.org/r/c/operations/puppet/+/695447/> around 20:00 UTC In order to make the NFS it works on more resilient and healthy, the cluster it runs on is migrating to DRBD failover like Tools home and project uses. The current setup is quite broken. If you are running something against scratch when the patch becomes active in your project, you may see some issues like a stale NFS handle. If that doesn’t resolve quickly, please let us know in Libera.chat: #wikimedia-cloud Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org

1 0

#wikimedia-cloud IRC channel available on libera.chat
by Bryan Davis 26 May '21

26 May '21

TL;DR: * The #wikimedia-cloud IRC channel is moving from Freenode to Libera.Chat. * Register an account on Libera.Chat and join us there! There has been a lot of activity over the last 2-3 days related to staffing changes on the Freenode IRC network [0]. The Wikimedia IRC Group Contacts (GCs) [1] evaluated the situation and decided that moving the Wikimedia IRC channels from Freenode to the brand new Libera.Chat IRC network [2] would be the best course of action [3]. So, we are moving! A new #wikimedia-cloud channel has been created on irc.libera.chat for this Wikimedia sub-community to use. The old channel on Freenode still exists and will be maintained at least until we can get all the bots moved, our documentation updated on wikitech, and we see more folks on the Libera channel than the Freenode one. Messages to our channel in either IRC network, as well as the Telegram channel [4], will be seen on all other channels. There is a new subpage on meta [5] for information on how to create a new account for yourself on Libera.Chat and other related information. There is also a tracking task [6] that you can look at to see various activities that the community hopes to take action on to complete the migration. One last thing: The #wmhack Freenode channel is bridged to #wikimedia-hackathon on Libera.Chat. The new channel name will make it easier for the GCs to help manage spam and other issues that come up occasionally on IRC. Don't miss the fun of our 2021 virtual hackathon from Friday, May 21st to Sunday, May 23rd! [7] [0]: https://www.kline.sh/ [1]: https://meta.wikimedia.org/wiki/IRC/Group_Contacts [2]: https://libera.chat/ [3]: https://meta.wikimedia.org/w/index.php?diff=21476411 [4]: https://t.me/wmcloudirc [5]: https://meta.wikimedia.org/wiki/IRC/Migrating_to_Libera_Chat [6]: https://phabricator.wikimedia.org/T283247 [7]: https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021 Bryan, on behalf of the WMCS team and the Cloud VPS and Toolforge admins -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 1

CloudVPS / Toolforge edge network maintenance 2021-05-06 @ 15:00 UTC
by Arturo Borrero Gonzalez 06 May '21

06 May '21

Hello there, We will be doing an upgrade to the CloudVPS edge network Thursday 2021-05-06 @ 15:00 UTC that will likely impact user experience, including Toolforge. We scheduled an 1h operation window. During that time, intermittent network interruption, packet loss and other network problems are to be expected. The edge network maintenance will affect how virtual machines (and Toolforge tools) contact NFS, wiki-replicas, wikis API endpoints, and, in general, any network traffic that flows leaving or entering the cloud (also known as north-south traffic). More information on the operation can be found in phabricator [0] and in wikitech [1]. Regards. [0]https://phabricator.wikimedia.org/T270704 [1] https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme… -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 2

DNS for wikireplicas shifting to the new cluster today
by Brooke Storm 04 May '21

04 May '21

We waited longer than planned in previous communications to move the DNS to the new cluster (scheduled for April 28th, see https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign>), but with the old cluster slated for decommissioning, I’m redirecting the DNS to the new systems today. The old labsdb names are already pointed at it. If you notice problems related to the change that aren’t discussed in the Wikitech page at https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign>, please reach out on Phabricator, the #wikimedia-cloud IRC channel, or the Cloud mailing list. Thanks! Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org

1 0

Wiki Replicas: Redirecting the old host names to the new cluster
by Joaquin Oltra Hernandez 29 Apr '21

29 Apr '21

Hi, Like we mentioned in the timeline [0], we are starting the work to redirect the old host names to the new ones [1]. We will share more details in the next few days when we actually deploy the DNS change. This is a heads up before the change actually happens in case you haven't tested your code with the new cluster, last warning. [0]: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeli… [1]: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#New_ho… -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation

1 0

cloud-vps maintenance at 14:00 UTC
by Andrew Bogott 27 Apr '21

27 Apr '21

We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time). The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services. -Andrew + the WMCS Team

4 5

[Toolforge] Outbound emails were stuck from 2021-03-31T14:56Z to 2021-04-20T21:52Z
by Bryan Davis 21 Apr '21

21 Apr '21

TL;DR: * We messed up when replacing the mail server in Toolforge * We didn't notice that we had messed up for nearly 3 weeks * Toolforge servers should be able to send outbound email again now We have been working to replace some of the Cloud VPS instances in the Toolforge project with new instances running Debian Buster (<https://phabricator.wikimedia.org/T275864>). One step in this process was to replace the mail server instance that handles all outbound mail. We setup a new mail server on 2021-03-31, but missed an important configuration step of telling the rest of the instances in the Toolforge project to use the new server when sending outgoing mail. A Toolforge user reported on irc at 2021-04-20T21:11Z that they had not received expected emails from their tool recently. Investigation revealed the broken configuration and work started to correct the problem. Around 2021-04-20T21:52Z we deployed the correct mail relay host configuration. Over the next 30 minutes or so this configuration update rolled out across the Toolforge instances, re-enabling outbound mail sending. Around 2021-04-20T22:20Z we ran commands to instruct all Toolforge instances to "unfreeze" emails which were queued for sending but marked as "frozen" due to the prior invalid configuration. Emails are now being sent out as expected. We apologize for the interruption in service. We will also be looking into some active monitoring system for outbound email delivery to catch problems similar to this more quickly in the future. Bryan, on behalf of the Toolforge admin team -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

cloud-vps maintenance tomorrow (2021-04-06) at 14:30 UTC
by Andrew Bogott 06 Apr '21

06 Apr '21

We will be upgrading the Cloud-VPS OpenStack install tomorrow beginning at 14:30 UTC (7:30 AM Pacific time). The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services. -Andrew + the WMCS Team

1 2

Wikireplicas: old cluster migrations start in 2 weeks. Please test your code with the new cluster
by Joaquin Oltra Hernandez 30 Mar '21

30 Mar '21

This is a reminder of the timeline dates shared on March 12, In two weeks -April 15- the old cluster will migrate to utilize new replication hosts, at which point replication may stop. In ~four weeks -April 28- the old hostnames will be redirected to the new cluster. If you can, please test your code works with the new cluster before the redirects are in place. - New host names <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#New_ho…> - Help:Toolforge/Database <https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database> For questions and support reach out to #wikimedia-cloud on freenode IRC or email cloud(a)lists.wikimedia.org -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation

1 0

PAWS ready to use the new DB replicas
by Joaquin Oltra Hernandez 30 Mar '21

30 Mar '21

TLDR: - PAWS can now connect to the new replicas, see News/Wiki Replicas 2020 Redesign#How should I connect to databases in PAWS? <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_sh…> for more info. - Report issues here: T276284 Establish a working setup for PAWS with multi-instance wikireplicas <https://phabricator.wikimedia.org/T276284> Hi! PAWS is now capable of connecting and using the new replicas. Here are some resources you can check: - News/Wiki Replicas 2020 Redesign#How should I connect to databases in PAWS? <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_sh…> - Accessing the new replicas, changes from the previous cluster <https://public.paws.wmcloud.org/User:JHernandez_(WMF)/Accessing%20the%20new…> - Using Wikireplicas from PAWS with Python <https://public.paws.wmcloud.org/User:JHernandez_(WMF)/Accessing%20Wikirepli…> In summary, due to issues with mysql-proxy and the new architecture, connecting to the replicas will be more in line with the Toolforge approach. There is a credentials file in $HOME/.my.cnf that you can use when connecting, instead of the environment variables. For the host name, you can use the same ones you would use when connecting from Toolforge (" {wiki}.{analytics,web}.db.svc.wikimedia.cloud"). To update a notebook, here is an example of the couple of changes when connecting: - import os import pymysql conn = pymysql.connect( - host = os.environ['MYSQL_HOST'], + host = "eswiki.analytics.db.svc.wikimedia.cloud", - user = os.environ['MYSQL_USERNAME'], - password = os.environ['MYSQL_PASSWORD'], + read_default_file = ".my.cnf", database = "eswiki_p" ) Note you have to connect to the host name of the DB you are going to query against. Existing notebooks remain readable with the output cached, and we are working on updating the documentation. In two weeks -April 15- the old cluster will migrate the old cluster to utilize new replication hosts, at which point replication may stop and running PAWS notebooks connecting to the old cluster may get stale results. In ~four weeks -April 28- the old hostnames will be redirected to the new cluster, and running notebooks connecting to MYSQL_HOST will not work and will need updating the credentials and DB host name. If you find any issues or problems or need help, please reach out via IRC, mailing list, or in the phabricator task T276284 Establish a working setup for PAWS with multi-instance wikireplicas <https://phabricator.wikimedia.org/T276284> -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce