Cloud-announce April 2021

cloud-announce@lists.wikimedia.org

6 participants
4 discussions

Wiki Replicas: Redirecting the old host names to the new cluster
by Joaquin Oltra Hernandez 30 Apr '21

30 Apr '21

Hi, Like we mentioned in the timeline [0], we are starting the work to redirect the old host names to the new ones [1]. We will share more details in the next few days when we actually deploy the DNS change. This is a heads up before the change actually happens in case you haven't tested your code with the new cluster, last warning. [0]: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeli… [1]: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#New_ho… -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation

1 0

cloud-vps maintenance at 14:00 UTC
by Andrew Bogott 28 Apr '21

28 Apr '21

We will be upgrading the Cloud-VPS OpenStack install later today beginning at 14:00 UTC (7:00 AM Pacific time). The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services. -Andrew + the WMCS Team

4 5

[Toolforge] Outbound emails were stuck from 2021-03-31T14:56Z to 2021-04-20T21:52Z
by Bryan Davis 21 Apr '21

21 Apr '21

TL;DR: * We messed up when replacing the mail server in Toolforge * We didn't notice that we had messed up for nearly 3 weeks * Toolforge servers should be able to send outbound email again now We have been working to replace some of the Cloud VPS instances in the Toolforge project with new instances running Debian Buster (<https://phabricator.wikimedia.org/T275864>). One step in this process was to replace the mail server instance that handles all outbound mail. We setup a new mail server on 2021-03-31, but missed an important configuration step of telling the rest of the instances in the Toolforge project to use the new server when sending outgoing mail. A Toolforge user reported on irc at 2021-04-20T21:11Z that they had not received expected emails from their tool recently. Investigation revealed the broken configuration and work started to correct the problem. Around 2021-04-20T21:52Z we deployed the correct mail relay host configuration. Over the next 30 minutes or so this configuration update rolled out across the Toolforge instances, re-enabling outbound mail sending. Around 2021-04-20T22:20Z we ran commands to instruct all Toolforge instances to "unfreeze" emails which were queued for sending but marked as "frozen" due to the prior invalid configuration. Emails are now being sent out as expected. We apologize for the interruption in service. We will also be looking into some active monitoring system for outbound email delivery to catch problems similar to this more quickly in the future. Bryan, on behalf of the Toolforge admin team -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

cloud-vps maintenance tomorrow (2021-04-06) at 14:30 UTC
by Andrew Bogott 07 Apr '21

07 Apr '21

We will be upgrading the Cloud-VPS OpenStack install tomorrow beginning at 14:30 UTC (7:30 AM Pacific time). The total upgrade should take 60-90 minutes. During the upgrade period Horizon will be disabled. There may also be brief network interruptions as we restart router services. -Andrew + the WMCS Team

1 2

2025

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce April 2021