Cloud June 2024

cloud@lists.wikimedia.org

14 participants
12 discussions

[Cloud-announce] cloud-vps and toolforge outage in progress
by Andrew Bogott 11 Jun '24

11 Jun '24

There is as-of-yet undiagnosed issue with our storage system (ceph) which is causing serious failures throughout cloud-vps and toolforge. Multiple people are working on the issue, so watch this list for updates. Sorry for the downtime! -Andrew + wmcs team _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

[Cloud-announce] Toolforge Redis timeout for idle connections
by Francesco Negri 03 Jun '24

03 Jun '24

Starting next week, May 27th, connections to Toolforge Redis [0] that remain idle for more than 10 minutes will be forcibly closed. [1] This change is in order to prevent a situation where too many connections are active [2] and Redis stops accepting new connections, causing an outage [3] for several tools. If your tool is using Toolforge Redis, we expect it to continue to work without issues, as most Redis clients automatically create a new connection to Redis in case an existing one is terminated. If you have a tool that relies on long-lived Redis connections and would be negatively impacted by this change, please let us know. [0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge [1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1029158 [1] https://phabricator.wikimedia.org/T363709 [2] https://wikitech.wikimedia.org/wiki/Incidents/2024-04-28_WMCS_Toolforge_Red… -- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud June 2024