Starting next week, May 27th, connections to Toolforge Redis [0] that
remain idle for more than 10 minutes will be forcibly closed. [1]
This change is in order to prevent a situation where too many
connections are active [2] and Redis stops accepting new connections,
causing an outage [3] for several tools.
If your tool is using Toolforge Redis, we expect it to continue to
work without issues, as most Redis clients automatically create a new
connection to Redis in case an existing one is terminated.
If you have a tool that relies on long-lived Redis connections and
would be negatively impacted by this change, please let us know.
[0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge
[1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1029158
[1] https://phabricator.wikimedia.org/T363709
[2] https://wikitech.wikimedia.org/wiki/Incidents/2024-04-28_WMCS_Toolforge_Red…
--
Francesco Negri (he/him) -- IRC: dhinus
Site Reliability Engineer, Cloud Services team
Wikimedia Foundation
We will be performing maintenance on the Cloud VPS network next
Tuesday (2024-05-21) starting at around 14:00 UTC. During this window
we will be replacing some software on the Cloud VPS network router.[0]
During the maintenance window there will be a brief period during
which Cloud VPS and any services hosted there (including Toolforge and
PAWS) will not have any external network connectivity. Based on tests
done in our staging environment the full outage should last for less
than a minute assuming there are no unexpected issues.
There is no action required on your side, unless your tools are not
resilient to unexpected network outages - in that case you may need to
manually restart those tools after the maintenance is complete.
[0]: https://phabricator.wikimedia.org/T364459
Taavi (+ the rest of the WMCS team)
--
Taavi Väänänen (he/him)
Site Reliability Engineer, Cloud Services
Wikimedia Foundation