Hi there,
We are currently working on replacing older hardware servers with newer
ones, in particular those dedicated to cloud networking [0].
We have discovered a few shortcomings related mostly to network
interface naming in the newer servers, and the latest openstack version
behaving differently to what it used to be, and also some base operating
system (debian) bugs [1]. Some of these are hardware-dependant and
difficult to reproduce/anticipate in our staging environment.
The result is that we are having a more challenging and noisy migration
than we would like. We already had a few (brief) network outages trying
to introduce the new servers into service.
We'll try to keep things as stable as possible in the next few days
until the migration is completed, but we can't discard having some more
(brief) network outages until we are safely on the other side of the
transition.
I'll send another note when we finish this network maintenance is over.
regards.
[0]
https://phabricator.wikimedia.org/T316284
[1]
https://bugs.debian.org/989162
--
Arturo Borrero Gonzalez
Senior Site Reliability Engineer
Wikimedia Cloud Services
Wikimedia Foundation