Hello,
I have taken this quiet (deployment) time to add automatic depooling and
repooling of services from the load balancers during deployments~[1]. From
now on, when you are doing a deploy, your service will be depooled from the
load balancers and repooled back after the service has been successfully
restarted on a target node. That means that during a deployment the current
target node (being deployed to) will not be in the pool of nodes responding
to requests, which in turn means that the number of failed requests due to
deployments will be nearly zero.
One thing will change from now on while you do the deployments. Because the
load balancers tolerate only 50% of nodes being depooled, deployments will
happen in groups of 2; instead of the usual two groups (canary, default),
there will now be 5 target groups (canary, default[1-4]). Once the
deployment of canaries finishes, you will see the prompt:
canary deploy successful. Continue? [y]es/[n]o/[c]ontinue all groups:
Here, you should start answering with a "c", which tells Scap3 to go
through all of the deployment groups. You can also answer "y", but then
Scap3 will repeat the question after each group.
I have added patches to the deploy repos of all of the service and tested
them. If you are interested in the patch for your respective service, see
the task~[1].
Happy holidays!
Cheers,
Marko
[1] https://phabricator.wikimedia.org/T144602
--
Marko Obrovac, PhD
Senior Services Engineer
Wikimedia Foundation