Hello,

I have taken this quiet (deployment) time to add automatic depooling and repooling of services from the load balancers during deployments~[1]. From now on, when you are doing a deploy, your service will be depooled from the load balancers and repooled back after the service has been successfully restarted on a target node. That means that during a deployment the current target node (being deployed to) will not be in the pool of nodes responding to requests, which in turn means that the number of failed requests due to deployments will be nearly zero.

One thing will change from now on while you do the deployments. Because the load balancers tolerate only 50% of nodes being depooled, deployments will happen in groups of 2; instead of the usual two groups (canary, default), there will now be 5 target groups (canary, default[1-4]). Once the deployment of canaries finishes, you will see the prompt:

canary deploy successful. Continue? [y]es/[n]o/[c]ontinue all groups:

Here, you should start answering with a "c", which tells Scap3 to go through all of the deployment groups. You can also answer "y", but then Scap3 will repeat the question after each group.

I have added patches to the deploy repos of all of the service and tested them. If you are interested in the patch for your respective service, see the task~[1].

Happy holidays!

Cheers, Marko

[1] https://phabricator.wikimedia.org/T144602