Hi everyone,
Lately the k8s cluster has been reaching it's cpu allocation limits, and to solve this we are changing a bit the default cpu limits that we were setting for jobs.
This will be effective today. You might see for a short amount of time that the reported cpu for your jobs is not what you expect, and that your continuous jobs have been restarted to apply the new changes.
This will not have a degrading effect for the jobs, on the other side, it might allow jobs that try to use more cpu to get it, and improve the overall cpu resource usage in the cluster.
Here comes the kubernetes specific notes if anyone is interested :)
Until now, the default CPU limit and request was set to 500m, and if you were requesting less than that, you would get the same request and limit that you requested. If you were asking for more than than, the request would be set to 1/2 of what you asked for, and the limit to exactly what you asked for.
From now on, the default CPU limit will be 4000m, allowing your pods to use way more than before by default if it's available, and the default request will be 100m (roughly the current cluster mean usage), this will help using better the current resources in the cluster.
And if you specify a cpu value, it will be used as both the request and the limit ensuring that your job is allocated to a node only if it has enough cpu.
More notes and details in the task https://phabricator.wikimedia.org/T404726
If we end up changing the memory limits, we will give some advance notice so you can change your resources accordingly if needed.
Thanks!