Hi!
We are working on upgrading one of the critical components of toolforge k8s[1], and we will be rolling it out tomorrow at 10:00UTC on toolforge production.
There's no downtime expected, though upgrading this component specifically has proven bumpy in the past, so we recommend to not run critical loads during that time.
I'll reply to this email when starting the upgrade and when finishing it.
Thanks!
[1] - https://phabricator.wikimedia.org/T394787
This upgrade will start in ~5min, I'll update when it's finished.
On 08/11 11:08, David Caro wrote:
Hi!
We are working on upgrading one of the critical components of toolforge k8s[1], and we will be rolling it out tomorrow at 10:00UTC on toolforge production.
There's no downtime expected, though upgrading this component specifically has proven bumpy in the past, so we recommend to not run critical loads during that time.
I'll reply to this email when starting the upgrade and when finishing it.
Thanks!
[1] - https://phabricator.wikimedia.org/T394787
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
Update on this, the upgrade finished only partially, we are looking into the bits and pieces that are left, but it will take a bit for us to clean up.
There has not been any downtime (reported or detected) and no downtime is expected, it's just taking a bit longer than we though. If you are curious there's more details in the task.
Thanks for your patience!
On 08/12 11:56, David Caro wrote:
This upgrade will start in ~5min, I'll update when it's finished.
On 08/11 11:08, David Caro wrote:
Hi!
We are working on upgrading one of the critical components of toolforge k8s[1], and we will be rolling it out tomorrow at 10:00UTC on toolforge production.
There's no downtime expected, though upgrading this component specifically has proven bumpy in the past, so we recommend to not run critical loads during that time.
I'll reply to this email when starting the upgrade and when finishing it.
Thanks!
[1] - https://phabricator.wikimedia.org/T394787
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
Hi again!
I'm happy to declare the upgrade as done!
There's some cleanup left, but the software is upgraded and running.
Enjoy!
On 08/12 13:04, David Caro wrote:
Update on this, the upgrade finished only partially, we are looking into the bits and pieces that are left, but it will take a bit for us to clean up.
There has not been any downtime (reported or detected) and no downtime is expected, it's just taking a bit longer than we though. If you are curious there's more details in the task.
Thanks for your patience!
On 08/12 11:56, David Caro wrote:
This upgrade will start in ~5min, I'll update when it's finished.
On 08/11 11:08, David Caro wrote:
Hi!
We are working on upgrading one of the critical components of toolforge k8s[1], and we will be rolling it out tomorrow at 10:00UTC on toolforge production.
There's no downtime expected, though upgrading this component specifically has proven bumpy in the past, so we recommend to not run critical loads during that time.
I'll reply to this email when starting the upgrade and when finishing it.
Thanks!
[1] - https://phabricator.wikimedia.org/T394787
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."