Hi everyone!
Next monday 8th of September at 08:00UTC we will start doing an upgrade to the Toolforge kubernetes cluster[1].
The upgrade will progress during the day and will eventually restart all running jobs (workers need to be rebooted, so jobs running on them need to be moved to a different worker).
The restarted jobs should restart gracefully by themselves, and there's no service downtime expected (most upgrades have been smooth so far), though there's always a chance that something will go unexpectedly, so if you have long-running reports or similar critical workloads, we recommend waiting until the upgrade is over to run them.
I'll update replying to this message when the upgrade start and finishes.
Thanks!
[1] https://phabricator.wikimedia.org/T402378
The upgrade will be starting now, there's no downtime expected though you might see some of your jobs restarting.
On 09/03 10:17, David Caro wrote:
Hi everyone!
Next monday 8th of September at 08:00UTC we will start doing an upgrade to the Toolforge kubernetes cluster[1].
The upgrade will progress during the day and will eventually restart all running jobs (workers need to be rebooted, so jobs running on them need to be moved to a different worker).
The restarted jobs should restart gracefully by themselves, and there's no service downtime expected (most upgrades have been smooth so far), though there's always a chance that something will go unexpectedly, so if you have long-running reports or similar critical workloads, we recommend waiting until the upgrade is over to run them.
I'll update replying to this message when the upgrade start and finishes.
Thanks!
[1] https://phabricator.wikimedia.org/T402378
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
And upgrade done, everything went nicely :)
If you find new issues or see any errors please open a task or reach out.
Thanks!
On 09/08 10:04, David Caro wrote:
The upgrade will be starting now, there's no downtime expected though you might see some of your jobs restarting.
On 09/03 10:17, David Caro wrote:
Hi everyone!
Next monday 8th of September at 08:00UTC we will start doing an upgrade to the Toolforge kubernetes cluster[1].
The upgrade will progress during the day and will eventually restart all running jobs (workers need to be rebooted, so jobs running on them need to be moved to a different worker).
The restarted jobs should restart gracefully by themselves, and there's no service downtime expected (most upgrades have been smooth so far), though there's always a chance that something will go unexpectedly, so if you have long-running reports or similar critical workloads, we recommend waiting until the upgrade is over to run them.
I'll update replying to this message when the upgrade start and finishes.
Thanks!
[1] https://phabricator.wikimedia.org/T402378
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
-- David Caro SRE - Cloud Services Wikimedia Foundation https://wikimediafoundation.org/ PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
cloud-announce@lists.wikimedia.org