Hello deployers,
TLDR: We are upgrading deployment servers, both physical hardware (older R430-> newer R440) and OS version (stretch->buster) [1]
What happened so far:
Today we switched the deployment server and scap master for codfw from deploy2001 to deploy2002. [2]
What happens next:
On Monday, March 1st, we want to switch the deployment server and scap master for eqiad from deploy1001 to deploy1002. [3][4]
The window is "20:00–22:00 UTC # 12:00–14:00 PST" after the morning backport window for up to 2 hours. In this time you won't be able to deploy.
https://wikitech.wikimedia.org/wiki/Deployments#Monday,_March_01
Do you have to do anything?
If you connect to "deployment.eqiad.wmnet" the host key will change, you can use the scripts to update host keys though and the fingerprints are also on wiktech on new (and protected) pages. [5]
Yes, you'll have to retrain muscle memory to switch to deploy1002, sorry for that but it's just a new generation every couple of years and this way we can also fall back to something if needed. In return you should get more performance from the new hardware.
No, you don't need to worry about losing data in your home dir, everything has been rsynced over straight into /home on these hosts.
Let us know if you have any questions,
Daniel & Mukunda
[1] https://phabricator.wikimedia.org/T265963 [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/667043 [3] https://wikitech.wikimedia.org/wiki/Deployments#Monday,_March_01 [4] https://gerrit.wikimedia.org/r/q/topic:%22deployment-switch%22+(status:open) [5] https://wikitech.wikimedia.org/wiki/Deploy1002
deploy1002.eqiad.wmnet is now the active production deployment server :)
Please stop using deploy1001. scap will also be locked though and the MOTD will remind you of it.
On Fri, Feb 26, 2021 at 5:20 PM Daniel Zahn dzahn@wikimedia.org wrote:
Hello deployers,
TLDR: We are upgrading deployment servers, both physical hardware (older R430-> newer R440) and OS version (stretch->buster) [1]
What happened so far:
Today we switched the deployment server and scap master for codfw from deploy2001 to deploy2002. [2]
What happens next:
On Monday, March 1st, we want to switch the deployment server and scap master for eqiad from deploy1001 to deploy1002. [3][4]
The window is "20:00–22:00 UTC # 12:00–14:00 PST" after the morning backport window for up to 2 hours. In this time you won't be able to deploy.
https://wikitech.wikimedia.org/wiki/Deployments#Monday,_March_01
Do you have to do anything?
If you connect to "deployment.eqiad.wmnet" the host key will change, you can use the scripts to update host keys though and the fingerprints are also on wiktech on new (and protected) pages. [5]
Yes, you'll have to retrain muscle memory to switch to deploy1002, sorry for that but it's just a new generation every couple of years and this way we can also fall back to something if needed. In return you should get more performance from the new hardware.
No, you don't need to worry about losing data in your home dir, everything has been rsynced over straight into /home on these hosts.
Let us know if you have any questions,
Daniel & Mukunda
[1] https://phabricator.wikimedia.org/T265963 [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/667043 [3] https://wikitech.wikimedia.org/wiki/Deployments#Monday,_March_01 [4] https://gerrit.wikimedia.org/r/q/topic:%22deployment-switch%22+(status:open) [5] https://wikitech.wikimedia.org/wiki/Deploy1002
-- Daniel Zahn dzahn@wikimedia.org Operations Engineer
deploy1001 is being decom'ed now and won't be able to boot anymore.
The contents of home dirs has been synced to deploy1002.
On Mon, Mar 1, 2021 at 2:58 PM Daniel Zahn dzahn@wikimedia.org wrote:
deploy1002.eqiad.wmnet is now the active production deployment server :)
Please stop using deploy1001. scap will also be locked though and the MOTD will remind you of it.
On Fri, Feb 26, 2021 at 5:20 PM Daniel Zahn dzahn@wikimedia.org wrote:
Hello deployers,
TLDR: We are upgrading deployment servers, both physical hardware (older R430-> newer R440) and OS version (stretch->buster) [1]
What happened so far:
Today we switched the deployment server and scap master for codfw from deploy2001 to deploy2002. [2]
What happens next:
On Monday, March 1st, we want to switch the deployment server and scap master for eqiad from deploy1001 to deploy1002. [3][4]
The window is "20:00–22:00 UTC # 12:00–14:00 PST" after the morning backport window for up to 2 hours. In this time you won't be able to deploy.
https://wikitech.wikimedia.org/wiki/Deployments#Monday,_March_01
Do you have to do anything?
If you connect to "deployment.eqiad.wmnet" the host key will change, you can use the scripts to update host keys though and the fingerprints are also on wiktech on new (and protected) pages. [5]
Yes, you'll have to retrain muscle memory to switch to deploy1002, sorry for that but it's just a new generation every couple of years and this way we can also fall back to something if needed. In return you should get more performance from the new hardware.
No, you don't need to worry about losing data in your home dir, everything has been rsynced over straight into /home on these hosts.
Let us know if you have any questions,
Daniel & Mukunda
[1] https://phabricator.wikimedia.org/T265963 [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/667043 [3] https://wikitech.wikimedia.org/wiki/Deployments#Monday,_March_01 [4] https://gerrit.wikimedia.org/r/q/topic:%22deployment-switch%22+(status:open) [5] https://wikitech.wikimedia.org/wiki/Deploy1002
-- Daniel Zahn dzahn@wikimedia.org Operations Engineer
-- Daniel Zahn dzahn@wikimedia.org Operations Engineer
wikitech-l@lists.wikimedia.org