Hello everyone,
TL;DR if you are not deploying services to the eqiad kubernetes cluster, you can safely skip this.
Long version:
After having tested thrice our cluster reinitialization procedure, next week, on Tuesday 2021-03-23 we will be reinitializing our eqiad kubernetes cluster. All traffic will be drained from it beforehand and we expect no user visible impact. However, for the duration of the process, the kubernetes eqiad cluster will be unavailable to deployers and thus efforts to deploy to it will fail or worse, not have the expected outcomes. This is normal until SRE serviceops announces that the cluster is fully operational again.
SRE service-ops will be deploying all services before marking the cluster as usable and pooling traffic back to it, so there will be no need for deployers to re-deploy their services.
For your convenience the list of services that are currently deployed on that cluster is: apertium api-gateway blubberoid changeprop changeprop-jobqueue citoid cxserver echostore eventgate-analytics eventgate-analytics-external eventgate-logging-external eventgate-main eventstreams eventstreams-internal linkrecommendation mathoid mobileapps proton push-notifications recommendation-api sessionstore similar-users termbox wikifeeds zotero
Regards,
-- Alexandros Kosiaris Principal Site Reliability Engineer Wikimedia Foundation
Hello everyone,
This is starting now. Keep in mind that if you try to deploy to eqiad k8s today, it WILL fail or just won't do what you expect it to do.
On Fri, Mar 19, 2021 at 10:02 PM Alexandros Kosiaris akosiaris@wikimedia.org wrote:
Hello everyone,
TL;DR if you are not deploying services to the eqiad kubernetes cluster, you can safely skip this.
Long version:
After having tested thrice our cluster reinitialization procedure, next week, on Tuesday 2021-03-23 we will be reinitializing our eqiad kubernetes cluster. All traffic will be drained from it beforehand and we expect no user visible impact. However, for the duration of the process, the kubernetes eqiad cluster will be unavailable to deployers and thus efforts to deploy to it will fail or worse, not have the expected outcomes. This is normal until SRE serviceops announces that the cluster is fully operational again.
SRE service-ops will be deploying all services before marking the cluster as usable and pooling traffic back to it, so there will be no need for deployers to re-deploy their services.
For your convenience the list of services that are currently deployed on that cluster is: apertium api-gateway blubberoid changeprop changeprop-jobqueue citoid cxserver echostore eventgate-analytics eventgate-analytics-external eventgate-logging-external eventgate-main eventstreams eventstreams-internal linkrecommendation mathoid mobileapps proton push-notifications recommendation-api sessionstore similar-users termbox wikifeeds zotero
Regards,
-- Alexandros Kosiaris Principal Site Reliability Engineer Wikimedia Foundation
Hello everyone,
This has happened. The cluster has been reinitialized and upgraded and all services have been redeployed by SRE Service Operations. So, the cluster is fully operational again, feel free to deploy. Traffic hasn't been switched yet back as we are still making sure that it's also fully traffic capable as well, but it's expected to happen at the latest tomorrow.
On Tue, Mar 23, 2021 at 10:23 AM Alexandros Kosiaris akosiaris@wikimedia.org wrote:
Hello everyone,
This is starting now. Keep in mind that if you try to deploy to eqiad k8s today, it WILL fail or just won't do what you expect it to do.
On Fri, Mar 19, 2021 at 10:02 PM Alexandros Kosiaris akosiaris@wikimedia.org wrote:
Hello everyone,
TL;DR if you are not deploying services to the eqiad kubernetes cluster, you can safely skip this.
Long version:
After having tested thrice our cluster reinitialization procedure, next week, on Tuesday 2021-03-23 we will be reinitializing our eqiad kubernetes cluster. All traffic will be drained from it beforehand and we expect no user visible impact. However, for the duration of the process, the kubernetes eqiad cluster will be unavailable to deployers and thus efforts to deploy to it will fail or worse, not have the expected outcomes. This is normal until SRE serviceops announces that the cluster is fully operational again.
SRE service-ops will be deploying all services before marking the cluster as usable and pooling traffic back to it, so there will be no need for deployers to re-deploy their services.
For your convenience the list of services that are currently deployed on that cluster is: apertium api-gateway blubberoid changeprop changeprop-jobqueue citoid cxserver echostore eventgate-analytics eventgate-analytics-external eventgate-logging-external eventgate-main eventstreams eventstreams-internal linkrecommendation mathoid mobileapps proton push-notifications recommendation-api sessionstore similar-users termbox wikifeeds zotero
Regards,
-- Alexandros Kosiaris Principal Site Reliability Engineer Wikimedia Foundation
-- Alexandros Kosiaris Principal Site Reliability Engineer Wikimedia Foundation
Hi,
all services in the cluster are completely repooled now. If you do see anything unexpected, please let us know in phabricator: https://phabricator.wikimedia.org/T277741
Cheers, Janis
On Dienstag, 23. März 2021 13:57:13 CET Alexandros Kosiaris wrote:
Hello everyone,
This has happened. The cluster has been reinitialized and upgraded and all services have been redeployed by SRE Service Operations. So, the cluster is fully operational again, feel free to deploy. Traffic hasn't been switched yet back as we are still making sure that it's also fully traffic capable as well, but it's expected to happen at the latest tomorrow.
On Tue, Mar 23, 2021 at 10:23 AM Alexandros Kosiaris
akosiaris@wikimedia.org wrote:
Hello everyone,
This is starting now. Keep in mind that if you try to deploy to eqiad k8s today, it WILL fail or just won't do what you expect it to do.
On Fri, Mar 19, 2021 at 10:02 PM Alexandros Kosiaris
akosiaris@wikimedia.org wrote:
Hello everyone,
TL;DR if you are not deploying services to the eqiad kubernetes cluster, you can safely skip this.
Long version:
After having tested thrice our cluster reinitialization procedure, next week, on Tuesday 2021-03-23 we will be reinitializing our eqiad kubernetes cluster. All traffic will be drained from it beforehand and we expect no user visible impact. However, for the duration of the process, the kubernetes eqiad cluster will be unavailable to deployers and thus efforts to deploy to it will fail or worse, not have the expected outcomes. This is normal until SRE serviceops announces that the cluster is fully operational again.
SRE service-ops will be deploying all services before marking the cluster as usable and pooling traffic back to it, so there will be no need for deployers to re-deploy their services.
For your convenience the list of services that are currently deployed on that cluster is: apertium api-gateway blubberoid changeprop changeprop-jobqueue citoid cxserver echostore eventgate-analytics eventgate-analytics-external eventgate-logging-external eventgate-main eventstreams eventstreams-internal linkrecommendation mathoid mobileapps proton push-notifications recommendation-api sessionstore similar-users termbox wikifeeds zotero
Regards,
-- Alexandros Kosiaris Principal Site Reliability Engineer Wikimedia Foundation
-- Alexandros Kosiaris Principal Site Reliability Engineer Wikimedia Foundation
wikitech-l@lists.wikimedia.org