Hello!
Next Wednesday 2019-10-16 at 09:00 UTC we will be doing another maintenance operation on some of our cloudvirts servers (the hypervisor servers) that involves rebooting both the physical servers and the virtual machines running on them. The reasons is that we ned to update the running linux kernel version they have.
In this window we will reboot 4 hypervisors: * cloudvirt1028 * cloudvirt1029 * cloudvirt1030
The procedure will be to reboot a server, wait for it to come back online (could take up to 5 minutes) and wait for all the VMs to come back online. Then move to the next server.
Toolforge users may see their tools and webservices briefly disrupted due to several components of the Toolforge infrastructure being rebooted in this operation.
If nothing changes (reallocated or new virtual machine, etc) this is the list of affected VM instances in each hypervisor:
* cloudvirt1028:
VM: ores-worker-05 PROJECT: ores VM: phamhi-puppetclient PROJECT: testlabs VM: integration-agent-jessie-docker-1001 PROJECT: integration VM: language-cx2 PROJECT: language VM: apertium PROJECT: language VM: janitor01 PROJECT: puppet VM: integration-agent-docker-1008 PROJECT: integration VM: integration-agent-docker-1005 PROJECT: integration VM: xtools-prod06 PROJECT: xtools VM: eventmetrics-prod02 PROJECT: eventmetrics VM: cloud-cumin-02 PROJECT: cloudinfra VM: appslabs PROJECT: mobile VM: meza-2 PROJECT: meza VM: meza-1 PROJECT: meza VM: labs-t224000-alex-osdev PROJECT: openstack VM: clouddb-wikilabels-01 PROJECT: clouddb-services VM: stretch PROJECT: thumbor VM: tools-puppetmaster-01 PROJECT: tools VM: commtech-nsfw PROJECT: commtech VM: builder-envoy PROJECT: packaging VM: cloudstore-dev-01 PROJECT: cloudstore VM: deployment-db05 PROJECT: deployment-prep VM: cloud-puppetmaster-02 PROJECT: cloudinfra VM: osmit-umap PROJECT: osmit VM: util-abogott-stretch PROJECT: testlabs VM: wdhqs-1 PROJECT: wikidata-history-query-service VM: tools-docker-registry-04 PROJECT: tools VM: adhoc-utils01 PROJECT: security-tools VM: tools-proxy-04 PROJECT: tools VM: tools-docker-builder-06 PROJECT: tools VM: tools-sgeexec-0921 PROJECT: tools VM: proxy-02 PROJECT: project-proxy VM: canary1028-01 PROJECT: testlabs VM: vconverter-instance PROJECT: videowiki VM: snuggle-enwiki-01 PROJECT: snuggle VM: saucelabs-02 PROJECT: integration VM: clm-test-01 PROJECT: community-labs-monitoring VM: deployment-memc05 PROJECT: deployment-prep VM: deployment-sca01 PROJECT: deployment-prep VM: af-puppetdb01 PROJECT: automation-framework VM: api PROJECT: openocr VM: a11y PROJECT: reading-web-staging VM: cyberbot-exec-iabot-01 PROJECT: cyberbot VM: fridolin PROJECT: catgraph
* cloudvirt1029:
VM: clouddb-wikilabels-02 PROJECT: clouddb-services VM: buster PROJECT: thumbor VM: gerrit-test5 PROJECT: git VM: toolsbeta-paws-worker-1001 PROJECT: toolsbeta VM: toolsbeta-flannel-etcd-01 PROJECT: toolsbeta VM: toolsbeta-services-01 PROJECT: toolsbeta VM: tools-worker-1021 PROJECT: tools VM: tools-worker-1015 PROJECT: tools VM: traffic-rpki PROJECT: traffic VM: paws-worker-03 PROJECT: paws VM: paws-worker-02 PROJECT: paws VM: cloud-bootstrapvz-buster-bootstrap PROJECT: openstack VM: structurednavigation PROJECT: structurednavigation VM: tools-worker-1027 PROJECT: tools VM: tools-sgebastion-07 PROJECT: tools VM: wmde-dashboards PROJECT: wmde-dashboards VM: wikidata-shex PROJECT: wikidata-dev VM: tools-proxy-03 PROJECT: tools VM: tools-sge-services-04 PROJECT: tools VM: abogott-proxy-canary PROJECT: testlabs VM: tools-sgewebgrid-lighttpd-0919 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0918 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0916 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0914 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0913 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0912 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0911 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0905 PROJECT: tools VM: tools-sgeexec-0934 PROJECT: tools VM: tools-sgeexec-0932 PROJECT: tools VM: tools-sgeexec-0931 PROJECT: tools VM: tools-sgeexec-0930 PROJECT: tools VM: tools-sgeexec-0928 PROJECT: tools VM: tools-sgeexec-0913 PROJECT: tools VM: tools-sgeexec-0912 PROJECT: tools VM: tools-sgeexec-0907 PROJECT: tools VM: tools-paws-worker-1007 PROJECT: tools VM: canary1029-01 PROJECT: testlabs VM: maps-puppetmaster PROJECT: maps VM: deployment-imagescaler03 PROJECT: deployment-prep VM: integration-slave-jessie-1004 PROJECT: integration
* cloudvirt1030:
VM: taxonbota-b PROJECT: dwl VM: irc-buster PROJECT: dwl VM: janus1-1 PROJECT: analytics VM: integration-agent-docker-1012 PROJECT: integration VM: integration-agent-docker-1011 PROJECT: integration VM: tools-sgeexec-0929 PROJECT: tools VM: canary1030-01 PROJECT: testlabs VM: t166878 PROJECT: otrs VM: phragile-pro PROJECT: phragile VM: signwriting-swserver PROJECT: signwriting VM: signwriting-swis PROJECT: signwriting VM: commonsarchive-prod PROJECT: commonsarchive VM: math-docker PROJECT: math VM: ldfclient-new PROJECT: wikidata-query VM: wikidata-misc PROJECT: wikidata-dev VM: packaging PROJECT: thumbor VM: neon PROJECT: rcm VM: oxygen PROJECT: rcm VM: hafnium PROJECT: rcm VM: hound-app-01 PROJECT: hound VM: mediawiki2latex PROJECT: collection-alt-renderer VM: deployment-sca02 PROJECT: deployment-prep VM: deployment-memc04 PROJECT: deployment-prep VM: deployment-fluorine02 PROJECT: deployment-prep VM: deployment-mcs01 PROJECT: deployment-prep VM: deployment-parsoid09 PROJECT: deployment-prep VM: deployment-sca04 PROJECT: deployment-prep VM: deployment-kafka-jumbo-2 PROJECT: deployment-prep VM: deployment-kafka-main-1 PROJECT: deployment-prep VM: deployment-mediawiki-09 PROJECT: deployment-prep VM: deployment-webperf12 PROJECT: deployment-prep VM: deployment-deploy02 PROJECT: deployment-prep VM: deployment-deploy01 PROJECT: deployment-prep VM: deployment-maps04 PROJECT: deployment-prep VM: twlight-tracker PROJECT: twl VM: encoding02 PROJECT: video VM: encoding03 PROJECT: video VM: wikispeech-tts-dev PROJECT: wikispeech VM: pub2 PROJECT: wikiapiary VM: integration-slave-jessie-1001 PROJECT: integration VM: ores-staging-01 PROJECT: ores-staging VM: ve-font PROJECT: design VM: visualeditor-test2 PROJECT: visualeditor VM: ores-redis-02 PROJECT: ores VM: quarry-worker-01 PROJECT: quarry VM: fastcci-new-master PROJECT: fastcci VM: cvn-app8 PROJECT: cvn
regards.
On 10/9/19 1:45 PM, Arturo Borrero Gonzalez wrote:
Hello!
Next Wednesday 2019-10-16 at 09:00 UTC we will be doing another maintenance operation on some of our cloudvirts servers (the hypervisor servers) that involves rebooting both the physical servers and the virtual machines running on them. The reasons is that we ned to update the running linux kernel version they have.
In this window we will reboot 4 hypervisors:
- cloudvirt1028
- cloudvirt1029
- cloudvirt1030
The procedure will be to reboot a server, wait for it to come back online (could take up to 5 minutes) and wait for all the VMs to come back online. Then move to the next server.
Toolforge users may see their tools and webservices briefly disrupted due to several components of the Toolforge infrastructure being rebooted in this operation.
Remainder, this is happening today in about 10 minutes!
regards
Hi, you said 4 servers but also you said cloudvirt1028, cloudvirt1029 and cloudvirt1030. Where is fourth?
Best regards, Zoran.
On 10/16/19 12:42 PM, Zoran Dori wrote:
Hi, you said 4 servers but also you said cloudvirt1028, cloudvirt1029 and cloudvirt1030. Where is fourth?
That's a typo. Sorry for that. We are rebooting *3* cloudvirts.
Good catch! :-P
regards.
No problem, thanks!
sre, 16. okt 2019. 14:39 Arturo Borrero Gonzalez aborrero@wikimedia.org je napisao/la:
On 10/16/19 12:42 PM, Zoran Dori wrote:
Hi, you said 4 servers but also you said cloudvirt1028, cloudvirt1029 and cloudvirt1030. Where is fourth?
That's a typo. Sorry for that. We are rebooting *3* cloudvirts.
Good catch! :-P
regards.
-- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation