July 2019 - Cloud-announce - lists.wikimedia.org

Instance downtime on August 5th and 6th
by Andrew Bogott 05 Aug '19

05 Aug '19

As part of routine networking and OS upgrades, I'll be emptying two more hypervisors (cloudvirt1021 and cloudvirt1022) on Monday and Tuesday next week, the 5th and 6th. This will result in downtime for many VMs as they are copied and restarted. A complete list of affected instances follows. Please let me know if you need to schedule a more specific window for your downtime. Better yet, if any of the listed VMs are defunct and can simply be deleted, please do that now and save me some time! -Andrew + the WMCS team Affected instances (shown as <project>: <instance name>): account-creation-assistance: accounts-db3 analytics: deployment-server analytics: zk1-1 analytics: zk1-3 antiharassment: antiharassment-web1 automation-framework: af-debmonitor automation-framework: af-puppetmaster02 bastion: bastion-eqiad1-01 bastion: bastion-restricted-eqiad1-01 catgraph: sylvester cloudinfra: mx-out01 cloudinfra: mx-out02 cyberbot: cyberbot-exec-01 design: wikimedia-ui discourse: discourse1002 download: download-01 dumps: dumps-1 dumps: dumps-2 dwl: taxonbot etcd: etcd04 etcd: etcd05 etcd: etcd06 extdist: extdist-01 fastcci: fastcci-worker1 fastcci: fastcci-worker2 git: gerrit-mysql git: gerrit-test git: puppet-paladox google-api-proxy: google-api-proxy-02 grantreview: grantreview-03 hashtags: hashtags-prod hat-imagescalers: k8s-test-builder hhvm: hhvm-jmm-vp9 hhvm: hhvm-stretch-jmm huggle: huggle-wl huggle: xmlrcs iiab: medbox3-iiab integration: integration-slave-docker-1043 k8splay: qube-master language: language-apertium2 language: language-cx language: language-eg language: language-mleb-master language: language-mleb-stable language: moses01-small logging: filippo-log-jessie01 logging: filippo-log-stretch01 mcr-dev: mcr-full mediawiki-vagrant: mwv-builder-02 mediawiki-vagrant: mwv-stretch-migration mediawiki-vagrant: T183456-jessie mediawiki-vagrant: T183456-stretch mix-n-match: mixnmatch mobile: apps-team-tools mobile: emoji mwoffliner: mwoffliner2 mwoffliner: mwoffliner4 mwstake: mwstake mwv-apt: mwv-apt-01 newsletter: newsletter-test openstack: labs-bootstrapvz-jessie openstack: vmbuilder-trusty ores: ores-web-01 ores: ores-web-03 orig: gnd-01 packaging: builder01 packaging: calico-builder01 petscan: petscan3 phabricator: phabricator phabricator: phab-tin phabricator: puppet-phabricator pluggableauth: pluggableauth-server puppet: puppet-jmm-pmaster puppet: puppet-jmm-pmaster-client qna: meza-new4 quarry: quarry-db-01 quarry: quarry-web-01 quarry: quarry-worker-02 reading-web-staging: experiments reading-web-staging: marvin-staging reading-web-staging: readingwebstaging recommendation-api: gapfinder-tools recommendation-api: tool-alpha search: cirrus-browser-bot search: relforge-search search: wdsearch2 snuggle: snuggle-wikidatawiki-01 social-tools: timeless2 striker: striker-uwsgi03 test-twemproxy: mc-clusterB-1 test-twemproxy: mc-clusterB-2 toolsbeta: toolsbeta-sgebastion-04 toolsbeta: toolsbeta-sgeexec-0901 toolsbeta: toolsbeta-sgegrid-master toolserver-legacy: relic-stretch traffic: diffscan traffic: traffic-puppetmaster traffic: traffic-text-stretch utrs: utrs-beta-production utrs: utrs-database2 utrs: utrs-production2 wcdo: wcdo wikibase-registry: orig-01 wikibase-registry: wbregistry-01 wikidata-dev: federated-wikis wikidata-dev: federated-wikis2 wikidata-primary-sources-tool: pst wikidiff2-wmde-dev: sqltest02 wikidiff2-wmde-dev: wmde-wikidiff2-patched wikidiff2-wmde-dev: wmde-wikidiff2-unpatched wikilabels: wikilabels-staging-01 wikimania-support: scholarships-02 wikispeech: wikispeech-tts-release wikitextexp: parsing-qa-01 wildcat: dannyb xtools: xtools-dev04 xtools: xtools-prod03 xtools: xtools-prod05

1 3

Brief toolforge grid interruption on Monday, August 5 at 14:00UTC
by Andrew Bogott 31 Jul '19

31 Jul '19

I will be moving the toolforge grid master on Monday. That will mean that for a few minutes it will be impossible to submit new grid jobs. Jobs that are already running will be unaffected. I'll make the move at 14:00UTC, which is about 7AM Pacific time. -Andrew

1 0

Re: [Cloud-announce] [Cloud] Instance downtime on August 5th and 6th
by Andrew Bogott 31 Jul '19

31 Jul '19

On 7/31/19 10:00 AM, Maximilian Doerr wrote: > I’m running a script that will need to run uninterrupted for 2 weeks. > If you can do it now, that would be okay too. Now is good! I'll start in about an hour, will let you know when it finishes. > > Cyberpower678 > English Wikipedia Account Creation Team > English Wikipedia Administrator > Global User Renamer > >> On Jul 31, 2019, at 10:56, Andrew Bogott <abogott(a)wikimedia.org >> <mailto:abogott@wikimedia.org>> wrote: >> >> On 7/31/19 9:46 AM, Maximilian Doerr wrote: >>> Oh please no. cyberbot-db-01 needs to remain up in the next two >>> weeks at least. >> >> Postponing isn't a good option (due to scheduling conflicts that will >> delay the entire maintenance project by as much as a month). I can >> move the VM during a pre-arranged window, though, if there's sometime >> that's better for you. Over the weekend is possible, even. >> >> -A >> >> >>> >>> Cyberpower678 >>> English Wikipedia Account Creation Team >>> English Wikipedia Administrator >>> Global User Renamer >>> >>>> On Jul 31, 2019, at 10:26, Andrew Bogott <abogott(a)wikimedia.org >>>> <mailto:abogott@wikimedia.org>> wrote: >>>> >>>> In the interest of finishing up this stage of upgrades, I'm going >>>> to try to also drain cloudvirt1023 during the same window. That >>>> includes the following additional VMs: >>>> >>>> services: appservice >>>> privpol-captcha: captcha-imageprocessing-11 >>>> privpol-captcha: captcha-tf-43 >>>> pluggableauth: cindy >>>> search: clickmodel >>>> codesearch: codesearch4 >>>> cvn: cvn-apache9 >>>> cvn: cvn-app8 >>>> cvn: cvn-app9 >>>> language: cxserver >>>> cyberbot: cyberbot-db-01 >>>> discourse: discourse-mw >>>> dumps: dumps-3 >>>> wikifactmine: elasticsearch-01 >>>> eventmetrics: eventmetrics-prod01 >>>> fastcci: fastcci-new-master >>>> catgraph: fishbone >>>> huggle: huggle-deb-builder >>>> analytics: k4-1 >>>> analytics: k4-2 >>>> puppet: keith-puppetmaster >>>> language: language-cx1 >>>> wikibase-registry: libcanada-01 >>>> catgraph: logintest2 >>>> matrix: matrix-synapse-01 >>>> test-twemproxy: mc-clusterA-2 >>>> mcr-dev: mcr-base >>>> mcr-dev: mcr-sdc >>>> wikibase-registry: misc-01 >>>> ores: ores-lb-03 >>>> ores-staging: ores-misc-01 >>>> ores: ores-redis-02 >>>> ores-staging: ores-staging-01 >>>> ores: ores-worker-01 >>>> ores: ores-worker-02 >>>> phlogiston: phlogiston-4 >>>> puppet: puppet-ema-2 >>>> puppet: puppet-jmm-kernel-jessie >>>> puppet: puppet-jmm-kernel-stretch2 >>>> thumbor: puppetmaster >>>> quarry: quarry-worker-01 >>>> k8splay: qube-node1 >>>> k8splay: qube-node2 >>>> recommendation-api: recommendation-api-build >>>> search: rel2 >>>> sentry: sentry-builder >>>> suggestbot: suggestbot-01 >>>> toolsbeta: toolsbeta-sgewebgrid-generic-0901 >>>> reading-web-staging: trending >>>> library-upgrader: upgrader-04 >>>> visualeditor: visualeditor-test2 >>>> wikidata-dev: wikibase-stretch >>>> wikibrain: wikibrain-embeddings-01 >>>> wikibrain: wikibrain-embeddings-02 >>>> globaleducation: wikiedubackups >>>> wikilabels: wikilabels-backups >>>> wikispeech: wikispeech-tts-dev >>>> wikispeech: wikispeech-wiki-stretch >>>> wikistats: wikistats-greyhound >>>> wikidiff2-wmde-dev: wmde-wikidiff2-debug >>>> wpx: wpx-mediawiki-02 >>>> analytics: zk1-2 >>>> >>>> >>>> On 7/31/19 7:55 AM, Andrew Bogott wrote: >>>>> As part of routine networking and OS upgrades, I'll be emptying >>>>> two more hypervisors (cloudvirt1021 and cloudvirt1022) on Monday >>>>> and Tuesday next week, the 5th and 6th. This will result in >>>>> downtime for many VMs as they are copied and restarted. A >>>>> complete list of affected instances follows. >>>>> >>>>> Please let me know if you need to schedule a more specific >>>>> window for your downtime. Better yet, if any of the listed VMs >>>>> are defunct and can simply be deleted, please do that now and save >>>>> me some time! >>>>> >>>>> -Andrew + the WMCS team >>>>> >>>>> >>>>> Affected instances (shown as <project>: <instance name>): >>>>> >>>>> >>>>> account-creation-assistance: accounts-db3 >>>>> analytics: deployment-server >>>>> analytics: zk1-1 >>>>> analytics: zk1-3 >>>>> antiharassment: antiharassment-web1 >>>>> automation-framework: af-debmonitor >>>>> automation-framework: af-puppetmaster02 >>>>> bastion: bastion-eqiad1-01 >>>>> bastion: bastion-restricted-eqiad1-01 >>>>> catgraph: sylvester >>>>> cloudinfra: mx-out01 >>>>> cloudinfra: mx-out02 >>>>> cyberbot: cyberbot-exec-01 >>>>> design: wikimedia-ui >>>>> discourse: discourse1002 >>>>> download: download-01 >>>>> dumps: dumps-1 >>>>> dumps: dumps-2 >>>>> dwl: taxonbot >>>>> etcd: etcd04 >>>>> etcd: etcd05 >>>>> etcd: etcd06 >>>>> extdist: extdist-01 >>>>> fastcci: fastcci-worker1 >>>>> fastcci: fastcci-worker2 >>>>> git: gerrit-mysql >>>>> git: gerrit-test >>>>> git: puppet-paladox >>>>> google-api-proxy: google-api-proxy-02 >>>>> grantreview: grantreview-03 >>>>> hashtags: hashtags-prod >>>>> hat-imagescalers: k8s-test-builder >>>>> hhvm: hhvm-jmm-vp9 >>>>> hhvm: hhvm-stretch-jmm >>>>> huggle: huggle-wl >>>>> huggle: xmlrcs >>>>> iiab: medbox3-iiab >>>>> integration: integration-slave-docker-1043 >>>>> k8splay: qube-master >>>>> language: language-apertium2 >>>>> language: language-cx >>>>> language: language-eg >>>>> language: language-mleb-master >>>>> language: language-mleb-stable >>>>> language: moses01-small >>>>> logging: filippo-log-jessie01 >>>>> logging: filippo-log-stretch01 >>>>> mcr-dev: mcr-full >>>>> mediawiki-vagrant: mwv-builder-02 >>>>> mediawiki-vagrant: mwv-stretch-migration >>>>> mediawiki-vagrant: T183456-jessie >>>>> mediawiki-vagrant: T183456-stretch >>>>> mix-n-match: mixnmatch >>>>> mobile: apps-team-tools >>>>> mobile: emoji >>>>> mwoffliner: mwoffliner2 >>>>> mwoffliner: mwoffliner4 >>>>> mwstake: mwstake >>>>> mwv-apt: mwv-apt-01 >>>>> newsletter: newsletter-test >>>>> openstack: labs-bootstrapvz-jessie >>>>> openstack: vmbuilder-trusty >>>>> ores: ores-web-01 >>>>> ores: ores-web-03 >>>>> orig: gnd-01 >>>>> packaging: builder01 >>>>> packaging: calico-builder01 >>>>> petscan: petscan3 >>>>> phabricator: phabricator >>>>> phabricator: phab-tin >>>>> phabricator: puppet-phabricator >>>>> pluggableauth: pluggableauth-server >>>>> puppet: puppet-jmm-pmaster >>>>> puppet: puppet-jmm-pmaster-client >>>>> qna: meza-new4 >>>>> quarry: quarry-db-01 >>>>> quarry: quarry-web-01 >>>>> quarry: quarry-worker-02 >>>>> reading-web-staging: experiments >>>>> reading-web-staging: marvin-staging >>>>> reading-web-staging: readingwebstaging >>>>> recommendation-api: gapfinder-tools >>>>> recommendation-api: tool-alpha >>>>> search: cirrus-browser-bot >>>>> search: relforge-search >>>>> search: wdsearch2 >>>>> snuggle: snuggle-wikidatawiki-01 >>>>> social-tools: timeless2 >>>>> striker: striker-uwsgi03 >>>>> test-twemproxy: mc-clusterB-1 >>>>> test-twemproxy: mc-clusterB-2 >>>>> toolsbeta: toolsbeta-sgebastion-04 >>>>> toolsbeta: toolsbeta-sgeexec-0901 >>>>> toolsbeta: toolsbeta-sgegrid-master >>>>> toolserver-legacy: relic-stretch >>>>> traffic: diffscan >>>>> traffic: traffic-puppetmaster >>>>> traffic: traffic-text-stretch >>>>> utrs: utrs-beta-production >>>>> utrs: utrs-database2 >>>>> utrs: utrs-production2 >>>>> wcdo: wcdo >>>>> wikibase-registry: orig-01 >>>>> wikibase-registry: wbregistry-01 >>>>> wikidata-dev: federated-wikis >>>>> wikidata-dev: federated-wikis2 >>>>> wikidata-primary-sources-tool: pst >>>>> wikidiff2-wmde-dev: sqltest02 >>>>> wikidiff2-wmde-dev: wmde-wikidiff2-patched >>>>> wikidiff2-wmde-dev: wmde-wikidiff2-unpatched >>>>> wikilabels: wikilabels-staging-01 >>>>> wikimania-support: scholarships-02 >>>>> wikispeech: wikispeech-tts-release >>>>> wikitextexp: parsing-qa-01 >>>>> wildcat: dannyb >>>>> xtools: xtools-dev04 >>>>> xtools: xtools-prod03 >>>>> xtools: xtools-prod05 >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Wikimedia Cloud Services announce mailing list >>>> Cloud-announce(a)lists.wikimedia.org >>>> <mailto:Cloud-announce@lists.wikimedia.org> (formerly >>>> labs-announce(a)lists.wikimedia.org >>>> <mailto:labs-announce@lists.wikimedia.org>) >>>> https://lists.wikimedia.org/mailman/listinfo/cloud-announce >>>> _______________________________________________ >>>> Wikimedia Cloud Services mailing list >>>> Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) >>>> https://lists.wikimedia.org/mailman/listinfo/cloud >>> >>> >>> _______________________________________________ >>> Wikimedia Cloud Services mailing list >>> Cloud(a)lists.wikimedia.org (formerlylabs-l(a)lists.wikimedia.org) >>> https://lists.wikimedia.org/mailman/listinfo/cloud >> >> >

1 0

Re: [Cloud-announce] [Cloud] Instance downtime on August 5th and 6th
by Andrew Bogott 31 Jul '19

31 Jul '19

On 7/31/19 9:46 AM, Maximilian Doerr wrote: > Oh please no. cyberbot-db-01 needs to remain up in the next two weeks > at least. Postponing isn't a good option (due to scheduling conflicts that will delay the entire maintenance project by as much as a month). I can move the VM during a pre-arranged window, though, if there's sometime that's better for you. Over the weekend is possible, even. -A > > Cyberpower678 > English Wikipedia Account Creation Team > English Wikipedia Administrator > Global User Renamer > >> On Jul 31, 2019, at 10:26, Andrew Bogott <abogott(a)wikimedia.org >> <mailto:abogott@wikimedia.org>> wrote: >> >> In the interest of finishing up this stage of upgrades, I'm going to >> try to also drain cloudvirt1023 during the same window. That >> includes the following additional VMs: >> >> services: appservice >> privpol-captcha: captcha-imageprocessing-11 >> privpol-captcha: captcha-tf-43 >> pluggableauth: cindy >> search: clickmodel >> codesearch: codesearch4 >> cvn: cvn-apache9 >> cvn: cvn-app8 >> cvn: cvn-app9 >> language: cxserver >> cyberbot: cyberbot-db-01 >> discourse: discourse-mw >> dumps: dumps-3 >> wikifactmine: elasticsearch-01 >> eventmetrics: eventmetrics-prod01 >> fastcci: fastcci-new-master >> catgraph: fishbone >> huggle: huggle-deb-builder >> analytics: k4-1 >> analytics: k4-2 >> puppet: keith-puppetmaster >> language: language-cx1 >> wikibase-registry: libcanada-01 >> catgraph: logintest2 >> matrix: matrix-synapse-01 >> test-twemproxy: mc-clusterA-2 >> mcr-dev: mcr-base >> mcr-dev: mcr-sdc >> wikibase-registry: misc-01 >> ores: ores-lb-03 >> ores-staging: ores-misc-01 >> ores: ores-redis-02 >> ores-staging: ores-staging-01 >> ores: ores-worker-01 >> ores: ores-worker-02 >> phlogiston: phlogiston-4 >> puppet: puppet-ema-2 >> puppet: puppet-jmm-kernel-jessie >> puppet: puppet-jmm-kernel-stretch2 >> thumbor: puppetmaster >> quarry: quarry-worker-01 >> k8splay: qube-node1 >> k8splay: qube-node2 >> recommendation-api: recommendation-api-build >> search: rel2 >> sentry: sentry-builder >> suggestbot: suggestbot-01 >> toolsbeta: toolsbeta-sgewebgrid-generic-0901 >> reading-web-staging: trending >> library-upgrader: upgrader-04 >> visualeditor: visualeditor-test2 >> wikidata-dev: wikibase-stretch >> wikibrain: wikibrain-embeddings-01 >> wikibrain: wikibrain-embeddings-02 >> globaleducation: wikiedubackups >> wikilabels: wikilabels-backups >> wikispeech: wikispeech-tts-dev >> wikispeech: wikispeech-wiki-stretch >> wikistats: wikistats-greyhound >> wikidiff2-wmde-dev: wmde-wikidiff2-debug >> wpx: wpx-mediawiki-02 >> analytics: zk1-2 >> >> >> On 7/31/19 7:55 AM, Andrew Bogott wrote: >>> As part of routine networking and OS upgrades, I'll be emptying two >>> more hypervisors (cloudvirt1021 and cloudvirt1022) on Monday and >>> Tuesday next week, the 5th and 6th. This will result in downtime >>> for many VMs as they are copied and restarted. A complete list of >>> affected instances follows. >>> >>> Please let me know if you need to schedule a more specific >>> window for your downtime. Better yet, if any of the listed VMs are >>> defunct and can simply be deleted, please do that now and save me >>> some time! >>> >>> -Andrew + the WMCS team >>> >>> >>> Affected instances (shown as <project>: <instance name>): >>> >>> >>> account-creation-assistance: accounts-db3 >>> analytics: deployment-server >>> analytics: zk1-1 >>> analytics: zk1-3 >>> antiharassment: antiharassment-web1 >>> automation-framework: af-debmonitor >>> automation-framework: af-puppetmaster02 >>> bastion: bastion-eqiad1-01 >>> bastion: bastion-restricted-eqiad1-01 >>> catgraph: sylvester >>> cloudinfra: mx-out01 >>> cloudinfra: mx-out02 >>> cyberbot: cyberbot-exec-01 >>> design: wikimedia-ui >>> discourse: discourse1002 >>> download: download-01 >>> dumps: dumps-1 >>> dumps: dumps-2 >>> dwl: taxonbot >>> etcd: etcd04 >>> etcd: etcd05 >>> etcd: etcd06 >>> extdist: extdist-01 >>> fastcci: fastcci-worker1 >>> fastcci: fastcci-worker2 >>> git: gerrit-mysql >>> git: gerrit-test >>> git: puppet-paladox >>> google-api-proxy: google-api-proxy-02 >>> grantreview: grantreview-03 >>> hashtags: hashtags-prod >>> hat-imagescalers: k8s-test-builder >>> hhvm: hhvm-jmm-vp9 >>> hhvm: hhvm-stretch-jmm >>> huggle: huggle-wl >>> huggle: xmlrcs >>> iiab: medbox3-iiab >>> integration: integration-slave-docker-1043 >>> k8splay: qube-master >>> language: language-apertium2 >>> language: language-cx >>> language: language-eg >>> language: language-mleb-master >>> language: language-mleb-stable >>> language: moses01-small >>> logging: filippo-log-jessie01 >>> logging: filippo-log-stretch01 >>> mcr-dev: mcr-full >>> mediawiki-vagrant: mwv-builder-02 >>> mediawiki-vagrant: mwv-stretch-migration >>> mediawiki-vagrant: T183456-jessie >>> mediawiki-vagrant: T183456-stretch >>> mix-n-match: mixnmatch >>> mobile: apps-team-tools >>> mobile: emoji >>> mwoffliner: mwoffliner2 >>> mwoffliner: mwoffliner4 >>> mwstake: mwstake >>> mwv-apt: mwv-apt-01 >>> newsletter: newsletter-test >>> openstack: labs-bootstrapvz-jessie >>> openstack: vmbuilder-trusty >>> ores: ores-web-01 >>> ores: ores-web-03 >>> orig: gnd-01 >>> packaging: builder01 >>> packaging: calico-builder01 >>> petscan: petscan3 >>> phabricator: phabricator >>> phabricator: phab-tin >>> phabricator: puppet-phabricator >>> pluggableauth: pluggableauth-server >>> puppet: puppet-jmm-pmaster >>> puppet: puppet-jmm-pmaster-client >>> qna: meza-new4 >>> quarry: quarry-db-01 >>> quarry: quarry-web-01 >>> quarry: quarry-worker-02 >>> reading-web-staging: experiments >>> reading-web-staging: marvin-staging >>> reading-web-staging: readingwebstaging >>> recommendation-api: gapfinder-tools >>> recommendation-api: tool-alpha >>> search: cirrus-browser-bot >>> search: relforge-search >>> search: wdsearch2 >>> snuggle: snuggle-wikidatawiki-01 >>> social-tools: timeless2 >>> striker: striker-uwsgi03 >>> test-twemproxy: mc-clusterB-1 >>> test-twemproxy: mc-clusterB-2 >>> toolsbeta: toolsbeta-sgebastion-04 >>> toolsbeta: toolsbeta-sgeexec-0901 >>> toolsbeta: toolsbeta-sgegrid-master >>> toolserver-legacy: relic-stretch >>> traffic: diffscan >>> traffic: traffic-puppetmaster >>> traffic: traffic-text-stretch >>> utrs: utrs-beta-production >>> utrs: utrs-database2 >>> utrs: utrs-production2 >>> wcdo: wcdo >>> wikibase-registry: orig-01 >>> wikibase-registry: wbregistry-01 >>> wikidata-dev: federated-wikis >>> wikidata-dev: federated-wikis2 >>> wikidata-primary-sources-tool: pst >>> wikidiff2-wmde-dev: sqltest02 >>> wikidiff2-wmde-dev: wmde-wikidiff2-patched >>> wikidiff2-wmde-dev: wmde-wikidiff2-unpatched >>> wikilabels: wikilabels-staging-01 >>> wikimania-support: scholarships-02 >>> wikispeech: wikispeech-tts-release >>> wikitextexp: parsing-qa-01 >>> wildcat: dannyb >>> xtools: xtools-dev04 >>> xtools: xtools-prod03 >>> xtools: xtools-prod05 >>> >> >> >> _______________________________________________ >> Wikimedia Cloud Services announce mailing list >> Cloud-announce(a)lists.wikimedia.org >> <mailto:Cloud-announce@lists.wikimedia.org> (formerly >> labs-announce(a)lists.wikimedia.org >> <mailto:labs-announce@lists.wikimedia.org>) >> https://lists.wikimedia.org/mailman/listinfo/cloud-announce >> _______________________________________________ >> Wikimedia Cloud Services mailing list >> Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) >> https://lists.wikimedia.org/mailman/listinfo/cloud > > > _______________________________________________ > Wikimedia Cloud Services mailing list > Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) > https://lists.wikimedia.org/mailman/listinfo/cloud

1 0

CloudVPS reboots for security updates
by Arturo Borrero Gonzalez 29 Jul '19

29 Jul '19

Hi there! Unrelated to other operations that were communicated recently (datacenter PDU upgrades, operating system upgrades, etc) we need to reboot all the cloudvirt servers to introduce some security updates for CPU vulnerabilities. Along with the physical hardware reboot we also need to reboot all the virtual machines running in CloudVPS. This operation is a bit disruptive but very quick and should not lead to any unexpected errors (is just a reboot). We already tried the same upgrades in some other servers. We will be doing the reboots during this week (starting 2019-07-29). If you see any problems related to this, please contact us. regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

20190726 - afl_log_id field to be removed from wiki replica views
by Brooke Storm 26 Jul '19

26 Jul '19

In the abuse_filter_log table, the afl_log_id field appears to never have been used and is being dropped upstream. To prepare for that and keep the replica view up and running, we are going to start removing the field from the abuse_filter_log views on the wiki replicas service, starting on Monday, July 26th. This should have little to no impact on anyone since the field seems to always be NULL everywhere and to never have been fully implemented, and it would only cause impact to an application if it was specifically queried. For more information on the dropping of the column: https://phabricator.wikimedia.org/T226851 <https://phabricator.wikimedia.org/T226851> Some of the reasons why and a bit more context: https://phabricator.wikimedia.org/T214592 <https://phabricator.wikimedia.org/T214592> Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

1 0

PDU upgrades in the eqiad datacenter affects CloudVPS hypervisor
by Arturo Borrero Gonzalez 25 Jul '19

25 Jul '19

Hi there! There is an ongoing maintenance in the eqiad datacenter that involves changing power connectors of the servers. More info in this phabricator task: T226778 [0]. The PDU upgrade could potentially leave our hypervisors without power briefly. For some hypervisors, we plan to take the risks of leaving them running. For some other hypervisors (those running important DBs in the form of virtual machines) we will probably do a controlled shutdown before the operations to ensure no data corruption happen in the databases. The PDU upgrades will happen this very week (see phab task [0]) and it could potentially affect every virtual machine we run in CloudVPS. This includes Toolforge. In the case of power loss, we expect the disruptions to be very briefly and to don't cause extended downtime in any case. Please, let us know any issue you may find related to this operation. regards. [0] https://phabricator.wikimedia.org/T226778 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

NFS Maintenance 2019-07-25 1500 UTC
by Jason Hedden 25 Jul '19

25 Jul '19

On Thursday, July 25th, 2019 between the hours of 1500 and 1700 UTC we will be performing system maintenance on the NFS servers that support Toolforge and the CloudVPS instances that are using NFS for home, project or scratch data. During this maintenance window, we will be applying rolling updates that require NFS service restarts. We are taking precautions to minimize impact, but there may be short periods of NFS service interruption or performance degradation. --- Jason Hedden Site Reliability Engineer - Wikimedia Cloud Services Wikimedia Foundation

1 2

Instance downtime on August 22nd and 23rd
by Andrew Bogott 22 Jul '19

22 Jul '19

As part of routine networking and OS upgrades, I'll be emptying two hypervisors (cloudvirt1016 and cloudvirt1017) on Monday and Tuesday, the 22nd and 23rd. This will result in downtime for many VMs as they are copied and restarted. A complete list of affected instances follows. I'll begin by moving the deployment-prep project at around 13:00 UTC on Monday. After that copies will proceed in roughly the order you see below, but the timing will be hard to predict. Please let me know if you need to schedule a more specific window for your downtime. Better yet, if any of the listed VMs are defunct and can simply be deleted, please do that now and save me some time! -Andrew + the WMCS team Affected instances (shown as <project>: <instance name>): account-creation-assistance: accounts-appserver4 account-creation-assistance: accounts-mwoauth automation-framework: af-puppetdb02 butterfly: butterfly-m4m2 cloudinfra: cloudinfra-db02 codereview: Krypton codereview: Radon commtech: commtech-2 community-labs-monitoring: clm-web-01 community-labs-monitoring: clm-worker-01 dashiki: dashiki-01 dashiki: dashiki-staging-01 deployment-prep: deployment-cache-text05 deployment-prep: deployment-changeprop deployment-prep: deployment-chromium01 deployment-prep: deployment-chromium02 deployment-prep: deployment-cpjobqueue deployment-prep: deployment-dumps-puppetmaster02 deployment-prep: deployment-elastic06 deployment-prep: deployment-elastic07 deployment-prep: deployment-etcd-01 deployment-prep: deployment-eventlog05 deployment-prep: deployment-imagescaler01 deployment-prep: deployment-imagescaler02 deployment-prep: deployment-ircd deployment-prep: deployment-jobrunner03 deployment-prep: deployment-kafka-jumbo-1 deployment-prep: deployment-logstash2 deployment-prep: deployment-mediawiki-07 deployment-prep: deployment-memc06 deployment-prep: deployment-memc07 deployment-prep: deployment-mwmaint01 deployment-prep: deployment-ores01 deployment-prep: deployment-puppetdb02 deployment-prep: deployment-puppetmaster03 deployment-prep: deployment-restbase01 deployment-prep: deployment-restbase02 deployment-prep: deployment-sentry01 deployment-prep: deployment-snapshot01 deployment-prep: deployment-urldownloader02 deployment-prep: deployment-zookeeper02 design: design-research-methods dumps: dumps-0 dwl: dwl dwl: taxonbota fa-wp: tofawiki02 getstarted: gitservices getstarted: webservices glampipe: Glampipe hound: hound-puppet-02 integration: integration-cumin integration: integration-r-lang-01 integration: integration-slave-docker-1040 integration: integration-slave-docker-1041 integration: integration-slave-jessie-1002 k8splay: k8s-dzahn lizenzhinweisgenerator: lizenzhinweisgenerator maps: maps-tiles1 maps: maps-warper3 openrefine: openrefine01 openstack: cloud-bootstrapvz-stretch otrs: otrs-oneclickspam-test packagist-mirror: packagist-mirror1 partnermetrics: partnermetrics-redis-01 puppet-diffs: compiler1001 qna: meza-new2 quotatest: novaadminmadethis6 reading-web-staging: readers-web-master recommendation-api: missing-sections recommendation-api: rec-wiki recommendation-api: related-articles recommendation-api: tool security-tools: logparse01 sentry: frama-test5 sentry: frama-test6-sb services: kask services: kask-client shinken: shinken-02 shiny-r: discovery-production-02 testlabs: abogott-puppetmaster testlabs: canary1016-01 tools: tools-sgecron-01 tools: tools-sgegrid-shadow toolsbeta: toolsbeta-sgecron-01 toolsbeta: toolsbeta-sgegrid-shadow toolsbeta: toolsbeta-sgewebgrid-lighttpd-0901 twl: wmil video: encoding01 video: gfg01 video: video-redis video: videodev videowiki: app-instance visualeditor: dumpgrepper webperf: disposable wikidata-dev: wikidata-constraints wikidata-federation: federated-commons wikidata-federation: federated-wikidata wikidiff2-wmde-dev: wmde-wikidiff2-jacnth wikidocumentaries: hupu wikidocumentaries: roope wikidumpparse: whgi wikifactmine: elasticsearch-20 wikifactmine: elasticsearch-21 wikifactmine: puppetmaster-01 wikilabels: wikilabels-02 wikilabels: wikilabels-experiment wikimetrics: wikimetrics-01 wikistream: ws-web wikitextexp: wikitextexp-base-1002 wikitextexp: wikitextexp-expt-1002 wm-bot: wm-bot-pg wm-bot: wm-bot2 wmf-research-tools: diegoTest wmf-research-tools: wikilabels wpx: wpx-redirects-01

1 3

Brief toolforge cron outage on Friday, 2019-07-19 at 14:00 UTC
by Andrew Bogott 19 Jul '19

19 Jul '19

On Friday I'll be moving the toolforge cron server to new hardware. During the move, any uses of the 'crontab' command will fail gracelessly. Any cron jobs scheduled to launch during the downtime will be skipped. The move should take 5-10 minutes but may take as long as 30 if there are complications. -Andrew

1 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce July 2019