- Cloud-announce - lists.wikimedia.org

Docker images used by Toolforge's Kubernetes cluster updated
by Bryan Davis 30 Aug '19

30 Aug '19

Today I rebuilt the Docker images that are used by the `webservice --backend=kubernetes` command. This is actually a normal thing that we do periodically in Toolforge to ensure that security patches are applied in the containers. This round of updates was a bit different however in that it is the first time the Debian Jessie based images have been rebuilt since the upstream Debian project removed the 'jessie-backports' apt repo. Everything should be fine, but if you see weirdness when restarting a webservice or other Kubernetes pod that looks like it could be related to software in the Docker image please let myself or one of the Toolforge admins know by either filing a Phabricator bug report or for faster response joining the #wikimedia-cloud IRC channel on Freenode and sending a "!help ...." message to the channel explaining your issue. Bryan - on behalf of the Toolforge admins -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

Cloud VPS database maintenance next Tuesday 8/3 13:00 UTC
by Jason Hedden 28 Aug '19

28 Aug '19

Next Tuesday on September 3rd, between 13:00 and 14:00 UTC we'll be performing backend database maintenance on the OpenStack VPS control plane. During this maintenance window the Horizon web dashboard will be unavailable and all VPS requests to create, modify or delete VPS resources like virtual machines and DNS entries will be unavailable. Existing VPS virtual machines will remain running and Toolforge users will not be affected by this maintenance. --- Wikimedia Cloud Services Team

1 0

Instance downtime on August 5th and 6th
by Andrew Bogott 06 Aug '19

06 Aug '19

As part of routine networking and OS upgrades, I'll be emptying two more hypervisors (cloudvirt1021 and cloudvirt1022) on Monday and Tuesday next week, the 5th and 6th. This will result in downtime for many VMs as they are copied and restarted. A complete list of affected instances follows. Please let me know if you need to schedule a more specific window for your downtime. Better yet, if any of the listed VMs are defunct and can simply be deleted, please do that now and save me some time! -Andrew + the WMCS team Affected instances (shown as <project>: <instance name>): account-creation-assistance: accounts-db3 analytics: deployment-server analytics: zk1-1 analytics: zk1-3 antiharassment: antiharassment-web1 automation-framework: af-debmonitor automation-framework: af-puppetmaster02 bastion: bastion-eqiad1-01 bastion: bastion-restricted-eqiad1-01 catgraph: sylvester cloudinfra: mx-out01 cloudinfra: mx-out02 cyberbot: cyberbot-exec-01 design: wikimedia-ui discourse: discourse1002 download: download-01 dumps: dumps-1 dumps: dumps-2 dwl: taxonbot etcd: etcd04 etcd: etcd05 etcd: etcd06 extdist: extdist-01 fastcci: fastcci-worker1 fastcci: fastcci-worker2 git: gerrit-mysql git: gerrit-test git: puppet-paladox google-api-proxy: google-api-proxy-02 grantreview: grantreview-03 hashtags: hashtags-prod hat-imagescalers: k8s-test-builder hhvm: hhvm-jmm-vp9 hhvm: hhvm-stretch-jmm huggle: huggle-wl huggle: xmlrcs iiab: medbox3-iiab integration: integration-slave-docker-1043 k8splay: qube-master language: language-apertium2 language: language-cx language: language-eg language: language-mleb-master language: language-mleb-stable language: moses01-small logging: filippo-log-jessie01 logging: filippo-log-stretch01 mcr-dev: mcr-full mediawiki-vagrant: mwv-builder-02 mediawiki-vagrant: mwv-stretch-migration mediawiki-vagrant: T183456-jessie mediawiki-vagrant: T183456-stretch mix-n-match: mixnmatch mobile: apps-team-tools mobile: emoji mwoffliner: mwoffliner2 mwoffliner: mwoffliner4 mwstake: mwstake mwv-apt: mwv-apt-01 newsletter: newsletter-test openstack: labs-bootstrapvz-jessie openstack: vmbuilder-trusty ores: ores-web-01 ores: ores-web-03 orig: gnd-01 packaging: builder01 packaging: calico-builder01 petscan: petscan3 phabricator: phabricator phabricator: phab-tin phabricator: puppet-phabricator pluggableauth: pluggableauth-server puppet: puppet-jmm-pmaster puppet: puppet-jmm-pmaster-client qna: meza-new4 quarry: quarry-db-01 quarry: quarry-web-01 quarry: quarry-worker-02 reading-web-staging: experiments reading-web-staging: marvin-staging reading-web-staging: readingwebstaging recommendation-api: gapfinder-tools recommendation-api: tool-alpha search: cirrus-browser-bot search: relforge-search search: wdsearch2 snuggle: snuggle-wikidatawiki-01 social-tools: timeless2 striker: striker-uwsgi03 test-twemproxy: mc-clusterB-1 test-twemproxy: mc-clusterB-2 toolsbeta: toolsbeta-sgebastion-04 toolsbeta: toolsbeta-sgeexec-0901 toolsbeta: toolsbeta-sgegrid-master toolserver-legacy: relic-stretch traffic: diffscan traffic: traffic-puppetmaster traffic: traffic-text-stretch utrs: utrs-beta-production utrs: utrs-database2 utrs: utrs-production2 wcdo: wcdo wikibase-registry: orig-01 wikibase-registry: wbregistry-01 wikidata-dev: federated-wikis wikidata-dev: federated-wikis2 wikidata-primary-sources-tool: pst wikidiff2-wmde-dev: sqltest02 wikidiff2-wmde-dev: wmde-wikidiff2-patched wikidiff2-wmde-dev: wmde-wikidiff2-unpatched wikilabels: wikilabels-staging-01 wikimania-support: scholarships-02 wikispeech: wikispeech-tts-release wikitextexp: parsing-qa-01 wildcat: dannyb xtools: xtools-dev04 xtools: xtools-prod03 xtools: xtools-prod05

1 3

Brief toolforge grid interruption on Monday, August 5 at 14:00UTC
by Andrew Bogott 01 Aug '19

01 Aug '19

I will be moving the toolforge grid master on Monday. That will mean that for a few minutes it will be impossible to submit new grid jobs. Jobs that are already running will be unaffected. I'll make the move at 14:00UTC, which is about 7AM Pacific time. -Andrew

1 0

Re: [Cloud-announce] [Cloud] Instance downtime on August 5th and 6th
by Andrew Bogott 31 Jul '19

31 Jul '19

On 7/31/19 10:00 AM, Maximilian Doerr wrote: > I’m running a script that will need to run uninterrupted for 2 weeks. > If you can do it now, that would be okay too. Now is good! I'll start in about an hour, will let you know when it finishes. > > Cyberpower678 > English Wikipedia Account Creation Team > English Wikipedia Administrator > Global User Renamer > >> On Jul 31, 2019, at 10:56, Andrew Bogott <abogott(a)wikimedia.org >> <mailto:abogott@wikimedia.org>> wrote: >> >> On 7/31/19 9:46 AM, Maximilian Doerr wrote: >>> Oh please no. cyberbot-db-01 needs to remain up in the next two >>> weeks at least. >> >> Postponing isn't a good option (due to scheduling conflicts that will >> delay the entire maintenance project by as much as a month). I can >> move the VM during a pre-arranged window, though, if there's sometime >> that's better for you. Over the weekend is possible, even. >> >> -A >> >> >>> >>> Cyberpower678 >>> English Wikipedia Account Creation Team >>> English Wikipedia Administrator >>> Global User Renamer >>> >>>> On Jul 31, 2019, at 10:26, Andrew Bogott <abogott(a)wikimedia.org >>>> <mailto:abogott@wikimedia.org>> wrote: >>>> >>>> In the interest of finishing up this stage of upgrades, I'm going >>>> to try to also drain cloudvirt1023 during the same window. That >>>> includes the following additional VMs: >>>> >>>> services: appservice >>>> privpol-captcha: captcha-imageprocessing-11 >>>> privpol-captcha: captcha-tf-43 >>>> pluggableauth: cindy >>>> search: clickmodel >>>> codesearch: codesearch4 >>>> cvn: cvn-apache9 >>>> cvn: cvn-app8 >>>> cvn: cvn-app9 >>>> language: cxserver >>>> cyberbot: cyberbot-db-01 >>>> discourse: discourse-mw >>>> dumps: dumps-3 >>>> wikifactmine: elasticsearch-01 >>>> eventmetrics: eventmetrics-prod01 >>>> fastcci: fastcci-new-master >>>> catgraph: fishbone >>>> huggle: huggle-deb-builder >>>> analytics: k4-1 >>>> analytics: k4-2 >>>> puppet: keith-puppetmaster >>>> language: language-cx1 >>>> wikibase-registry: libcanada-01 >>>> catgraph: logintest2 >>>> matrix: matrix-synapse-01 >>>> test-twemproxy: mc-clusterA-2 >>>> mcr-dev: mcr-base >>>> mcr-dev: mcr-sdc >>>> wikibase-registry: misc-01 >>>> ores: ores-lb-03 >>>> ores-staging: ores-misc-01 >>>> ores: ores-redis-02 >>>> ores-staging: ores-staging-01 >>>> ores: ores-worker-01 >>>> ores: ores-worker-02 >>>> phlogiston: phlogiston-4 >>>> puppet: puppet-ema-2 >>>> puppet: puppet-jmm-kernel-jessie >>>> puppet: puppet-jmm-kernel-stretch2 >>>> thumbor: puppetmaster >>>> quarry: quarry-worker-01 >>>> k8splay: qube-node1 >>>> k8splay: qube-node2 >>>> recommendation-api: recommendation-api-build >>>> search: rel2 >>>> sentry: sentry-builder >>>> suggestbot: suggestbot-01 >>>> toolsbeta: toolsbeta-sgewebgrid-generic-0901 >>>> reading-web-staging: trending >>>> library-upgrader: upgrader-04 >>>> visualeditor: visualeditor-test2 >>>> wikidata-dev: wikibase-stretch >>>> wikibrain: wikibrain-embeddings-01 >>>> wikibrain: wikibrain-embeddings-02 >>>> globaleducation: wikiedubackups >>>> wikilabels: wikilabels-backups >>>> wikispeech: wikispeech-tts-dev >>>> wikispeech: wikispeech-wiki-stretch >>>> wikistats: wikistats-greyhound >>>> wikidiff2-wmde-dev: wmde-wikidiff2-debug >>>> wpx: wpx-mediawiki-02 >>>> analytics: zk1-2 >>>> >>>> >>>> On 7/31/19 7:55 AM, Andrew Bogott wrote: >>>>> As part of routine networking and OS upgrades, I'll be emptying >>>>> two more hypervisors (cloudvirt1021 and cloudvirt1022) on Monday >>>>> and Tuesday next week, the 5th and 6th. This will result in >>>>> downtime for many VMs as they are copied and restarted. A >>>>> complete list of affected instances follows. >>>>> >>>>> Please let me know if you need to schedule a more specific >>>>> window for your downtime. Better yet, if any of the listed VMs >>>>> are defunct and can simply be deleted, please do that now and save >>>>> me some time! >>>>> >>>>> -Andrew + the WMCS team >>>>> >>>>> >>>>> Affected instances (shown as <project>: <instance name>): >>>>> >>>>> >>>>> account-creation-assistance: accounts-db3 >>>>> analytics: deployment-server >>>>> analytics: zk1-1 >>>>> analytics: zk1-3 >>>>> antiharassment: antiharassment-web1 >>>>> automation-framework: af-debmonitor >>>>> automation-framework: af-puppetmaster02 >>>>> bastion: bastion-eqiad1-01 >>>>> bastion: bastion-restricted-eqiad1-01 >>>>> catgraph: sylvester >>>>> cloudinfra: mx-out01 >>>>> cloudinfra: mx-out02 >>>>> cyberbot: cyberbot-exec-01 >>>>> design: wikimedia-ui >>>>> discourse: discourse1002 >>>>> download: download-01 >>>>> dumps: dumps-1 >>>>> dumps: dumps-2 >>>>> dwl: taxonbot >>>>> etcd: etcd04 >>>>> etcd: etcd05 >>>>> etcd: etcd06 >>>>> extdist: extdist-01 >>>>> fastcci: fastcci-worker1 >>>>> fastcci: fastcci-worker2 >>>>> git: gerrit-mysql >>>>> git: gerrit-test >>>>> git: puppet-paladox >>>>> google-api-proxy: google-api-proxy-02 >>>>> grantreview: grantreview-03 >>>>> hashtags: hashtags-prod >>>>> hat-imagescalers: k8s-test-builder >>>>> hhvm: hhvm-jmm-vp9 >>>>> hhvm: hhvm-stretch-jmm >>>>> huggle: huggle-wl >>>>> huggle: xmlrcs >>>>> iiab: medbox3-iiab >>>>> integration: integration-slave-docker-1043 >>>>> k8splay: qube-master >>>>> language: language-apertium2 >>>>> language: language-cx >>>>> language: language-eg >>>>> language: language-mleb-master >>>>> language: language-mleb-stable >>>>> language: moses01-small >>>>> logging: filippo-log-jessie01 >>>>> logging: filippo-log-stretch01 >>>>> mcr-dev: mcr-full >>>>> mediawiki-vagrant: mwv-builder-02 >>>>> mediawiki-vagrant: mwv-stretch-migration >>>>> mediawiki-vagrant: T183456-jessie >>>>> mediawiki-vagrant: T183456-stretch >>>>> mix-n-match: mixnmatch >>>>> mobile: apps-team-tools >>>>> mobile: emoji >>>>> mwoffliner: mwoffliner2 >>>>> mwoffliner: mwoffliner4 >>>>> mwstake: mwstake >>>>> mwv-apt: mwv-apt-01 >>>>> newsletter: newsletter-test >>>>> openstack: labs-bootstrapvz-jessie >>>>> openstack: vmbuilder-trusty >>>>> ores: ores-web-01 >>>>> ores: ores-web-03 >>>>> orig: gnd-01 >>>>> packaging: builder01 >>>>> packaging: calico-builder01 >>>>> petscan: petscan3 >>>>> phabricator: phabricator >>>>> phabricator: phab-tin >>>>> phabricator: puppet-phabricator >>>>> pluggableauth: pluggableauth-server >>>>> puppet: puppet-jmm-pmaster >>>>> puppet: puppet-jmm-pmaster-client >>>>> qna: meza-new4 >>>>> quarry: quarry-db-01 >>>>> quarry: quarry-web-01 >>>>> quarry: quarry-worker-02 >>>>> reading-web-staging: experiments >>>>> reading-web-staging: marvin-staging >>>>> reading-web-staging: readingwebstaging >>>>> recommendation-api: gapfinder-tools >>>>> recommendation-api: tool-alpha >>>>> search: cirrus-browser-bot >>>>> search: relforge-search >>>>> search: wdsearch2 >>>>> snuggle: snuggle-wikidatawiki-01 >>>>> social-tools: timeless2 >>>>> striker: striker-uwsgi03 >>>>> test-twemproxy: mc-clusterB-1 >>>>> test-twemproxy: mc-clusterB-2 >>>>> toolsbeta: toolsbeta-sgebastion-04 >>>>> toolsbeta: toolsbeta-sgeexec-0901 >>>>> toolsbeta: toolsbeta-sgegrid-master >>>>> toolserver-legacy: relic-stretch >>>>> traffic: diffscan >>>>> traffic: traffic-puppetmaster >>>>> traffic: traffic-text-stretch >>>>> utrs: utrs-beta-production >>>>> utrs: utrs-database2 >>>>> utrs: utrs-production2 >>>>> wcdo: wcdo >>>>> wikibase-registry: orig-01 >>>>> wikibase-registry: wbregistry-01 >>>>> wikidata-dev: federated-wikis >>>>> wikidata-dev: federated-wikis2 >>>>> wikidata-primary-sources-tool: pst >>>>> wikidiff2-wmde-dev: sqltest02 >>>>> wikidiff2-wmde-dev: wmde-wikidiff2-patched >>>>> wikidiff2-wmde-dev: wmde-wikidiff2-unpatched >>>>> wikilabels: wikilabels-staging-01 >>>>> wikimania-support: scholarships-02 >>>>> wikispeech: wikispeech-tts-release >>>>> wikitextexp: parsing-qa-01 >>>>> wildcat: dannyb >>>>> xtools: xtools-dev04 >>>>> xtools: xtools-prod03 >>>>> xtools: xtools-prod05 >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Wikimedia Cloud Services announce mailing list >>>> Cloud-announce(a)lists.wikimedia.org >>>> <mailto:Cloud-announce@lists.wikimedia.org> (formerly >>>> labs-announce(a)lists.wikimedia.org >>>> <mailto:labs-announce@lists.wikimedia.org>) >>>> https://lists.wikimedia.org/mailman/listinfo/cloud-announce >>>> _______________________________________________ >>>> Wikimedia Cloud Services mailing list >>>> Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) >>>> https://lists.wikimedia.org/mailman/listinfo/cloud >>> >>> >>> _______________________________________________ >>> Wikimedia Cloud Services mailing list >>> Cloud(a)lists.wikimedia.org (formerlylabs-l(a)lists.wikimedia.org) >>> https://lists.wikimedia.org/mailman/listinfo/cloud >> >> >

1 0

Re: [Cloud-announce] [Cloud] Instance downtime on August 5th and 6th
by Andrew Bogott 31 Jul '19

31 Jul '19

On 7/31/19 9:46 AM, Maximilian Doerr wrote: > Oh please no. cyberbot-db-01 needs to remain up in the next two weeks > at least. Postponing isn't a good option (due to scheduling conflicts that will delay the entire maintenance project by as much as a month). I can move the VM during a pre-arranged window, though, if there's sometime that's better for you. Over the weekend is possible, even. -A > > Cyberpower678 > English Wikipedia Account Creation Team > English Wikipedia Administrator > Global User Renamer > >> On Jul 31, 2019, at 10:26, Andrew Bogott <abogott(a)wikimedia.org >> <mailto:abogott@wikimedia.org>> wrote: >> >> In the interest of finishing up this stage of upgrades, I'm going to >> try to also drain cloudvirt1023 during the same window. That >> includes the following additional VMs: >> >> services: appservice >> privpol-captcha: captcha-imageprocessing-11 >> privpol-captcha: captcha-tf-43 >> pluggableauth: cindy >> search: clickmodel >> codesearch: codesearch4 >> cvn: cvn-apache9 >> cvn: cvn-app8 >> cvn: cvn-app9 >> language: cxserver >> cyberbot: cyberbot-db-01 >> discourse: discourse-mw >> dumps: dumps-3 >> wikifactmine: elasticsearch-01 >> eventmetrics: eventmetrics-prod01 >> fastcci: fastcci-new-master >> catgraph: fishbone >> huggle: huggle-deb-builder >> analytics: k4-1 >> analytics: k4-2 >> puppet: keith-puppetmaster >> language: language-cx1 >> wikibase-registry: libcanada-01 >> catgraph: logintest2 >> matrix: matrix-synapse-01 >> test-twemproxy: mc-clusterA-2 >> mcr-dev: mcr-base >> mcr-dev: mcr-sdc >> wikibase-registry: misc-01 >> ores: ores-lb-03 >> ores-staging: ores-misc-01 >> ores: ores-redis-02 >> ores-staging: ores-staging-01 >> ores: ores-worker-01 >> ores: ores-worker-02 >> phlogiston: phlogiston-4 >> puppet: puppet-ema-2 >> puppet: puppet-jmm-kernel-jessie >> puppet: puppet-jmm-kernel-stretch2 >> thumbor: puppetmaster >> quarry: quarry-worker-01 >> k8splay: qube-node1 >> k8splay: qube-node2 >> recommendation-api: recommendation-api-build >> search: rel2 >> sentry: sentry-builder >> suggestbot: suggestbot-01 >> toolsbeta: toolsbeta-sgewebgrid-generic-0901 >> reading-web-staging: trending >> library-upgrader: upgrader-04 >> visualeditor: visualeditor-test2 >> wikidata-dev: wikibase-stretch >> wikibrain: wikibrain-embeddings-01 >> wikibrain: wikibrain-embeddings-02 >> globaleducation: wikiedubackups >> wikilabels: wikilabels-backups >> wikispeech: wikispeech-tts-dev >> wikispeech: wikispeech-wiki-stretch >> wikistats: wikistats-greyhound >> wikidiff2-wmde-dev: wmde-wikidiff2-debug >> wpx: wpx-mediawiki-02 >> analytics: zk1-2 >> >> >> On 7/31/19 7:55 AM, Andrew Bogott wrote: >>> As part of routine networking and OS upgrades, I'll be emptying two >>> more hypervisors (cloudvirt1021 and cloudvirt1022) on Monday and >>> Tuesday next week, the 5th and 6th. This will result in downtime >>> for many VMs as they are copied and restarted. A complete list of >>> affected instances follows. >>> >>> Please let me know if you need to schedule a more specific >>> window for your downtime. Better yet, if any of the listed VMs are >>> defunct and can simply be deleted, please do that now and save me >>> some time! >>> >>> -Andrew + the WMCS team >>> >>> >>> Affected instances (shown as <project>: <instance name>): >>> >>> >>> account-creation-assistance: accounts-db3 >>> analytics: deployment-server >>> analytics: zk1-1 >>> analytics: zk1-3 >>> antiharassment: antiharassment-web1 >>> automation-framework: af-debmonitor >>> automation-framework: af-puppetmaster02 >>> bastion: bastion-eqiad1-01 >>> bastion: bastion-restricted-eqiad1-01 >>> catgraph: sylvester >>> cloudinfra: mx-out01 >>> cloudinfra: mx-out02 >>> cyberbot: cyberbot-exec-01 >>> design: wikimedia-ui >>> discourse: discourse1002 >>> download: download-01 >>> dumps: dumps-1 >>> dumps: dumps-2 >>> dwl: taxonbot >>> etcd: etcd04 >>> etcd: etcd05 >>> etcd: etcd06 >>> extdist: extdist-01 >>> fastcci: fastcci-worker1 >>> fastcci: fastcci-worker2 >>> git: gerrit-mysql >>> git: gerrit-test >>> git: puppet-paladox >>> google-api-proxy: google-api-proxy-02 >>> grantreview: grantreview-03 >>> hashtags: hashtags-prod >>> hat-imagescalers: k8s-test-builder >>> hhvm: hhvm-jmm-vp9 >>> hhvm: hhvm-stretch-jmm >>> huggle: huggle-wl >>> huggle: xmlrcs >>> iiab: medbox3-iiab >>> integration: integration-slave-docker-1043 >>> k8splay: qube-master >>> language: language-apertium2 >>> language: language-cx >>> language: language-eg >>> language: language-mleb-master >>> language: language-mleb-stable >>> language: moses01-small >>> logging: filippo-log-jessie01 >>> logging: filippo-log-stretch01 >>> mcr-dev: mcr-full >>> mediawiki-vagrant: mwv-builder-02 >>> mediawiki-vagrant: mwv-stretch-migration >>> mediawiki-vagrant: T183456-jessie >>> mediawiki-vagrant: T183456-stretch >>> mix-n-match: mixnmatch >>> mobile: apps-team-tools >>> mobile: emoji >>> mwoffliner: mwoffliner2 >>> mwoffliner: mwoffliner4 >>> mwstake: mwstake >>> mwv-apt: mwv-apt-01 >>> newsletter: newsletter-test >>> openstack: labs-bootstrapvz-jessie >>> openstack: vmbuilder-trusty >>> ores: ores-web-01 >>> ores: ores-web-03 >>> orig: gnd-01 >>> packaging: builder01 >>> packaging: calico-builder01 >>> petscan: petscan3 >>> phabricator: phabricator >>> phabricator: phab-tin >>> phabricator: puppet-phabricator >>> pluggableauth: pluggableauth-server >>> puppet: puppet-jmm-pmaster >>> puppet: puppet-jmm-pmaster-client >>> qna: meza-new4 >>> quarry: quarry-db-01 >>> quarry: quarry-web-01 >>> quarry: quarry-worker-02 >>> reading-web-staging: experiments >>> reading-web-staging: marvin-staging >>> reading-web-staging: readingwebstaging >>> recommendation-api: gapfinder-tools >>> recommendation-api: tool-alpha >>> search: cirrus-browser-bot >>> search: relforge-search >>> search: wdsearch2 >>> snuggle: snuggle-wikidatawiki-01 >>> social-tools: timeless2 >>> striker: striker-uwsgi03 >>> test-twemproxy: mc-clusterB-1 >>> test-twemproxy: mc-clusterB-2 >>> toolsbeta: toolsbeta-sgebastion-04 >>> toolsbeta: toolsbeta-sgeexec-0901 >>> toolsbeta: toolsbeta-sgegrid-master >>> toolserver-legacy: relic-stretch >>> traffic: diffscan >>> traffic: traffic-puppetmaster >>> traffic: traffic-text-stretch >>> utrs: utrs-beta-production >>> utrs: utrs-database2 >>> utrs: utrs-production2 >>> wcdo: wcdo >>> wikibase-registry: orig-01 >>> wikibase-registry: wbregistry-01 >>> wikidata-dev: federated-wikis >>> wikidata-dev: federated-wikis2 >>> wikidata-primary-sources-tool: pst >>> wikidiff2-wmde-dev: sqltest02 >>> wikidiff2-wmde-dev: wmde-wikidiff2-patched >>> wikidiff2-wmde-dev: wmde-wikidiff2-unpatched >>> wikilabels: wikilabels-staging-01 >>> wikimania-support: scholarships-02 >>> wikispeech: wikispeech-tts-release >>> wikitextexp: parsing-qa-01 >>> wildcat: dannyb >>> xtools: xtools-dev04 >>> xtools: xtools-prod03 >>> xtools: xtools-prod05 >>> >> >> >> _______________________________________________ >> Wikimedia Cloud Services announce mailing list >> Cloud-announce(a)lists.wikimedia.org >> <mailto:Cloud-announce@lists.wikimedia.org> (formerly >> labs-announce(a)lists.wikimedia.org >> <mailto:labs-announce@lists.wikimedia.org>) >> https://lists.wikimedia.org/mailman/listinfo/cloud-announce >> _______________________________________________ >> Wikimedia Cloud Services mailing list >> Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) >> https://lists.wikimedia.org/mailman/listinfo/cloud > > > _______________________________________________ > Wikimedia Cloud Services mailing list > Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) > https://lists.wikimedia.org/mailman/listinfo/cloud

1 0

CloudVPS reboots for security updates
by Arturo Borrero Gonzalez 29 Jul '19

29 Jul '19

Hi there! Unrelated to other operations that were communicated recently (datacenter PDU upgrades, operating system upgrades, etc) we need to reboot all the cloudvirt servers to introduce some security updates for CPU vulnerabilities. Along with the physical hardware reboot we also need to reboot all the virtual machines running in CloudVPS. This operation is a bit disruptive but very quick and should not lead to any unexpected errors (is just a reboot). We already tried the same upgrades in some other servers. We will be doing the reboots during this week (starting 2019-07-29). If you see any problems related to this, please contact us. regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

20190726 - afl_log_id field to be removed from wiki replica views
by Brooke Storm 26 Jul '19

26 Jul '19

In the abuse_filter_log table, the afl_log_id field appears to never have been used and is being dropped upstream. To prepare for that and keep the replica view up and running, we are going to start removing the field from the abuse_filter_log views on the wiki replicas service, starting on Monday, July 26th. This should have little to no impact on anyone since the field seems to always be NULL everywhere and to never have been fully implemented, and it would only cause impact to an application if it was specifically queried. For more information on the dropping of the column: https://phabricator.wikimedia.org/T226851 <https://phabricator.wikimedia.org/T226851> Some of the reasons why and a bit more context: https://phabricator.wikimedia.org/T214592 <https://phabricator.wikimedia.org/T214592> Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

1 0

PDU upgrades in the eqiad datacenter affects CloudVPS hypervisor
by Arturo Borrero Gonzalez 26 Jul '19

26 Jul '19

Hi there! There is an ongoing maintenance in the eqiad datacenter that involves changing power connectors of the servers. More info in this phabricator task: T226778 [0]. The PDU upgrade could potentially leave our hypervisors without power briefly. For some hypervisors, we plan to take the risks of leaving them running. For some other hypervisors (those running important DBs in the form of virtual machines) we will probably do a controlled shutdown before the operations to ensure no data corruption happen in the databases. The PDU upgrades will happen this very week (see phab task [0]) and it could potentially affect every virtual machine we run in CloudVPS. This includes Toolforge. In the case of power loss, we expect the disruptions to be very briefly and to don't cause extended downtime in any case. Please, let us know any issue you may find related to this operation. regards. [0] https://phabricator.wikimedia.org/T226778 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

NFS Maintenance 2019-07-25 1500 UTC
by Jason Hedden 26 Jul '19

26 Jul '19

On Thursday, July 25th, 2019 between the hours of 1500 and 1700 UTC we will be performing system maintenance on the NFS servers that support Toolforge and the CloudVPS instances that are using NFS for home, project or scratch data. During this maintenance window, we will be applying rolling updates that require NFS service restarts. We are taking precautions to minimize impact, but there may be short periods of NFS service interruption or performance degradation. --- Jason Hedden Site Reliability Engineer - Wikimedia Cloud Services Wikimedia Foundation

1 2

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce