Hi there!
Unrelated to other operations that were communicated recently (datacenter PDU
upgrades, operating system upgrades, etc) we need to reboot all the cloudvirt
servers to introduce some security updates for CPU vulnerabilities.
Along with the physical hardware reboot we also need to reboot all the virtual
machines running in CloudVPS.
This operation is a bit disruptive but very quick and should not lead to any
unexpected errors (is just a reboot). We already tried the same upgrades in some
other servers.
We will be doing the reboots during this week (starting 2019-07-29). If you see
any problems related to this, please contact us.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi there!
There is an ongoing maintenance in the eqiad datacenter that involves changing
power connectors of the servers. More info in this phabricator task: T226778 [0].
The PDU upgrade could potentially leave our hypervisors without power briefly.
For some hypervisors, we plan to take the risks of leaving them running. For
some other hypervisors (those running important DBs in the form of virtual
machines) we will probably do a controlled shutdown before the operations to
ensure no data corruption happen in the databases.
The PDU upgrades will happen this very week (see phab task [0]) and it could
potentially affect every virtual machine we run in CloudVPS. This includes
Toolforge.
In the case of power loss, we expect the disruptions to be very briefly and to
don't cause extended downtime in any case.
Please, let us know any issue you may find related to this operation.
regards.
[0] https://phabricator.wikimedia.org/T226778
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
I tried to clean this up myself, but I don't have the password for the mailman
admin panel.
regards.
-------- Forwarded Message --------
Subject: 18 Cloud-announce moderator request(s) waiting
Date: Tue, 23 Jul 2019 08:00:20 +0000
From: cloud-announce-bounces(a)lists.wikimedia.org
To: cloud-announce-owner(a)lists.wikimedia.org
Notice: 2 old request(s) automatically expired.
The Cloud-announce(a)lists.wikimedia.org mailing list has 18 request(s)
waiting for your consideration at:
https://lists.wikimedia.org/mailman/admindb/cloud-announce
Please attend to this at your earliest convenience. This notice of
pending requests, if any, will be sent out daily.
Pending posts:
From: office.amortransoil(a)mail.ru on Thu Jul 18 10:29:59 2019
Subject: AMOR SCO.
Cause: Post by non-member to a members-only list
From: comercial(a)tiendasmuybaratas.es on Thu Jul 18 12:41:53 2019
Subject: Creación Tiendas Internet
Cause: Post by non-member to a members-only list
From: salesco2(a)rexel.ae on Thu Jul 18 13:39:50 2019
Subject: Request for Quotation
Cause: Post by non-member to a members-only list
From: kate(a)sigairhandling.hu on Thu Jul 18 23:21:18 2019
Subject: SIG Air Handling Hungary Kft PI request
Cause: Post by non-member to a members-only list
From: tahasinauddina(a)gmail.com on Fri Jul 19 00:20:11 2019
Subject: Cause: Post by non-member to a members-only list
From: tahasinauddina(a)gmail.com on Fri Jul 19 00:22:57 2019
Subject: Cause: Post by non-member to a members-only list
From: mans247(a)inter.net.il on Fri Jul 19 06:54:42 2019
Subject: Invoice Copy
Cause: Post by non-member to a members-only list
From: noreply-location-sharing+528e70db(a)google.com on Sat Jul 20 00:35:18 2019
Subject: See তাহসিন উদ্দিন's real-time location on Google Maps
Cause: Post by non-member to a members-only list
From: tahasinauddina(a)gmail.com on Sat Jul 20 02:12:17 2019
Subject: See event "Bangladesh floods" on Google Maps
Cause: Post by non-member to a members-only list
From: noreply-2d52644cb32ddf2381cec81070b3ae55(a)google.com on Sat Jul 20 10:45:37
2019
Subject: তাহসিন উদ্দিন shared “Nur” with you
Cause: Post by non-member to a members-only list
From: inv(a)ui.prato.it on Sat Jul 20 11:58:46 2019
Subject: Budget Cleaners Tax Invoice
Cause: Post by non-member to a members-only list
From: narisissara41(a)gmail.com on Sun Jul 21 11:25:20 2019
Subject: Cause: Post by non-member to a members-only list
From: zhu(a)lg.com on Mon Jul 22 05:49:04 2019
Subject: Fw: Sumex inv97s4301
Cause: Post by non-member to a members-only list
From: richard(a)nissenken.or.jp on Mon Jul 22 06:29:46 2019
Subject: Order 3005389 Tokyo//Nissenken Zheng 07/19 CN
Cause: Post by non-member to a members-only list
From: contact(a)aduna-co.info on Mon Jul 22 12:11:22 2019
Subject: Fwd:Bank Transfer
Cause: Post by non-member to a members-only list
From: santosh.malekar(a)meed.com on Mon Jul 22 16:23:53 2019
Subject: MEED INV-042079 D269955
Cause: Post by non-member to a members-only list
From: info(a)tas.ge on Tue Jul 23 03:39:48 2019
Subject: You have been selected to receive a private donation from us:
Cause: Post by non-member to a members-only list
From: richard(a)nissenken.or.jp on Tue Jul 23 07:52:30 2019
Subject: Tokyo//Nissenken Zheng 07/19 CN
Cause: Post by non-member to a members-only list
I saw Arturo referring to the decisions we made in Dublin about DNS
naming in a phab task today. When I went to look at the page on
Wikitech I realized that we had not strongly signaled there what the
outcome of the discussions were and that there was a bit of ambiguity
in the "winning" proposal. I have attempted to write up what I believe
we collectively agreed on as
<https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme…>.
Please review this write up and be ready to quickly thumbs up/thumbs
down the write up in the 2019-07-23 WMCS team meeting. Folks on this
list who do not attend that meeting but who wish to object to the
write up can do so by responding to this thread. +1 votes via email
are not necessary.
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA
irc: bd808 v:415.839.6885 x6855