One consequence of this outage is that the server behind the Toolforge Stretch bastion (login-stretch.tools.wmflabs.org) has changed. If you are seeing a scary warning like this:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:8fLy4F9XDYdR/uHihWoPihKDhPaxCh0au/paSdGB7K8.
Please contact your system administrator.
Add correct host key in HOME/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in HOME/.ssh/known_hosts:LINE
ECDSA host key for login-stretch.tools.wmflabs.org has changed and you have requested strict checking.                                                                     
Host key verification failed.

then you will need to update your known_hosts file. It probably contains a line like this:

login-stretch.tools.wmflabs.org,185.15.56.48 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEEMihgdO9CXKJvpoO4LMOt1cU43zIQJiXOm1doVMh0z+uXntQkNDyF\
eHJ9//T983eL8efbCBEgnB9POGfYfoas=

You can either change this to

login-stretch.tools.wmflabs.org,185.15.56.48 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFnJSjCGW7kli+cdgtmndPAl4xLZNc9uqP9KWlsnVDqr8yQ2RkR5ACb\
Xe6XZ+dS09Wc9ulOmGTOwCImMi9Fho78=

or remove the line and then look for the following output the next time you SSH into the bastion:

The authenticity of host 'login-stretch.tools.wmflabs.org (185.15.56.48)' can't be established.

ECDSA key fingerprint is SHA256:8fLy4F9XDYdR/uHihWoPihKDhPaxCh0au/paSdGB7K8.

Good luck!

Cheers,

Lucas

Am Mi., 13. Feb. 2019 um 22:53 Uhr schrieb Bryan Davis <bd808@wikimedia.org>:

On Wed, Feb 13, 2019 at 2:45 PM Maximilian Doerr
<maximilian.doerr@gmail.com> wrote:
>
> I ask this because of these failures.  Where does cyberbot-db-01 live?

Per <https://tools.wmflabs.org/openstack-browser/project/cyberbot> it
is on cloudvirt1023.eqiad.wmnet

> The data on there is critical.

As you probably know, we do not currently have a trusted back up
solution for Cloud VPS projects. Our best recommendation for
'critical' data is for you to setup some manual or automated backup to
an offsite location (your laptop, a VPS hosted outside Cloud VPS,
etc). Hopefully we will have some news on an actual reliable backup
service in the coming months. We have some hardware to build an
initial system for this, but have not yet had time to design and
implement the backup service itself.

Bryan
--
Bryan Davis              Wikimedia Foundation    <bd808@wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Technical Engagement    Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855

_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

--

Lucas Werkmeister
Full Stack Developer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30 219 158 26-0
https://wikimedia.de

Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision!
https://spenden.wikimedia.de

Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.