One consequence of this outage is that the server behind the Toolforge Stretch bastion (login-stretch.tools.wmflabs.org) has changed. If you are seeing a scary warning like this:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is SHA256:8fLy4F9XDYdR/uHihWoPihKDhPaxCh0au/paSdGB7K8. Please contact your system administrator. Add correct host key in *HOME*/.ssh/known_hosts to get rid of this message. Offending ECDSA key in *HOME*/.ssh/known_hosts:*LINE* ECDSA host key for login-stretch.tools.wmflabs.org has changed and you have requested strict checking.
Host key verification failed.
then you will need to update your known_hosts file. It probably contains a line like this:
login-stretch.tools.wmflabs.org,185.15.56.48 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEEMihgdO9CXKJvpoO4LMOt1cU43zIQJiXOm1doVMh0z+uXntQkNDyF\ eHJ9//T983eL8efbCBEgnB9POGfYfoas=
You can either change this to
login-stretch.tools.wmflabs.org,185.15.56.48 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFnJSjCGW7kli+cdgtmndPAl4xLZNc9uqP9KWlsnVDqr8yQ2RkR5ACb\ Xe6XZ+dS09Wc9ulOmGTOwCImMi9Fho78=
or remove the line and then look for the following output the next time you SSH into the bastion:
The authenticity of host 'login-stretch.tools.wmflabs.org (185.15.56.48)' can't be established. ECDSA key fingerprint is SHA256:8fLy4F9XDYdR/uHihWoPihKDhPaxCh0au/paSdGB7K8.
Good luck! Cheers, Lucas
Am Mi., 13. Feb. 2019 um 22:53 Uhr schrieb Bryan Davis <bd808@wikimedia.org
:
On Wed, Feb 13, 2019 at 2:45 PM Maximilian Doerr maximilian.doerr@gmail.com wrote:
I ask this because of these failures. Where does cyberbot-db-01 live?
Per https://tools.wmflabs.org/openstack-browser/project/cyberbot it is on cloudvirt1023.eqiad.wmnet
The data on there is critical.
As you probably know, we do not currently have a trusted back up solution for Cloud VPS projects. Our best recommendation for 'critical' data is for you to setup some manual or automated backup to an offsite location (your laptop, a VPS hosted outside Cloud VPS, etc). Hopefully we will have some news on an actual reliable backup service in the coming months. We have some hardware to build an initial system for this, but have not yet had time to design and implement the backup service itself.
Bryan
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud