Re: [Wikitech-l] translatewiki.net compromised: server reinstalled

This is an email to shell account holders on translatewiki.net and to wikitech-l, so that you are informed. Today at 08:10 UTC Niklas noticed that the translatewiki.net server had been compromised. We saw some suspicious files in /tmp and a few processes that didn't belong: elastic+ 22862 0.0 0.0 2684 2388 ? S 04:53 0:00 /tmp/freeBSD /tmp/freeBSD 1 elastic+ 31575 0.0 0.0 2684 2388 ? S 06:38 0:00 /tmp/freeBSD /tmp/freeBSD 1 elastic+ 31580 16.7 0.0 90816 724 ? Ssl 06:38 16:26 [.Linux_time_y_2] We gathered data and looked at our recent traffic statistics. We drew the following conclusions: - Only the Elasticsearch account had been compromised. The intruder did not gain access to other accounts. - The attack could be made because the Elasticsearch process was bound to all interfaces, instead of only the localhost interface, and dynamic scripting was enabled, because it is required by CirrusSearch (CVE-2014-3120). - A virtual machine was started, and given the traffic that was generated (about 1TB in the past 4 days), we think this was a DDoS drone. The process reported to an IP address in China. - A server reinstall is the right thing to do (better safe than sorry). The compromised server was taken off-line around 10:00 UTC today. Actions taken: - Bind Elasticsearch only to localhost from now on: https://gerrit.wikimedia.org/r/#/c/145262/ - Reinstall the server Actions to be taken: - Configure a firewall to only allow expected traffic to enter and exit the translatewiki.net server so that something like the added virtual machine could not have communicated to the outside world. - As a precaution, shell account holders should change any secret that they have used on the translatewiki.net server in the past 7 days.