This is an email to shell account holders on translatewiki.net and to wikitech-l, so that you are informed.
Today at 08:10 UTC Niklas noticed that the translatewiki.net server had been compromised. We saw some suspicious files in /tmp and a few processes that didn't belong:
elastic+ 22862 0.0 0.0 2684 2388 ? S 04:53 0:00 /tmp/freeBSD /tmp/freeBSD 1 elastic+ 31575 0.0 0.0 2684 2388 ? S 06:38 0:00 /tmp/freeBSD /tmp/freeBSD 1 elastic+ 31580 16.7 0.0 90816 724 ? Ssl 06:38 16:26 [.Linux_time_y_2]
We gathered data and looked at our recent traffic statistics. We drew the following conclusions:
- Only the Elasticsearch account had been compromised. The intruder did not gain access to other accounts. - The attack could be made because the Elasticsearch process was bound to all interfaces, instead of only the localhost interface, and dynamic scripting was enabled, because it is required by CirrusSearch (CVE-2014-3120). - A virtual machine was started, and given the traffic that was generated (about 1TB in the past 4 days), we think this was a DDoS drone. The process reported to an IP address in China. - A server reinstall is the right thing to do (better safe than sorry).
The compromised server was taken off-line around 10:00 UTC today.
Actions taken: - Bind Elasticsearch only to localhost from now on: https://gerrit.wikimedia.org/r/#/c/145262/ - Reinstall the server
Actions to be taken: - Configure a firewall to only allow expected traffic to enter and exit the translatewiki.net server so that something like the added virtual machine could not have communicated to the outside world. - As a precaution, shell account holders should change any secret that they have used on the translatewiki.net server in the past 7 days.
We are thankful to the people in the MediaWiki security IRC channel and Henri Salo for helping us with data gathering on the attack, and how to proceed.
We have re-installed the translatewiki.net server, and are currently re-importing the databases. We expect to be back online in a few hours. Once we come back online, we'll still have to rebuild some non-critical meta data stores, like populating the search database.
Cheers!
Siebrand
On Thu, Jul 10, 2014 at 10:09 AM, Siebrand Mazeland siebrand@kitano.nl wrote:
This is an email to shell account holders on translatewiki.net and to wikitech-l, so that you are informed.
Today at 08:10 UTC Niklas noticed that the translatewiki.net server had been compromised. We saw some suspicious files in /tmp and a few processes that didn't belong:
elastic+ 22862 0.0 0.0 2684 2388 ? S 04:53 0:00 /tmp/freeBSD /tmp/freeBSD 1 elastic+ 31575 0.0 0.0 2684 2388 ? S 06:38 0:00 /tmp/freeBSD /tmp/freeBSD 1 elastic+ 31580 16.7 0.0 90816 724 ? Ssl 06:38 16:26 [.Linux_time_y_2]
We gathered data and looked at our recent traffic statistics. We drew the following conclusions:
- Only the Elasticsearch account had been compromised. The intruder did not
gain access to other accounts.
- The attack could be made because the Elasticsearch process was bound to
all interfaces, instead of only the localhost interface, and dynamic scripting was enabled, because it is required by CirrusSearch (CVE-2014-3120).
- A virtual machine was started, and given the traffic that was generated
(about 1TB in the past 4 days), we think this was a DDoS drone. The process reported to an IP address in China.
- A server reinstall is the right thing to do (better safe than sorry).
The compromised server was taken off-line around 10:00 UTC today.
Actions taken:
- Bind Elasticsearch only to localhost from now on:
https://gerrit.wikimedia.org/r/#/c/145262/
- Reinstall the server
Actions to be taken:
- Configure a firewall to only allow expected traffic to enter and exit the
translatewiki.net server so that something like the added virtual machine could not have communicated to the outside world.
- As a precaution, shell account holders should change any secret that they
have used on the translatewiki.net server in the past 7 days.
Did this server have access to private ssh keys that are used to push/merge code for upstream repos? If so, will they be rotated as well?
- Ryan
On Thu, Jul 10, 2014 at 7:12 PM, Ryan Lane rlane32@gmail.com wrote:
- As a precaution, shell account holders should change any secret that
they
have used on the translatewiki.net server in the past 7 days.
Did this server have access to private ssh keys that are used to push/merge code for upstream repos? If so, will they be rotated as well?
Yes, by means of key forwarding. Those keys have been updated. For two projects it still has to be done (OpenStreetMap and MantisBT); for all other projects, this has already been done.
Cheers!
wikitech-l@lists.wikimedia.org