Wikigadugi.org has been under a massive Bot-Net generated denial of service attack since late yesterday. The IP addresses are from China, Korea, Turkey, and Russia. Blocking at the firewall or proxy just results in more spawned attacks from hundreds of new and unrelated IP addresses. I found one solution which was limit the number of connections httpd allows concurrently and this seems to allow legitimate users to access the system though the attacks persist. The attack pattern seems very specific to MediaWiki behavior. It attempts to load an article then aborts the HTTP request while MediaWiki is churning through the database, then immediately issues another request for another article. It in essense shotguns through the entire name space of articles rapidly. It has trouble taking MediaWiki to its knees but had no trouble taking squid down to a crawl on the proxies and choking the network with garbage.
What do you guys do to deal with these zombie bot-net attacks on this scale?
Jeff
On 23/03/07, Jeff V. Merkey jmerkey@wolfmountaingroup.com wrote:
What do you guys do to deal with these zombie bot-net attacks on this scale?
Be a top-10 website that could slashdot other sites if it wanted to? :-)
- d.
David Gerard wrote:
On 23/03/07, Jeff V. Merkey jmerkey@wolfmountaingroup.com wrote:
What do you guys do to deal with these zombie bot-net attacks on this scale?
Be a top-10 website that could slashdot other sites if it wanted to? :-)
Well, I found if you take squid out of the mix, MediaWiki is resilent and withstands these attacks very well if you limit how much of the pipe gets chewed up by the excessive connections. Squid did not handle it well.
Jeff
- d.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l
Jeff V. Merkey wrote:
Wikigadugi.org has been under a massive Bot-Net generated denial of service attack since late yesterday. The IP addresses are from China, Korea, Turkey, and Russia. Blocking at the firewall or proxy just results in more spawned attacks from hundreds of new and unrelated IP addresses. I found one solution which was limit the number of connections httpd allows concurrently and this seems to allow legitimate users to access the system though the attacks persist. The attack pattern seems very specific to MediaWiki behavior. It attempts to load an article then aborts the HTTP request while MediaWiki is churning through the database, then immediately issues another request for another article. It in essense shotguns through the entire name space of articles rapidly. It has trouble taking MediaWiki to its knees but had no trouble taking squid down to a crawl on the proxies and choking the network with garbage.
What do you guys do to deal with these zombie bot-net attacks on this scale?
Block all involved IPs at the firewall. It's a workable solution as long as you can identify the problem requests in your logs. It's entirely feasible to block thousands of IP addresses in this way. Per-IP limits such as apache's mod_throttle will also help, as will system optimisation and caching.
-- Tim Starling
On Fri, Mar 23, 2007 at 07:39:39PM +0000, Tim Starling wrote:
Block all involved IPs at the firewall. It's a workable solution as long as you can identify the problem requests in your logs. It's entirely feasible to block thousands of IP addresses in this way. Per-IP limits such as apache's mod_throttle will also help, as will system optimisation and caching.
Has anyone ever looked into getting mod_throttle or an IDS to write the DROP rules for the firewall?
Cheers, -- jra
Tim Starling wrote:
Jeff V. Merkey wrote:
Wikigadugi.org has been under a massive Bot-Net generated denial of service attack since late yesterday. The IP addresses are from China, Korea, Turkey, and Russia. Blocking at the firewall or proxy just results in more spawned attacks from hundreds of new and unrelated IP addresses. I found one solution which was limit the number of connections httpd allows concurrently and this seems to allow legitimate users to access the system though the attacks persist. The attack pattern seems very specific to MediaWiki behavior. It attempts to load an article then aborts the HTTP request while MediaWiki is churning through the database, then immediately issues another request for another article. It in essense shotguns through the entire name space of articles rapidly. It has trouble taking MediaWiki to its knees but had no trouble taking squid down to a crawl on the proxies and choking the network with garbage.
What do you guys do to deal with these zombie bot-net attacks on this scale?
Block all involved IPs at the firewall.
Already started. Didn't help. I ended up blocking thousands of ranges and they just kept cloning Zombies from new ranges faster than I could enter them. Throttling seems to make sense however.
It's a workable solution as long as you can identify the problem requests in your logs. It's entirely feasible to block thousands of IP addresses in this way. Per-IP limits such as apache's mod_throttle will also help, as will system optimisation and caching.
I changed the cache settings on the squid proxy servers this morning -- it helped. They are slowing down. I guess the person doing it is getting bored, I am 99% certain based on a single Russian IP address (which I know was being used to monitor the attack and its progress) that I blocked who is doing this. It's the guy running GNAA -- the one who called me one night and posted an MP3 of the conversation online and on the GNAA website and the Wikipedia chatroom.
When I blocked his "monitoring IP Address" he got really pissed and intentified the attack. The Russian Guy who works at a software company by day and trolls the internet by night. He lives somewhere in Oregon, and runs these attacks from bot-nets all over the world.
At any rate, the network is stable and holding its own. I wrote a port 80 shim last night that throttles IP traffic by address range and shunted to over to the main Web Server at about 11:00 p.m. I was unaware apache had something similiar. I'll look at it and get it enabled.
Jeff
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 3/23/07, Jeff V. Merkey jmerkey@wolfmountaingroup.com wrote:
Tim Starling wrote:
Jeff V. Merkey wrote:
Wikigadugi.org has been under a massive Bot-Net generated denial of service attack since late yesterday. The IP addresses are from China, Korea, Turkey, and Russia. Blocking at the firewall or proxy just results in more spawned attacks from hundreds of new and unrelated IP addresses. I found one solution which was limit the number of connections httpd allows concurrently and this seems to allow legitimate users to access the system though the attacks persist. The attack pattern seems very specific to MediaWiki behavior. It attempts to load an article then aborts the HTTP request while MediaWiki is churning through the database, then immediately issues another request for another article. It in essense shotguns through the entire name space of articles rapidly. It has trouble taking MediaWiki to its knees but had no trouble taking squid down to a crawl on the proxies and choking the network with garbage.
What do you guys do to deal with these zombie bot-net attacks on this scale?
Block all involved IPs at the firewall.
Already started. Didn't help. I ended up blocking thousands of ranges and they just kept cloning Zombies from new ranges faster than I could enter them. Throttling seems to make sense however.
It's a workable solution as long as you can identify the problem requests in your logs. It's entirely feasible to block thousands of IP addresses in this way. Per-IP limits such as apache's mod_throttle will also help, as will system optimisation and caching.
I changed the cache settings on the squid proxy servers this morning -- it helped. They are slowing down. I guess the person doing it is getting bored, I am 99% certain based on a single Russian IP address (which I know was being used to monitor the attack and its progress) that I blocked who is doing this. It's the guy running GNAA -- the one who called me one night and posted an MP3 of the conversation online and on the GNAA website and the Wikipedia chatroom.
When I blocked his "monitoring IP Address" he got really pissed and intentified the attack. The Russian Guy who works at a software company by day and trolls the internet by night. He lives somewhere in Oregon, and runs these attacks from bot-nets all over the world.
At any rate, the network is stable and holding its own. I wrote a port 80 shim last night that throttles IP traffic by address range and shunted to over to the main Web Server at about 11:00 p.m. I was unaware apache had something similiar. I'll look at it and get it enabled.
Jeff
-- Tim Starling
If he's living in Oregon, stop dicking around with us, call the FBI and your local police. It's a federal crime, and if he's in the US, they'll go get him...
On Fri, Mar 23, 2007 at 01:27:39PM -0800, George Herbert wrote:
If he's living in Oregon, stop dicking around with us, call the FBI and your local police. It's a federal crime, and if he's in the US, they'll go get him...
Wouldn't it be pretty to think so.
Some pretty high-profile sites have tried this approach to DOS attacks in the past, I'm told. Not much came of it, except that some people got to be laughed at by Special Agents on the phone.
In general, unless there are at least 6 digits in the amount of money someone's stealing from you, and preferably 7, the FBI hasn't proven to be really interested.
Maybe someone should call Scott Bakula, instead.
Cheers, -- jra
Jay R. Ashworth wrote:
On Fri, Mar 23, 2007 at 01:27:39PM -0800, George Herbert wrote:
If he's living in Oregon, stop dicking around with us, call the FBI and your local police. It's a federal crime, and if he's in the US, they'll go get him...
Wouldn't it be pretty to think so.
Some pretty high-profile sites have tried this approach to DOS attacks in the past, I'm told. Not much came of it, except that some people got to be laughed at by Special Agents on the phone.
In general, unless there are at least 6 digits in the amount of money someone's stealing from you, and preferably 7, the FBI hasn't proven to be really interested.
Maybe someone should call Scott Bakula, instead.
Cheers, -- jra
This is more accurate. We had one of these trolls/phisers break into a border server here at Solera Networks last year with an older Linux version from Hungry and they hijacked the server and used it to "harvest" over 300,000 credit card and bank account passwords/numbers from the Federal Credit Union. An FBI agent showed up and took the hard drive, then returned it a month later and told us "we cannot go after people in Hungary, but thanks for the hard drive so we could find out whose accounts were stolen and can refund their money."
They will do NOTHING unless you have a dead nun laying on the floor of your data center with a least four stab wounds and 7 digits stolen from her.
Jeff
wikitech-l@lists.wikimedia.org