Tim Starling wrote:
Jeff V. Merkey wrote:
Wikigadugi.org has been under a massive Bot-Net generated denial of service attack since late yesterday. The IP addresses are from China, Korea, Turkey, and Russia. Blocking at the firewall or proxy just results in more spawned attacks from hundreds of new and unrelated IP addresses. I found one solution which was limit the number of connections httpd allows concurrently and this seems to allow legitimate users to access the system though the attacks persist. The attack pattern seems very specific to MediaWiki behavior. It attempts to load an article then aborts the HTTP request while MediaWiki is churning through the database, then immediately issues another request for another article. It in essense shotguns through the entire name space of articles rapidly. It has trouble taking MediaWiki to its knees but had no trouble taking squid down to a crawl on the proxies and choking the network with garbage.
What do you guys do to deal with these zombie bot-net attacks on this scale?
Block all involved IPs at the firewall.
Already started. Didn't help. I ended up blocking thousands of ranges and they just kept cloning Zombies from new ranges faster than I could enter them. Throttling seems to make sense however.
It's a workable solution as long as you can identify the problem requests in your logs. It's entirely feasible to block thousands of IP addresses in this way. Per-IP limits such as apache's mod_throttle will also help, as will system optimisation and caching.
I changed the cache settings on the squid proxy servers this morning -- it helped. They are slowing down. I guess the person doing it is getting bored, I am 99% certain based on a single Russian IP address (which I know was being used to monitor the attack and its progress) that I blocked who is doing this. It's the guy running GNAA -- the one who called me one night and posted an MP3 of the conversation online and on the GNAA website and the Wikipedia chatroom.
When I blocked his "monitoring IP Address" he got really pissed and intentified the attack. The Russian Guy who works at a software company by day and trolls the internet by night. He lives somewhere in Oregon, and runs these attacks from bot-nets all over the world.
At any rate, the network is stable and holding its own. I wrote a port 80 shim last night that throttles IP traffic by address range and shunted to over to the main Web Server at about 11:00 p.m. I was unaware apache had something similiar. I'll look at it and get it enabled.
Jeff
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l