The epic guide to filtering DDoS attacks

[eLight]

This thread will probably end up in the tutorials section eventually, but for now I'll place it here. It should be a pretty handy guide for a lot of people. DDoS attacks aren't uncommon and sometimes it's possible to filter them out server side, sometimes not. For the times that are however, the following suggestions should be pretty handy. These are mainly for the "common HTTP flood". There are many, many types of attacks, but we'll focus on this one for now.

This guide will assume a few of things:

1. You own the server or have root access to this.
2. You're able to generate logs of the traffic (such as an access log, or error log for example).
3. You have permissions to install and use things such as Perl and iptables.

1.
Working out if you're the victim of a DDoS attack.

Now, let's say that you're browsing your site, or in SSH, or uploading something and you notice there's a slow down or you're even getting disconnected. First things first, you have to work out if the problem is with you or whether it's with the network you're on. For example, another DDoS on a server in the same switch will usually slow down anything connected to it depending on the severity. It's even possible that the switch will start distributing the packets amongst the ports even where it's not intended. This usually happens in overload so you'll see traffic coming towards your server even though you're not the actual target.

If the server is very slow, run either top or uptime from SSH. It'll return a load figure. Depending on the amount of work the server does it's usually possible to see a load spike, depending on what is being attacked of course. Many servers hover around 1, 2, 3, some 5, some more or less. But you'll see a spike if there's a slow down and I usually say that if you're server is going anywhere over 10, there's something wrong anyways. Also check the CPU usage on top, that will give you another idea if there's something fishy up. If it's constantly sitting on full CPU usage and that's out of the ordinary, there's a good chance that's due to a flood or someone executing some poorly coded script. Something that I've noticed is that a lot of control panels or things such as AWStats can cause excessive load at certain times of the day. If you're unsure if this is happening, or maybe something else is being executed on the server via a cron job, run these:

crontab -l
date

Compare the crontabs to the time on the server. If they coincide, that may be the problem. Otherwise, let's assume the worst

So secondly, check your servers bandwidth charts. Usually this is possible via your hosting providers website.



If they don't offer bandwidth charts don't stress! It's still possible to monitor bandwidth on the server itself. To do this, there are many tools. Personally, I use ibmonitor. It's an old tool which hasn't been updated in a long time, but it gets the job done. To use this tool:

wget http://prdownloads.sourceforge.net/ibmonitor/ibmonitor-1.4.tar.gz?download
tar -zxf ibmonitor-1.4.tar.gz
cd ibmonitor
./ibmonitor --max --colors --data -avg

That will give an output similar to this:


If you have heaps of devices loaded but internet traffic is only via one, you can use the above command with the --dev option to monitor it. In this instance, we'll monitor eth0.

./ibmonitor --max --colors --data -avg --dev eth0

As you can see, it's now only showing bandwidth on eth0. But how can you tell from this information whether or not you're being attacked? Well, you can't precisely though it should give you a fair idea. What kind of traffic does your site use on a month to month basis? For instance: 300GB a month should mean that your internet traffic will hover around 1,000Kbps +/- in the total part of eth0 during regular traffic hours. If you're site is larger, 20,000Kbps may not be out of the ordinary. Though 50,000 may be.

2.
I am being attacked! My bandwidth levels are crazy! What now?

First of all, don't stress too much. The thing about DDoS attacks is that you can't really control them from hitting your server. And in many cases, you won't be able to do much at all. But it's worth trying. Just remember if you're rushing and typing stuff freely into the command line it's not hard to screw things up.

Alright, so the following things are noticed:

1. Bandwidth in or out is high
2. Server load is high

Let's turn on some access logs for your web server if they're not on already. Amongst the configuration files you'll usually find something like this:

Apache:
CustomLog "/path/to/access_log" common

lighttpd:
accesslog.filename  = "/path/to/access.log"

nginx:
access_log /path/to/access.log

You'll need to restart the service after making that change if it wasn't there already.

Let's see if there's a lot of traffic hitting something specific on the server at once. This can be done pretty easily by tailing the log. In this instance, I've downloaded an openly available acccess.log from a Google search for these examples.

tail -f /home/ddos/access.log

This will show all events as they're occurring. Sometimes there may be too much for you to watch without it screwing up your connection, so use it wisely. If you think that the access.log is growing a lot faster than usual (there's more hits occurring than usual) you can use this command to monitor the access.log

watch -d -n3 "wc -l /home/ddos/access.log"

This command will count how many lines are in the access.log and update on an interval of your choosing. In this example, we've used an interval of 30 seconds so that I can take screenshots. When monitoring, use something much lower such as 3 (in the quoted command).

In the screenshot below, you'll see that the total amount of lines of the access.log is 1,424,471



In the next, you'll notice the last 3 digits updated (they're highlighted) from 471 to 844.



That's an increase of 373 hits to the web server in 30 seconds. An average of 12.5 or so per second. It's not unusual to see this amount much, much higher (potentially several thousand per second).

So what's being flooded? A page? An image? We can analyze the logs to work this out. This part needs a little bit of your own intuition as it's sometimes not easy to work out what's being flooded, as it's not always obvious. For example, a poorly written piece of PHP code could be hit 100 times in 10 seconds which basically renders the server useless. While an image may be requested 500 times in 10 seconds just fine.

So let's check out if things are out of the ordinary. To make this faster I recommend creating a new log, as a log that doesn't contain 3 months of prior hits to your website is much faster to scan and also more accurate of the current situation.

Is it a PHP script?

tail -n 500 /home/ddos/access.log | grep .php | wc -l

In our instance, 401 of the most recent 500 requests are to a PHP file. So that's definitely the most requested thing by a long margin. But, if you think it may be something else, such as an image, try using this (perhaps edit in the extension you're looking for):

tail -n500 /home/ddos/access.log | perl -ne 'if(/\w+\.(png|gif|jpg)/i){print "$_\n"};' | wc -l

3.
Um, my page.php is being flooded! How can I stop this?

There's no perfect solution to blocking attacking IP's only. Most of the time the requests are made to appear completely legit. However, a lot of botnet owners are lazy (or just script kiddies with no real idea). So you can notice patterns if you look closely. So, we've established that page.php is being attacked, let's look at a few of these requests and see if we can find anything.

In our example access.log, the page that I am pretending is being hit is "/logs//admin/..." so my command consists of that. Ignore the fact that these are 404'd.

tail -n500 /home/ddos/access.log | grep /logs//admin/ | uniq

The output of this command is this:


Notice anything that's out of the ordinary? For starters, there's many, many requests to a supposed admin page. But secondly, the UserAgent for all requests is libwww-perl (and many different versions of libwww). That's not a regular browser like FireFox or IE? It's similar to Curl for PHP. It can be used from web servers to make requests to other web servers. And WTF are they doing accessing the admin page anyway? The command above made a list of all unique requests to the admin page in the past 500 requests. Let's do this on the entire log and only for requests that are from this dreaded libwww-perl.

cat /home/ddos/access.log | perl -ne 'if(/libwww/){print "$_\n"};' | wc -l

There's some 9,000+ matches for libwww in the access log. Let's put these into their own file.

cat /home/ddos/access.log | perl -ne 'if(/libwww/){print "$_\n"};' > libwww.log

Now we'll use libwww.log and find which IP's are the main culprits.

cat /home/ddos/access.log | perl -ne  'if(/\d+\.\d+\.\d+\.\d+/){chomp;s/^(\d+\.\d+\.\d+\.\d+).*$/$1/;print "$_\n"}'; | uniq -c | sort -n

The output of this command lists the IP's prefixed by the amount of times they exist in the file. For example, 195.38.16.4 exists 64 times. If there's a real stand out, it may be wise to block that IP first before moving onto the others in order to get some stability back.

     45 212.52.167.210
     48 78.129.139.235
     50 216.245.192.42
     54 78.129.139.235
     54 85.214.116.153
     60 72.29.93.26
     64 195.38.16.4

Let's create a uniq list of the commands in another file that we can use. Tip: This can all be done in one command to make it much easier, I've just broken down the process to make it easier to under stand

cat libwww.log | perl -ne 'if(/\d+\.\d+\.\d+\.\d+/){chomp;s/^(\d+\.\d+\.\d+\.\d+).*$/$1/;print "$_\n"};' | sort | uniq > uniqlog

Is there heaps of requests from certain ranges? Check with this:

cat uniqlog | perl -ne 'if(/\d+\.\d+\.\d+\./){chomp;s/^(\d+\.\d+\.\d+\.).*$/$1/;print "$_\n"};' | sort | uniq -c | sort

That will give you a list of the most common ranges. Something similar tot his:

      1 99.198.111.
      2 180.210.205.
      2 187.61.61.
      2 196.35.68.
      2 81.28.98.
      2 91.121.79.
      4 91.215.216.

You can block the worst only if you so wish. But regardless, we're now assuming that all of these IP's are unwanted traffic. Let's block them all, shall we? Everyone has their own preference for how to block an IP. So it's up to you which iptables command you want to use (if you use iptables at all). It is possible to block these directly from the command line. However, I prefer to log them to a file and then run that file via bash just for peace of mind. If you let a command run loose that blocks anything, you could be up shit creek. Sometimes you might HAVE to do this if load is bad. But otherwise, at least check over what you're about to block. Especially if there's hundreds or thousands of IP's.

cat /home/ddos/uniqlog | perl -ne 'if(/\d+\.\d+\.\d+\.\d+/){chomp;s/^(\d+\.\d+\.\d+\.\d+).*$/$1/;print "iptables -t mangle -I PREROUTING -s $_ -p tcp --syn -j DROP\n"};' > toblock.sh

Opening toblock.sh should show similar stuff to this:

iptables -t mangle -I PREROUTING -s 118.127.4.50 -p tcp --syn -j DROP
iptables -t mangle -I PREROUTING -s 119.252.166.226 -p tcp --syn -j DROP
iptables -t mangle -I PREROUTING -s 122.116.65.237 -p tcp --syn -j DROP
iptables -t mangle -I PREROUTING -s 122.201.104.71 -p tcp --syn -j DROP
iptables -t mangle -I PREROUTING -s 122.201.76.100 -p tcp --syn -j DROP
iptables -t mangle -I PREROUTING -s 124.150.142.86 -p tcp --syn -j DROP
iptables -t mangle -I PREROUTING -s 151.24.106.151 -p tcp --syn -j DROP
iptables -t mangle -I PREROUTING -s 173.192.190.178 -p tcp --syn -j DROP
iptables -t mangle -I PREROUTING -s 173.199.133.140 -p tcp --syn -j DROP
iptables -t mangle -I PREROUTING -s 173.224.125.200 -p tcp --syn -j DROP

To execute it run:

sh /home/ddos/toblock.sh

Did it work? Let's see.

iptables-save -c > iptables
nano iptables

The saved iptables can be restored after a reboot also. So it's wise to make occasional backups. If you're a little more confident in SSH, you'll realize that all the commands I've listed here are possible via the execution of 3 or 4 of them combined. It's possible to go from realizing your site is being attacked to blocking IPs within 60 seconds or less.

Hopefully this helped someone out

If people are interested I may expand tutorials to blocking other less common forms of DDoS attacks.

P.S.
If it's not a DDoS but only a "DoS" that being from a singular IP, you can use the famous "netstat" command.

netstat -anp | grep ":80" | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

This will display the IPs with the most connections to the server. To make things easier (rather than copy and pasting each IP) perhaps use something like the following:

netstat -anp | grep ":80" | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | tail -n10 && netstat -anp | grep ":80" | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | tail -n10 | awk '{print $2}' | perl -ne 'if(/\d+\.\d+\.\d+\.\d+/){chomp;s/(\d+\.\d+\.\d+\.\d+).*$/$1/;print "iptables -t mangle -I PREROUTING -s $_ -p tcp --syn -j DROP\n"}';

This will show the IPs with the most connections and underneath those same IPs in a command to block them. It's as easy and copy and pasting directly this way. To display more or less, change the tail -n value.



Until next time

eLight Reviewed by eLight on 14th May 2011. The epic guide to filtering DDoS attacks This thread will probably end up in the tutorials section eventually, but for now I'll place it here. It should be a pretty handy guide for a lot of people. DDoS attacks aren't uncommon and sometimes it's possible to filter them out server side, sometimes not. For the times that are however, the following suggestions should be pretty handy. These are mainly for the "common HTTP flood". There are many, many types of attacks, but we'll focus on this one for now. This guide will assume a few of Rating: 5

[snkr007]

wow thanks

[Daniel]

If I'm under a dos attack. Now I know where to go. Thanks

[Jesse]

Very good, e

[dsn]

Thanks

[eLight]

Fixed an error in that last command, the screenshot is a bit wrong but it works as it should now

[jooker]

epic guide ..

[chrishdman87]

Haha nice job e
I had a hunch you were going to make a thread like this.

[Sponge Bob]

wow thats some deadly tutorial.

i see u learnt all this when KWWHunction was under attack

[eLight]

Haha nice job e
I had a hunch you were going to make a thread like this.

Chris & Sp0nge: Lol, someone should benefit anyway