Stop DoS attacks with an IP tables rule?

For something that offers flexibility, look into the recent (and limit) module(s). Recent will keep a track of the number of connections made by an IP, over a given time frame, and can be used to trigger a specific rule. The recent module is (relatively) CPU heavy - but when compared to loading a dynamic page, can be quite acceptable.

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p tcp -m multiport --dports 80,443 -m recent --update --seconds 3600 --name BANNED --rsource -j DROP
-A INPUT -p tcp -m multiport --dports 80,443 -m state --state NEW -j ATTK_CHECK

-A ATTACKED -m limit --limit 5/min -j LOG --log-prefix "IPTABLES (Rule ATTACKED): " --log-level 7
-A ATTACKED -m recent --set --name BANNED --rsource -j DROP
-A ATTK_CHECK -m recent --set --name ATTK –-rsource
-A ATTK_CHECK -m recent --update --seconds 600 --hitcount 150 --name ATTK --rsource -j ATTACKED
-A ATTK_CHECK -m recent --update --seconds 60 --hitcount 50 --name ATTK --rsource -j ATTACKED
-A ATTK_CHECK -j ACCEPT

Essentially, the above amounts to:

• Accept connections that have been established already (i.e. they have passed our rules)
• Drop all connections on ports 80 and 443 (SSL) from banned IPs - if they keep trying to connect, extend the time - wait an hour (without any connection attempts) before unblocking them.
• Check new connections against our rules:
  • Consider more than 150 connections in 10 minutes an attack
  • Consider more than 50 connections in 1 minute an attack
  • Log the attack (but not more than 5 entries per minute - don't want to flood our logs); and block the connection

For a more detailed explanation, and a tiered version of the above, which I use, see this article

Keep in mind that ipTables will not help with network congestion at all, and is fairly ineffective against any distributed attacks - its advantage is in security and reducing the load on your server.

You can use iptables to block an abuser like this, but you have to be careful because iptables is very CPU-intensive. Every packet is checked against every rule, so adding rules can quickly cause the server to run out of cycles. I had a client once that would add a new rule every time they wanted to block someone, and then one day they had slightly higher than normal traffic and the system fell over. Tables can reduce the number of rules looked at, but you can't just throw a lot of rules in iptables.

Another alternative is black-hole routing:

ip route add blackhole $IP_HERE
ip route flush cache

The routing table is hashed (with a hashtable), whereas iptables rules have to be looked at sequentially, which is much less efficient. This way, you can have hundreds of thousands of routes without a problem, where thousands of iptables rules would become problematic to run through for every request.

As far as your second question, you can use the hashlimit module to create an automatic blacklist for abusers. See my post about preventing a DDoS attack for more details, but the short form is:

iptables -A INPUT -p tcp --dport 80 -m hashlimit --hashlimit-upto 50/min \
    --hashlimit-burst 500 --hashlimit-mode srcip --hashlimit-name http -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j DROP