Limit max connections per IP address and new connections per second with iptables

iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 15 --connlimit-mask 32 -j REJECT --reject-with tcp-reset  

This will reject connections above 15 from one source IP.

iptables -A INPUT -m state --state RELATED,ESTABLISHED -m limit --limit 150/second --limit-burst 160 -j ACCEPT  

In this 160 new connections (packets really) are allowed before the limit of 150 NEW connections (packets) per second is applied.

You want the following rules in your iptables to answer both requirements in your question:

iptables -t filter -I INPUT -p tcp --dport 80 -j ACCEPT

iptables -t filter -I INPUT -p tcp --dport 80 -m state \
  --state RELATED,ESTABLISHED -j ACCEPT

# Adjust "--connlimit-above NN" to limit the maximum connections per IP
#   that you need.
iptables -t filter -I INPUT -p tcp --syn --dport 80 -m connlimit \
  --connlimit-above 10 --connlimit-mask 32 -j DROP

# Adjust "--connlimit-above NNN" to the maximum total connections you
#   want your web server to support
iptables -t filter -I INPUT -p tcp --syn --dport 80 -m connlimit \
  --connlimit-above 150 -j DROP

Because we are using -I (as per the OP request) we have to do them in reverse order so 'read' them from the bottom up.

I also suggest considering --connlimit-mask NN change from 32 to 24. This will limit a full Class-C network (max 256 IP addresses in the same range) to 10 connections. You could also use any other classless number like 22 or 30 depending on how you think your service might be used.

Also depending on how you want the client to behave, you might want to use "-j REJECT --reject-with tcp-reset" instead of "-j DROP" in the two rules above, or even only in the 150 connections max rule.

If you REJECT the connection the browser or software using port 80 will show a "not available" status immediately, but the DROP option will cause the client to wait and retry a few times before reporting the site as not available. I tend to lean to the DROP myself as it behaves more like a bad connection than an offline server.

Also, if the connection limit drops back down below 150 (or 10) while it is still retrying, then it will finally get through to your server.

The REJECT option will cause a fraction less traffic to your site however, as DROP will cause it to send additional packets while it retries. Probably not all that relevant.

If on the other hand your port 80 traffic is part of a cluster then REJECT will tell the cluster controller that it's down and to stop sending traffic to it for the duration of it's retry timeout.

The RELATED,ESTABLISHED rule is there under the assumption your default rule is to block all traffic (iptables -t filter -P INPUT DROP). This just accepts futher packets belonging to accepted connections.

Also --syn tells it to pay attention to (or count) the packets that set up a TCP connection.

You need to use the connlimit modules which allows you to restrict the number of parallel TCP connections to a server per client IP address (or address block).

/sbin/iptables -I INPUT -p tcp --syn --dport 80 -m connlimit \
      --connlimit-above 10 -j DROP