What limits the maximum number of connections on a Linux server?
Solution 1:
I finally found the setting that was really limiting the number of connections: net.ipv4.netfilter.ip_conntrack_max. This was set to 11,776 and whatever I set it to is the number of requests I can serve in my test before having to wait tcp_fin_timeout seconds for more connections to become available. The conntrack table is what the kernel uses to track the state of connections so once it's full, the kernel starts dropping packets and printing this in the log:
Jun 2 20:39:14 XXXX-XXX kernel: ip_conntrack: table full, dropping packet.
The next step was getting the kernel to recycle all those connections in the TIME_WAIT state rather than dropping packets. I could get that to happen either by turning on tcp_tw_recycle or increasing ip_conntrack_max to be larger than the number of local ports made available for connections by ip_local_port_range. I guess once the kernel is out of local ports it starts recycling connections. This uses more memory tracking connections but it seems like the better solution than turning on tcp_tw_recycle since the docs imply that that is dangerous.
With this configuration I can run ab all day and never run out of connections:
net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 32768 61000
The tcp_max_orphans setting didn't have any effect on my tests and I don't know why. I would think it would close the connections in TIME_WAIT state once there were 8192 of them but it doesn't do that for me.
Solution 2:
You really want to look at what the /proc filesystem has to offer you in this regard.
- TCP Tuning Guide at the US Dept. of Energy
- TCP Tuning parameters for different OS:es
- IBM's "Administer Linux on the fly"
- Documentation at LinuxInsight.com vis-a-vis /proc/sys/net/ipv4
On that last page, you might find the following to be of interest to you:
- /proc/sys/net/ipv4/tcp_max_orphans, which controls the maximum number of sockets held by the system not attached to something. Raising this can consume as much as 64kbyte of non-swappable memory per orphan socket.
- /proc/sys/net/ipv4/tcp_orphan_retries, which controls the amount of retries before a socket is orphaned and closed. There is a specific note on that page about web servers that is of direct interest to you...
Solution 3:
I don't think there is a tunable to set that directly. This falls under the category of TCP/IP tuning. To find out what you can tune, try 'man 7 tcp'. The sysctl ( 'man 8 sysctl' ) is used to set these. 'sysctl -a | grep tcp' will show you most of what you can tune, but I am not sure if it will show all of them. Also, unless this has changed, TCP/IP sockets open up look like file descriptors. So this and the next section in that link might be what you are looking for.
Solution 4:
Try setting the following as well setting tcp_fin_timeout. This should close out TIME_WAIT more quickly.
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
Solution 5:
The stock apache(1) used to come predefined to only support 250 concurrent connections - if you wanted more, there was one header file to modify to allow more concurrent sessions. I don't know if this is still true with Apache 2.
Also, you need to add an option to allow loads of more open file descriptors for the account that runs Apache - something that the previous comments fail to point out.
Pay attention to your worker settings and what sort of keepalive timeouts you have inside Apache itself, how many spare ones servers you have running at once, and how fast these extra processes are getting killed.