Linux networking port exhaustion

You have two main questions here:

1.

Strictly IPv4 speaking, is port exhaustion actually possible?

Yes. Take, for example, a load balancing router sending all connections to a NAT IP address. This is likely to happen when you have many SRC IPs connecting to the bottleneck of a single DST IP.

This means that your webserver could have a bunch of connections like:

root@buglab:~# netstat -pnt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 173.200.1.18:80      10.100.1.100:49923        ESTABLISHED 13939/nginx: worker
tcp        0      0 173.200.1.18:80      10.200.1.200:10155        ESTABLISHED 13939/nginx: worker
tcp        0      0 173.200.1.18:80      10.10.1.10:14400        ESTABLISHED 13939/nginx: worker
tcp        0      0 173.200.1.18:80      10.10.1.10:50652        ESTABLISHED 13939/nginx: worker
tcp        0      0 173.200.1.18:80      10.20.1.20:57554        ESTABLISHED 13939/nginx: worker 

and that's perfectly fine. However, if all the 'Foreign Addresses' were the same, this can cause an issue (e.g. 'big router that performs NAT <---> server with one IP address').

If I had to postulate as to why ephemeral port exhaustion isn't a common issue, I'd suggest it's because every port requires a listening service and enough resources to respond -- another resource (memory, cpu) is normally a bottleneck first.

However, I've personally come across a few port exhaustion issues when working at a load balancing company.

2. Why can a used port present an issue for a listening service?

"Which allows the use of ephemeral ports from 1024-65535, that if I have services that bind on port 3306 (mySQL, for example), they will sometimes fail to start because the port is in use."

The mySQL server cannot bind to that port if it's in use - say by localhost:3306 or on all interfaces. For example, see the 0.0.0.0:80 line in the following netstat output?

root@buglab:~# netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      PID/Program name
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      964/php-fpm.conf)
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1660/mysqld     
tcp        0      0 0.0.0.0:842             0.0.0.0:*               LISTEN      1317/inetd      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      13938/nginx     

That means that port 80 is listening across all interfaces local to the server. If another process holds port 80 before my nginx server starts, nginx will not be able to take control of that port and will likely fail its startup procedure.

Normally, port 3306 is fine because listening services have pre-defined ports (or ranges) that are requested from the host machine - e.g. port 80 and 443 for webservers.