SSH remote port forwarding failed
Solution 1:
I agree with MadHatter, that it is likely to be port forwardings from defunct ssh connections. Even if your current problem turns out to be something else, you can expect to run into such defunct ssh connections sooner or later.
There are three ways such defunct connections can happen:
- One of the two endpoints got rebooted while the other end of the connection was completely idle.
- One of the two endpoints closed the connection, but at the time where the connection was closed, there was a temporary outage on the connection. The outage lasted for a few minutes after the connection was closed, and thus the other end never learned about the closed connection.
- The connection is still completely functional at both endpoints of the ssh connection, but somebody has put a stateful device somewhere between them, which timed out the connection due to idleness. This stateful device would be either a NAT or a firewall, the firewall you already mentioned is a prime suspect.
Figuring out which of the above three is happening is not highly important, because there is a method, which will address all three. That is the use of keepalive messages.
You should look into the ClientAliveInterval
keyword for sshd_config
and the ServerAliveInterval
interval for ssh_config
or ~/.ssh/config
.
Running the ssh
command in a loop can work fine. It is a good idea to insert a sleep in the loop as well such that you don't end up flooding the server when the connection for some reason fails.
If the client reconnect before the connection has terminated on the server, you can end up in a situation where the new ssh connection is live, but has no port forwardings. In order to avoid that, you need to use the ExitOnForwardFailure
keyword on the client side.
Solution 2:
You can find the process that's binding the port on that server with
sudo netstat -apn|grep -w X
It seems very likely to be the half-defunct sshd
, but why make assumptions when you can have data? It's also a good way for a script to find a PID to send signal 9 to before trying to bring the tunnel up again.
Solution 3:
For me when a ssh
tunnel disconnects it takes awhile for the connection to reset so the ssh
process continues to block leaving me with no active tunnels and I don't know why. A workaround solution is to put ssh
into the background with -f
and to spawn new connections without waiting for old connections to reset. The -o ExitOnForwardFailure=yes
can be used to limt the number of new processes. The -o ServerAliveInterval=60
improves the reliability of your current connection.
You can repeat the ssh
command frequently, say, in a cron
, or, in a loop in your script, for example, in the following, we run the ssh
command every 3 minutes:
while (1)
do
ssh -f user@hostname -Rport:host:hostport -N -o ExitOnForwardFailure=yes -o ServerAliveInterval=60
sleep 180
done