Why do consoles sometimes hang forever when SSH connection breaks?
Solution 1:
There is a "secret" keyboard shortcut to force an exit :~) From the frozen session, hit these keys in order: Enter~. The tilde (only after a newline) is recognized as an escape sequence by the ssh client, and the period tells the client to terminate it's business without further ado.
The long-hang behavior on communication issues is not a bug, the SSH session is hanging out hoping the other side will come back. If the network breaks, sometimes even days later you can get an SSH session back. Of course you can specifically tell it to give up and die with the sequence above. There are also various things you can do such as setting keep-alive timeouts in your client so that if it doesn't have an active link for a certain amount of time it shuts off on it's own, but the default behavior is to stay as connected as possible!
Edit: Another useful application of this interrupt key is to get the attention of the local ssh client and background it to get back to your local shell for a minute —say to get something from your history— then forground it to keep working remotely. Enter~ Ctrl+Z to send the ssh client to the background job queue of your local shell, then fg as normal to get it back.
Edit: When dealing with nested SSH sessions, you can add multiple tilde characters to only break out of one of the SSH sessions in the chain, but retain the others. For example, if you're nested in 3 levels, (i.e. you ssh from local->Machine1->Machine2->Machine3), Enter~. will get you back to your local session, Enter~~. will leave you in Machine1, and Enter~~~. will leave you in Machine2. This works for other escape sequences as well, such as moving the ssh session to background temporarily. The above works for any level of nesting, by just adding more tilde's.
Finally, you can use Enter~? to print a help menu of available escape commands.
TL;DR - the supported escape commands are Supported escape sequences:
~. - terminate connection (and any multiplexed sessions)
~B - send a BREAK to the remote system
~C - open a command line
~R - request rekey
~V/v - decrease/increase verbosity (LogLevel)
~^Z - suspend ssh
~# - list forwarded connections
~& - background ssh (when waiting for connections to terminate)
~? - this message
~~ - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)
Solution 2:
SSH offers a keep-alive facility. Add the following to your local ~/.ssh/config (create if it doesn't exist):
ServerAliveInterval 15
ServerAliveCount 3
This setting will establish a keep-alive signal sent every 15 seconds through the secure tunnel. After three consecutive failures, the SSH client will exit.
Note that on some systems (including macOS 10.14) instead it needs to be:
ServerAliveInterval 15
ServerAliveCountMax 3
Taken from this answer on ask.ubuntu: https://askubuntu.com/a/29967/30266
Solution 3:
The fact that it hangs is a function of TCP, not SSH. The application has no way of knowing that the TCP session/connection has been severed unless TCP informs the application using the connection that the connection no longer exists. From the perspective of each host, the TCP session is still in the Established state and there's nothing to say that a long idle (no data flowing) session is not valid other than a RST or a lack of response to a TCP keepalive packet (which is not universally implemented). That doesn't appear to be a bug in SSH to me, I'd expect that behavior.
Solution 4:
If the connection is properly hung any magic keystroke will not get through as the connection is already hung before you issue the key presses. You can tell the client to terminate but this won't affect the history kept (or not) at the server end.
While this doesn't actually answer the question it may help reduce the effects of a connection hang: when I'm working remotely (and even usually when I'm not) I run through screen (with or without the byobu wrapper depending on its availability) so that if there is any sort of connection drop my session, with all its history, is preserved and available in the state I left it when I reconnect.