How can I reconnect to a ssh session after a broken pipe?

The answer to your proper question is: you can't. I think the main problem is that the authentication procedures will be out of sync. It Just Doesn't Work Like That.

As you have yourself noticed, the solution is to use screen when possible (by the way, tmux is an alternative to screen).


While you can't reattach to a broken SSH session, you can reparent the process running inside SSH – functionally equivalent to what you want.

Instructions

In your case, you would take over the apt-get process to be controlled from a new SSH session, screen session or the like. My favourite for this is the reptyr command:

$ sudo apt-get install reptyr
$ ps ax | grep apt-get
10626 pts/8   R+     0:32 apt-get upgrade

Then, with the pid you found for your process:

$ sudo reptyr -T 10626

Or if that does not work, try:

$ reptyr 10626

After this stage, all your keyboard input goes to the program you took over. Unfortunately you will not see old output of the SSH session, such as the apt-get output asking you for confirmation.

Explanations

There are multiple other tools that basically work the same as reptyr (that is, via ptrace debug attachment). See the following questions and answers where they are discussed:

  • How can I disown a running process and associate it to a new screen shell?
  • Resume command running in dropped SSH session

In the instructions above, the reptyr 10626 uses ptrace debug attachment while the sudo reptyr -T 10626 command uses TTY stealing and is preferable (details).

Finally, the reason why you can't take over a SSH session this way is because a sshd process is not controlled by a host terminal, instead it provides the slave part of a terminal – a pts device – while the master part controlling it resides on the client machine, here with a broken-down SSH session in between. When you force taking over such a sshd process with reptyr -s <pid>, your keyboard input goes to that process, not its active child process. So a "Ctrl+Z" will simply kill off that sshd.


For running long lasting processes, I use screen, or byobu if you want a more friendly interface.

For screen, you can use:

screen [program] [args]

This will run [program] and its [args] inside a screen session. Once the program is finished, the session is automatically closed. If you wish to keep the session after your program runs, just run screen without any arguments and a new prompt will appear inside the session. CTRL+A+D detaches the terminal from the current session.

To re-attach to a previous session:

screen -r

If there is only one session open, it will reattach immediately. If multiple sessions are ongoing, it will ask you which one you want to attach to. If you know the session name, you can just add it as an argument to this command line.

Byobu is a nice improvement. It's based on screen, but provides a bar at the bottom that shows all current sessions as tabs and gives easier shortcuts to move around those. You can:

  • F2 start a new session
  • F3 move to the next session tab on the left
  • F4 move to the next session tab on the right
  • F8 give a friendly name to the current session tab
  • F9 opens a options menu
  • CTRL+A+D detaches all sessions from the terminal.

WORD OF ADVICE: avoid leaving a session opened with the user root. If anyone gains access to your terminal (locally or remotely), they can easily re-attach to an ongoing session and use your system as root. If needed, it's best to start a session using a common user and sudo indivudual command lines as necessary.

Tags:

Ssh

Openssh

Recent Posts