Why does my Python background process end when SSH session is terminated?
I would disconnect the command from its standard input/output and error flows:
nohup python3 -u <script> </dev/null >/dev/null 2>&1 &
ssh needs an indicator that doesn't have any more output and that it does not require any more input. Having something else be the input and redirecting the output means ssh can safely exit, as input/output is not coming from or going to the terminal. This means the input has to come from somewhere else, and the output (both STDOUT and STDERR) should go somewhere else.
The </dev/null part specifies /dev/null as the input for <script>. Why that is useful here:
Redirecting /dev/null to stdin will give an immediate EOF to any read call from that process. This is typically useful to detach a process from a tty (such a process is called a daemon). For example, when starting a background process remotely over ssh, you must redirect stdin to prevent the process waiting for local input. https://stackoverflow.com/questions/19955260/what-is-dev-null-in-bash/19955475#19955475
Alternatively, redirecting from another input source should be relatively safe as long as the current ssh session doesn't need to be kept open.
With the >/dev/null part the shell redirects the standard output into /dev/null essentially discarding it. >/path/to/file will also work.
The last part 2>&1 is redirecting STDERR to STDOUT.
There are three standard sources of input and output for a program. Standard input usually comes from the keyboard if it’s an interactive program, or from another program if it’s processing the other program’s output. The program usually prints to standard output, and sometimes prints to standard error. These three file descriptors (you can think of them as “data pipes”) are often called STDIN, STDOUT, and STDERR.
Sometimes they’re not named, they’re numbered! The built-in numberings for them are 0, 1, and 2, in that order. By default, if you don’t name or number one explicitly, you’re talking about STDOUT.
Given that context, you can see the command above is redirecting standard output into /dev/null, which is a place you can dump anything you don’t want (often called the bit-bucket), then redirecting standard error into standard output (you have to put an & in front of the destination when you do this).
The short explanation, therefore, is “all output from this command should be shoved into a black hole.” That’s one good way to make a program be really quiet!
What does > /dev/null 2>&1 mean? | Xaprb
Look at man ssh:
ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec] [-D [bind_address:]port] [-e escape_char] [-F configfile] [-I pkcs11] [-i identity_file] [-L [bind_address:]port:host:hostport] [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port] [-R [bind_address:]port:host:hostport] [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]] [user@]hostname [command]
When you run ssh -i <keyfile> -o StrictHostKeyChecking=no <user>@<hostname> "./startup.sh" you are running the shell script startup.sh as an ssh command.
From the description:
If command is specified, it is executed on the remote host instead of a login shell.
Based on this, it should be running the script remotely.
The difference between that and running nohup python3 -u <script> & in your local terminal is that this runs as a local background process while the ssh command attempts to run it as a remote background process.
If you intend to run the script locally then don't run startup.sh as part of the ssh command. You might try something like ssh -i <keyfile> -o StrictHostKeyChecking=no <user>@<hostname> && "./startup.sh"
If your intention is to run the script remotely and you want this process to continue after your ssh session is terminated, you would have to first start a screen session on the remote host. Then you have to run the python script within screen and it will continue to run after you end your ssh session.
See Screen User's Manual
While I think screen is your best option, if you must use nohup, consider setting shopt -s huponexit on the remote host before running the nohup command. Alternatively, you can use disown -h [jobID] to mark the process so SIGHUP will not be sent to it.1
How do I keep running job after I exit from a shell prompt in background?
The SIGHUP (Hangup) signal is used by your system on controlling terminal or death of controlling process. You can use SIGHUP to reload configuration files and open/close log files too. In other words if you logout from your terminal all running jobs will be terminated. To avoid this you can pass the -h option to disown command. This option mark each jobID so that SIGHUP is not sent to the job if the shell receives a SIGHUP.
Also, see this summary of how huponexit works when a shell is exited, killed or dropped. I'm guessing your current issue is related to how the shell session ends.2
All child processes, backgrounded or not of a shell opened over an ssh connection are killed with SIGHUP when the ssh connection is closed only if the huponexit option is set: run shopt huponexit to see if this is true.
If huponexit is true, then you can use nohup or disown to dissociate the process from the shell so it does not get killed when you exit. Or, run things with screen.
If huponexit is false, which is the default on at least some linuxes these days, then backgrounded jobs will not be killed on normal logout.
- But even if huponexit is false, then if the ssh connection gets killed, or drops (different than normal logout), then backgrounded processes will still get killed. This can be avoided by disown or nohup as in (2).
Finally, here are some examples of how to use shopt huponexit.3
$ shopt -s huponexit; shopt | grep huponexit
huponexit on
# Background jobs will be terminated with SIGHUP when shell exits
$ shopt -u huponexit; shopt | grep huponexit
huponexit off
# Background jobs will NOT be terminated with SIGHUP when shell exits
Maybe worth trying -n option when starting a ssh? It will prevent remote process dependency on a local stdin, which of course closes as soon as ssh session ends. And this will cause remote prices termination whenever it tries to access its stdin.