What exactly determines if a backgrounded job is killed when the shell is exited, or killed?
A process isn't "killed with SIGHUP" -- at least, not in the strict sense of the word. Rather, when the connection is dropped, the terminal's controlling process (in this case, Bash) is sent a hang-up signal*, which is commonly abbreviated the "HUP signal", or just SIGHUP.
Now, when a process receives a signal, it can handle it any way it wants**. The default for most signals (including HUP) is to exit immediately. However, the program is free to ignore the signal instead, or even to run some kind of signal handler function.
Bash chooses the last option. Its HUP signal handler checks to see if the "huponexit" option is true, and if so, sends SIGHUP to each of its child processes. Only once its finished with that does Bash exit.
Likewise, each child process is free to do whatever it wants when it receives the signal: leave it set to the default (i.e. die immediately), ignore it, or run a signal handler.
Nohup only changes the default action for the child process to "ignore". Once the child process is running, however, it's free change its own response to the signal.
This, I think, is why some programs die even though you ran them with nohup:
- Nohup sets the default action to "ignore".
- The program needs to do some kind of cleanup when it exits, so it installs a SIGHUP handler, incidentally overwriting the "ignore" flag.
- When the SIGHUP arrives, the handler runs, cleaning up the program's data files (or whatever needed to be done) and exits the program.
- The user doesn't know or care about the handler or cleanup, and just sees that the program exited despite nohup.
This is where "disown" comes in. A process that's been disowned by Bash is never sent the HUP signal, regardless of the huponexit option. So even if the program sets up its own signal handler, the signal is never actually sent, so the handler never runs. Note, however, that if the program tries to display some text to a user that's logged out, it will cause an I/O error, which could cause the program to exit anyway.
* And, yes, before you ask, the "hang-up" terminology is left over from UNIX's dialup mainframe days.
** Most signals, anyway. SIGKILL, for instance, always causes the program to terminate immediately, period.
Points 1-4 are correct.I know nothing about point 5. As for your final point, a fine application, screen, will allow you to let all processes run to their natural end, regardless of how you terminate your connection. Screen is in the repos.
The man description of screen is not easy to read, but, among other things, it states:
When screen is called, it creates a single window with a shell in it (or the specified command) and then gets out of your way so that you can use the program as you normally would. Then, at any time, you can create new (full-screen) windows with other programs in them (including more shells), kill existing windows, view a list of windows, turn output logging on and off, copy-and-paste text between windows, view the scrollback history, switch between windows in whatever manner you wish, etc. All windows run their programs completely indepen‐ dent of each other. Programs continue to run when their window is currently not visible and even when the whole screen session is detached from the user's terminal. When a program terminates, screen (per default) kills the window that contained it. If this window was in the foreground, the display switches to the previ‐ ous window; if none are left, screen exits.
I have highlighted the most important part: you can detach the window with the command Ctrl+a+d, and then you may kill/logout your session, and the now-detached window will continue to live, with the programs inside still running. When you connect back, for instance by initiating a new ssh session, the command screen -r will resume the screen session which had been detached earlier, with all output to standard error/output clearly visible.