Why do I get permission denied error when I log out of the SSH session?
Thanks to Mark Plotnick I was able to identify and fix the issue.
The problem is the interaction between the AFS file system used by the server and Kerberos handling the authentication. The same issue was brought up in this question on SO.
Basically what is happening is that when I ssh into the server, Kerberos gives the authentication token to the session. This token is used also to access the AFS file system. When closing the SSH session this token gets destroyed and the processes running start to get permission denied errors when trying to access files on the AFS.
The way to fix this is to start a new window inside screen
/tmux
and launch the command:
kinit && aklog
After that you can detach from screen
/tmux
and close the ssh session safely.
The commands above create new Kerberos tokens and associate those with the screen
/tmux
session, in this way when the ssh connection is closed the initial tokens get revoked but since the subprocesses now use those you created they don't suffer permission denied errors.
To summarize:
ssh username@server
tmux
- Launch the process you need to keep running
- Create a new window with Ctrl+B, C
kinit && aklog
- Detach from the session with Ctrl+B, D
- Close ssh session