bash script executed over ssh returns incorrect exit code 0
I am able to duplicate this using the command you used, and I am able to resolve it by wrapping the remote command in quotes. Here are my test cases:
#!/bin/bash -x
echo 'Unquoted Test:'
ssh evil sh -x -c exit 5 && echo OK || echo FAIL
echo 'Quoted Test 1:'
ssh evil sh -x -c 'exit 5' && echo OK || echo FAIL
echo 'Quoted Test 2:'
ssh evil 'sh -x -c "exit 5"' && echo OK || echo FAIL
Here are the results:
bash-[540]$ bash -x test.sh
+ echo 'Unquoted Test:'
Unquoted Test:
+ ssh evil sh -x -c exit 5
+ exit
+ echo OK
OK
+ echo 'Quoted Test 1:'
Quoted Test 1:
+ ssh evil sh -x -c 'exit 5'
+ exit
+ echo OK
OK
+ echo 'Quoted Test 2:'
Quoted Test 2:
+ ssh evil 'sh -x -c "exit 5"'
+ exit 5
+ echo FAIL
FAIL
In the first test and second tests, it seems the 5
is not being passed to exit
as we would expect it to be. It just seems to be disappearing. It's not going to exit
, sh
isn't complaining about 5: command not found
, and ssh
isn't complaining about it.
In the third test, exit 5
is quoted within the larger command to run on the remote host, same as in the second test. This ensures that the 5
is passed to exit
, and both are executed as the -c
option to sh
. The difference between the second and third tests is that the whole set of commands and arguments is sent to the remote host quoted as a single command argument to ssh
.
As noted in the answer you already have, the remote sh
is not executing exit 5
. Just exit
:
$ ssh test sh -x -c 'exit 5'; echo $?
+ exit
0
What is happening here is explained, for instance, in this answer:
ssh
executes a remote shell and passes a string to it, not a list of arguments.
When we execute ssh host sh -c 'exit 5'
:
- The local shell removes the single quotes (quote removal);
- The
ssh
client gets the argumentshost
,sh
,-c
, andexit 5
. It concatenates them to a string and sends it to the remote host; - On the remote host,
ssh
invokes a shell and passes it the stringsh -c exit 5
; - The remote shell invokes
sh
and passes it the-c
option,exit
as the command string, and5
as the command name.
Note that, if we add words after exit 5
, they are just passed to sh
as further arguments - no error related to them not being recognized by the shell:
$ ssh test sh -x -c 'exit 5' a b c; echo $?
+ exit
0
strace
confirms that 5
is not part of the command string given to sh
, here; it is an argument:
$ ssh test strace -e execve sh -c 'exit 5'; echo $?
execve("/usr/bin/sh", ["sh", "-c", "exit", "5"], 0x7ffc0d744c38 /* 14 vars */) = 0
+++ exited with 0 +++
0
In order to execute sh -c 'command'
on a remote host as intended, we have to be sure to properly send it the quotes too:
$ ssh test "sh -x -c 'exit 5'"; echo $?
+ exit 5
5
To make it clear that quoting the whole remote command is not relevant to our current issue, we could just write:
$ ssh test sh -x -c "'exit 5'"; echo $?
+ exit 5
5
Escaping the inner quotes with backslashes, instead of quoting two times, would work as well.
A note about the command ssh host sh -c ':; exit 5'
(from the comments to your question). What it does is:
$ ssh test sh -x -c ':; exit 5'; echo $?
+ :
5
That is, exit 5
is executed by the outer shell, not by sh
. Again, to let sh
exit with the desired code:
$ ssh test sh -x -c "':; exit 5'"; echo $?
+ :
+ exit 5
5
The other answers are good at answering the question in lieu of the examples given. My real-world application is more complicated and involves a series of scripts and sub-processes. Here is a boiled-down example script I want to execute:
#!/bin/bash
sub-process-that-fails
# store and echo returncode for debug purposes
rc=$?
echo $rc
exit $rc
Trying to make sure that the remotely executed shell was actually bash and not dash (as pointed out by @JeffSchaller), I tried calling the script like this:
~$ ssh -t -t host /bin/bash -x /srv/scripts/run.sh ; echo $?
Which led to this weird output:
+ sub-process-that-fails
+ rc=5
+ echo 5
5
+ exit 5
0
After hours of poking around, I noticed there was a trap 'kill 0' EXIT
set in the .bashrc
. This is done to kill all sub-processes in case bash is killed. bash's trace does not seem to display this trap's execution. I moved the trap into the wrapper script. Now I can see what actually is executed:
+ trap 'kill 0' EXIT
+ sub-process-that-fails
+ rc=5
5
+ echo 5
+ exit 5
+ kill 0
0
The remote shell exits with the last command's exit code. It's kill 0
and it exits with 0.