bash script executed over ssh returns incorrect exit code 0

I am able to duplicate this using the command you used, and I am able to resolve it by wrapping the remote command in quotes. Here are my test cases:

#!/bin/bash -x

echo 'Unquoted Test:'
ssh evil sh -x -c exit 5 && echo OK || echo FAIL

echo 'Quoted Test 1:'
ssh evil sh -x -c 'exit 5' && echo OK || echo FAIL

echo 'Quoted Test 2:'
ssh evil 'sh -x -c "exit 5"' && echo OK || echo FAIL

Here are the results:

bash-[540]$ bash -x test.sh
+ echo 'Unquoted Test:'
Unquoted Test:
+ ssh evil sh -x -c exit 5
+ exit
+ echo OK
OK
+ echo 'Quoted Test 1:'
Quoted Test 1:
+ ssh evil sh -x -c 'exit 5'
+ exit
+ echo OK
OK
+ echo 'Quoted Test 2:'
Quoted Test 2:
+ ssh evil 'sh -x -c "exit 5"'
+ exit 5
+ echo FAIL
FAIL

In the first test and second tests, it seems the 5 is not being passed to exit as we would expect it to be. It just seems to be disappearing. It's not going to exit, sh isn't complaining about 5: command not found, and ssh isn't complaining about it.

In the third test, exit 5 is quoted within the larger command to run on the remote host, same as in the second test. This ensures that the 5 is passed to exit, and both are executed as the -c option to sh. The difference between the second and third tests is that the whole set of commands and arguments is sent to the remote host quoted as a single command argument to ssh.


As noted in the answer you already have, the remote sh is not executing exit 5. Just exit:

$ ssh test sh -x -c 'exit 5'; echo $?
+ exit
0

What is happening here is explained, for instance, in this answer:

ssh executes a remote shell and passes a string to it, not a list of arguments.

When we execute ssh host sh -c 'exit 5':

  1. The local shell removes the single quotes (quote removal);
  2. The ssh client gets the arguments host, sh, -c, and exit 5. It concatenates them to a string and sends it to the remote host;
  3. On the remote host, ssh invokes a shell and passes it the string sh -c exit 5;
  4. The remote shell invokes sh and passes it the -c option, exit as the command string, and 5 as the command name.

Note that, if we add words after exit 5, they are just passed to sh as further arguments - no error related to them not being recognized by the shell:

$ ssh test sh -x -c 'exit 5' a b c; echo $?
+ exit
0

strace confirms that 5 is not part of the command string given to sh, here; it is an argument:

$ ssh test strace -e execve sh -c 'exit 5'; echo $?
execve("/usr/bin/sh", ["sh", "-c", "exit", "5"], 0x7ffc0d744c38 /* 14 vars */) = 0
+++ exited with 0 +++
0

In order to execute sh -c 'command' on a remote host as intended, we have to be sure to properly send it the quotes too:

$ ssh test "sh -x -c 'exit 5'"; echo $?
+ exit 5
5

To make it clear that quoting the whole remote command is not relevant to our current issue, we could just write:

$ ssh test sh -x -c "'exit 5'"; echo $?
+ exit 5
5

Escaping the inner quotes with backslashes, instead of quoting two times, would work as well.


A note about the command ssh host sh -c ':; exit 5' (from the comments to your question). What it does is:

$ ssh test sh -x -c ':; exit 5'; echo $?
+ :
5

That is, exit 5 is executed by the outer shell, not by sh. Again, to let sh exit with the desired code:

$ ssh test sh -x -c "':; exit 5'"; echo $?
+ :
+ exit 5
5

The other answers are good at answering the question in lieu of the examples given. My real-world application is more complicated and involves a series of scripts and sub-processes. Here is a boiled-down example script I want to execute:

#!/bin/bash
sub-process-that-fails
# store and echo returncode for debug purposes
rc=$?
echo $rc
exit $rc

Trying to make sure that the remotely executed shell was actually bash and not dash (as pointed out by @JeffSchaller), I tried calling the script like this:

~$ ssh -t -t host /bin/bash -x /srv/scripts/run.sh ; echo $?

Which led to this weird output:

+ sub-process-that-fails
+ rc=5
+ echo 5
5
+ exit 5
0

After hours of poking around, I noticed there was a trap 'kill 0' EXIT set in the .bashrc. This is done to kill all sub-processes in case bash is killed. bash's trace does not seem to display this trap's execution. I moved the trap into the wrapper script. Now I can see what actually is executed:

+ trap 'kill 0' EXIT
+ sub-process-that-fails
+ rc=5
5
+ echo 5
+ exit 5
+ kill 0
0

The remote shell exits with the last command's exit code. It's kill 0 and it exits with 0.