SetUID bit not working in Ubuntu?

For the compiled executable, from man 2 chown:

When the owner or group  of  an  executable  file  are  changed  by  an
unprivileged user the S_ISUID and S_ISGID mode bits are cleared.  POSIX
does not specify whether this also should happen  when  root  does  the
chown();  the Linux behavior depends on the kernel version.

Reversing the chown and chmod order works for me:

$ sudo chmod 4770 foo
$ sudo chown root:root foo
$ stat foo
  File: 'foo'
  Size: 8712        Blocks: 24         IO Block: 4096   regular file
Device: 801h/2049d  Inode: 967977      Links: 1
Access: (0770/-rwxrwx---)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-04-18 15:15:15.074425000 +0900
Modify: 2017-04-18 15:15:15.074425000 +0900
Change: 2017-04-18 15:15:33.683725000 +0900
 Birth: -
$ sudo chmod 4777 foo
$ ./foo
1000,0

In your first case, it's Bash that doesn't like being run as setuid.

If Bash is started with the effective user (group) id not equal to the real user (group) id,..., and the effective user id is set to the real user id.

See: Bash's manual on startup files, also Setuid bit seems to have no effect on bash .

In the second case, it's the order of chmod and chown that matters, as muru already answered. Changing the owner resets the setuid bit.

It could also be that the filesystem containing the test executable was mounted with the nosuid option; I have heard that newer distributions will do this by default for /tmp, and there are good arguments for applying it to /home as well. nosuid causes the kernel to ignore the setuid and setgid bits on all executables within the filesystem. (The unrelated thing that happens when you make a directory setgid is unaffected.)