Why does the setuid bit work inconsistently?

What changed is that /bin/sh either became bash or stayed dash which got an additional flag -p mimicking bash's behaviour.

Bash requires the -p flag to not drop setuid privilege as explained in its man page:

If the shell is started with the effective user (group) id not equal to the real user (group) id, and the -p option is not supplied, no startup files are read, shell functions are not inherited from the environment, the SHELLOPTS, BASHOPTS, CDPATH, and GLOBIGNORE variables, if they appear in the environment, are ignored, and the effective user id is set to the real user id. If the -p option is supplied at invocation, the startup behavior is the same, but the effective user id is not reset.

Before, dash didn't care about this and allowed setuid execution (by doing nothing to prevent it). But Ubuntu 16.04's dash's manpage has an additional option described, similar to bash:

-p priv
Do not attempt to reset effective uid if it does not match uid. This is not set by default to help avoid incorrect usage by setuid root programs via system(3) or popen(3).

This option didn't exist in upstream (which might not be have been reactive to a proposed patch^*) nor Debian 9 but is present in Debian buster which got the patch since 2018.

NOTE: as explained by Stéphane Chazelas, it's too late to invoke "/bin/sh -p" in system() because system() runs anything given through /bin/sh and so the setuid is already dropped. derobert's answer explains how to handle this, in the code before system().

^* more details on history here and there.

Probably the shell is changing its effective user ID back to the real user ID as part of its startup for some reason or another. You could verify this by adding:

/* needs _GNU_SOURCE; non-Linux users see setregid/setreuid instead */
uid_t euid = geteuid(), egid = getegid();
setresgid(egid, egid, egid);
setresuid(euid, euid, euid);

before your system(). (Actually, even on Linux, you probably only need to set the real ones; the saved ones should be fine to leave alone. This is just brute force to debug. Depending on why you're set-id, you may of course need to save the real IDs somewhere as well.)

[Also, if this isn't just an exercise learning how setid works, then there are lot of security issues to worry about, especially when calling a shell. There are many environment variables, for example, that affect the shell's behavior. Prefer an already-existing approach like sudo if at all possible.]