Bash function manipulation explained

What do setresuid() and setresgid() do?

In order to better understand this example, we need to have a look at these two functions and the three parameters they take.

From this answer (and the credentials(7) man page):

On Linux, each process has the following user and group identifiers:

• Real user ID and real group ID. These IDs determine who owns the process. To summarize, this is who you are.

• Effective user ID and effective group ID. These IDs are used by the kernel to determine the permissions that the process will have when accessing shared resources such as message queues, shared memory, and semaphores. On most UNIX systems, these IDs also determine the permissions when accessing files. However, Linux uses the file system IDs for this task. To summarize, this is what you can do.

• Saved set-user-ID and saved set-group-ID. These IDs are used in set-user-ID and set-group-ID programs to save a copy of the corresponding effective IDs that were set when the program was executed. A set-user-ID program can assume and drop privileges by switching its effective user ID back and forth between the values in its real user ID and saved set-user-ID. To summarize, this is who you were before.

Further, it's important to understand the SetUID-Bit. What this does is it runs an executable program with the rights of the owner, not the one who calls it.

A security nightmare

Your question specifically mentions that nightmare is owned by root and has the SetUID-Bit set. What this means is that any user who is allowed to execute nightmare essentially executes it as root. This means that all child processes of nightmare - such as they are spawned when calling system() - are also executed as root.

But really now, what does setresuid() do?

In this context? Nothing. Since the user is already executing things as root, it doesn't affect the further exploitation in any way.

Could you use /dev/tty?

This question does not make much sense in this context. You have to look at the relevant code snippet to see where the string /dev/tty is coming from.

if (open("/dev/tty", O_RDWR) != -1) {
        fire();
        rax = sub_4008d0();
}

This means the program is trying to open /dev/tty for reading and writing and if that succeeds, calls fire().

By naming a bash function /dev/tty(), you do not get the desired results, as open() requires a path to a file as its first parameter. It's completely independent of bash and has no concept of bash functions.

What about puts()?

Similarly, this doesn't make much sense. strings isn't a magic and doesn't understand context. All it does is look for series of bytes that are printable ASCII characters.

puts() is just a C-function which outputs a string.

What about /usr/bin/aafire?

While I didn't personally try, it seems as if practically /usr/bin/aafire could have been used instead of /usr/bin/sl -al. If this is not correct, please comment and I will edit this part.

Would it still work if a relative path would be used?

Yes, actually even better! system() internally calls sh -c, which has to resolve the $PATH variable to find sl. If the path variable was modified to include . or any other user-writable directory with higher priority than /usr/bin, then I could just symlink ./sl to /bin/bash.

Does it matter what kind of binary/ELF the root SUID file is?

Yes, because the SetUID-Bit gets ignored on shell scripts. It must be a native executable.