cryptsetup - how does it print prompt bypassing stdout/stdin redirection?

If you strace it, you will probably see that it uses /dev/tty directly.

...
open("/dev/tty", O_RDWR)                = 6
ioctl(6, TCGETS, {B38400 opost isig icanon echo ...}) = 0
write(6, "Enter passphrase for .......: ", 30) = 30
ioctl(6, SNDCTL_TMR_CONTINUE or TCSETSF, {B38400 opost isig icanon -echo ...}) = 0
...

In the source code (utils_crypt.c):

static int interactive_pass(const char *prompt, char *pass, size_t maxlen,
                long timeout)
{
[...]
        /* Read and write to /dev/tty if available */
        infd = open("/dev/tty", O_RDWR);
        if (infd == -1) {
                infd = STDIN_FILENO;
                outfd = STDERR_FILENO;
        } else
                outfd = infd;

        if (tcgetattr(infd, &orig))
                goto out_err;

So it tests for /dev/tty by opening it, and if that works it uses that. If it fails, it falls back to regular stdin, stdout, and then you wouldn't see the prompt anymore.

As for the /dev/tty, it's the terminal of the process, for more detail see man 4 tty.

Presumably, It writes directly to /dev/tty (at any rate, you can get the same behavior)

#!/bin/bash

# set up the new file descriptor
exec 3> /dev/tty

# test
echo "Stdout"
echo "Stderr" >&2
echo "Directly to tty" >&3

alternatively, you can simply do:

echo "Directly to tty" >/dev/tty

---

$ ./foo.sh >/dev/null 2>/dev/null
Directly to tty 

read still works if you do this.

---

The exec is required to keep the redirection for the duration of the present shell.

A redirection on a simple command:

$ echo yes       3>file

lasts while the command is being executed. Once the command (echo in this example) ends, the shell removes the redirection and reverts back to the "present shell" execution environment.

A:

$ 3>file

is still a "simple command" where the command executed is "none", the redirection will not live for long.

Instead in:

$ exec 3>file

the exec replaces the "present shell" with a new one which includes the redirection. That makes the redirection stay alive for as long as the "present shell" exists. That could be undone (well, actually close fd 3) with:

$ exec 3>&-