How do I check whether my shell is running in a terminal?

isatty is a function for checking this, and the -t flag of the test command makes that accessible from a shell script:

-t file_descriptor

True if file descriptor number file_descriptor is open and is associated with a terminal. False if file_descriptor is not a valid file descriptor number, or if file descriptor number file_descriptor is not open, or if it is open but is not associated with a terminal.

You can check if FD 0 (standard input) is a TTY with:

test -t 0

You can do the same for FDs 1 and 2 to check the output and error streams, or all of them:

test -t 0 -a -t 1 -a -t 2

The command returns 0 (succeeds) if the descriptors are hooked up to a terminal, and is false otherwise.

test is also available as the [ command for a "bracket test":

 if [ -t 0 ] ; then ...

is an idiomatic way to write this conditional.

I imagine this is a duplicate, but I can’t find it. Use

[ -t 0 ]

and

[ -t 1 ]

to test respectively whether standard input and output are connected to a terminal. man test has the details.

Just an extra note on top of the fine answers that have already been given. Note that [ -t 0 ] tests that the file descriptor 0 is open one a file that is a device file with a tty line discipline (typically, that's done by checking that a harmless termio(s) ioctl() succeeds).

Also, that doesn't necessarily mean there's a terminal or terminal emulator (with a real user typing on a keyboard) on the other end (though in the immense majority of cases and probably most of the ones you care about, that's good enough an approximation).

tty and pty devices can also be used for data transfer or as an interprocess communication mechanism.

For instance, one could do:

(stty raw -echo; myscript) < /dev/ttyS0

To feed what's received over RS232 to myscript.

echo test | ssh -tt host myscript

would have myscript's stdin being a pty device (with sshd at the other end, and eventually (across the ssh connection) not a terminal, but a pipe fed by echo)

To further check that there is a terminal at the other end of that RS232 line or pty, you could also check that a $TERM variable is set and non-empty ([ -n "$TERM" ]) and send a Device Status Report escape sequence over that fd and check that you receive a response (in addition to the [ -t 0 ] and [ -n "$TERM" ]).

printf >&0 '\e[5n'

Is replied with a \e[0n by most terminals.

Now there are several problems with that, so I wouldn't recommend doing that except in the case where you want to check that because you want to run a visual TUI application (in which case, you'd be better off using libraries like ncurses, and instead of the DSR, you'd rather want to send a device identification escape sequence to query the type of terminal more precisely than via $TERM):

• Thankfully, in most cases where stdin is not a terminal, it will have been open in read-only mode which would cause that printf to fail, but in case stdin is a tty device open in read+write mode, that will have the side effect of sending that sequence to the other end. For instance in our ssh example above, that will actually send the sequence to a terminal (but the reply will not be coming on stdin)
• It's hard to read the reply reliably and portably. You'd need to change the tty line discipline temporarily and read one byte at a time. You'll also need to decide a timeout by which if the reply is not seen, you give up and decide there's no terminal. If you want to consider people dialing in over satellite connections, that means a long timeout.
• Reading from a terminal when in background would suspend your script with a SIGTTIN signal.