How do I determine if a program is running when I start a shell and start that program if it is not already running?

Use pgrep like this:

if ! pgrep -u $USER top >/dev/null 2>&1; then
    exec top
fi

Explanation:

• The pgrep utility exits with a zero exit code if the command can be found running. Therefore, negate the test.
• Also restrict it to the current user with -u $USER, so that it doesn't for some reason pick up someone else's processes. $USER is a standard shell variable. If that for some reason isn't defined, try using $LOGNAME instead, and if that fails, use $( id -n -u ) (but at that point you have to ask yourself what's going on, because $LOGNAME is also a standard shell variable).
• We don't need the output from pgrep, so redirect the standard output and error streams to /dev/null.

If top (or whatever) isn't found to be running, exec it to replace the shell, as per the OP's question.

This should be working in all sh-compatible shells.

EDIT: The pgrep implementation on BSD systems (OS X, OpenBSD, ...) has a -q option that makes it discard any output, but the pgrep on Linux apparently doesn't have this flag. On BSD systems, use this rather than the slightly ugly redirect to /dev/null.

If the program is always started from the shell and you have full control over it (ie. it is not hardcoded inside 3rd party program), then you can use some locking mechanism like lockfile or flock. For example by adding

flock -n /tmp/top -c top

to the .bashrc/.zshrc you will ensure that only first instance of the shell will run the top command (unless you kill it, then another one which will be started).

You may want to create a function in order to be able to start the command also by hand (not only when shell initialize):

mytop () { flock -n /tmp/$0 -c top; }

And just run mytop whenever you want.

Agreeing that pgrep is the way to go, there are some pitfalls. You need not use the -q option, since you can redirect the output of pgrep to eliminate that:

pgrep -U $USER top >/dev/null || exec top

The pitfalls lie in the fact that it is not a standard utility, so you have to keep in mind that it may not be portable. Here are a few manpage links, to help:

• FreeBSD
• Linux
• Solaris

The USER environment variable is more well-established, but does not appear to be in POSIX's list for the shell (see 2.5.3 Shell Variables). Rather, it notes that USER is one of many variables known to be in use (see note in 8.1 Environment Variable Definition).

This runs top as yourself, of course. If you chose to make it run as root, then you could change the $USER to root and then run top using either sudo (preferred) or su. Neither is in POSIX (see list). The latter (su) is available everywhere on Unix-like systems, but sudo is generally preferred. So you would have

pgrep -U root top >/dev/null || exec sudo top

Unless you have configured sudo to not require a password, you will be prompted for a password when your shell starts top.