How does a program decide whether or not to have coloured output?

Most such programs only output colour codes to a terminal by default; they check to see if their output is a TTY, using isatty(3). There are usually options to override this behaviour: disable colours in all cases, or enable colours in all cases. For GNU grep for example, --color=never disables colours and --color=always enables them.

In a shell you can perform the same test using the -t test operator: [ -t 1 ] will succeed only if the standard output is a terminal.

Is there some environment variable?

Yes. It is the TERM environment variable. This is because there are several things that are used as part of the decision process.

It's difficult to generalize here, because not all programs agree on a single decision flowchart. In fact GNU grep, mentioned in M. Kitt's answer, is a good example of an outlier that uses a somewhat unusual decision process with unexpected outcomes. In very general terms, therefore:

• The standard output must be a terminal device, as determined by isatty().
• The program must be able to look up the record for the terminal type in the termcap/terminfo database.
• So therefore there must be a terminal type to look up. The TERM environment variable must exist and its value must match a database record.
• There must therefore be a terminfo/termcap database. On some implementations of the subsystem, the location of the termcap database can be specified using a TERMCAP environment variable. So on some implementations there is a second environment variable.
• The termcap/terminfo record must state that the terminal type supports colours. There's a max_colors field in terminfo. It's not set for terminal types that do not actually have colour capabilities. Indeed, there's a terminfo convention that for every colourable terminal type there's another record with -m or -mono appended to the name that states no colour capability.
• The termcap/terminfo record must provide the way for the program to change colours. There are set_a_foreground and set_a_background fields in terminfo.

It's a bit more complex than just checking isatty(). It is made further complicated by several things:

• Some applications add command-line options or configuration flags that override the isatty() check, so that the program always or never assumes that it has a (colourable) terminal as its output. For examples:
  • GNU ls has the --color command-line option.
  • BSD ls looks at the CLICOLOR (its absence meaning never) and CLICOLOR_FORCE (its presence meaning always) environment variables, and also sports the -G command-line option.
• Some applications don't use termcap/terminfo and have hardwired responses to the value of TERM.
• Not all terminals use ECMA-48 or ISO 8613-6 SGR sequences, which are slightly mis-named "ANSI escape sequences", for changing colours. The termcap/terminfo mechanism is in fact designed to insulate applications from direct knowledge of the exact control sequences. (Moreover, there's an argument to be had that no-one uses ISO 8613-6 SGR sequences, because everyone agrees on the bug of using semi-colon as the delimiter for RGB colour SGR sequences. The standard actually specifies colon.)

As mentioned, GNU grep actually exhibits some of these additional complexities. It doesn't consult termcap/terminfo, hardwires the control sequences to emit, and hardwires a response to the TERM environment variable.

The Linux/Unix port of it has this code, which enables colourization only when the TERM environment variable exists and its value doesn't match the hardwired name dumb:

int
should_colorize (void)
{
  char const *t = getenv ("TERM");
  return t && strcmp (t, "dumb") != 0;
}

So even if your TERM is xterm-mono, GNU grep will decide to emit colours, even though other programs such as vim will not.

The Win32 port of it has this code, which enables colourization either when the TERM environment variable does not exist or when it exists and its value doesn't match the hardwired name dumb:

int
should_colorize (void)
{
  char const *t = getenv ("TERM");
  return ! (t && strcmp (t, "dumb") == 0);
}

GNU grep's problems with colour

GNU grep's colourization is actually notorious. Because it doesn't actually do a proper job of constructing terminal output, but rather just blams in a few hardwired control sequences at various points in its output in the vain hope that that is good enough, it actually displays incorrect output in certain circumstances.

These circumstances are where it has to colourize something that is at the right hand margin of the terminal. Programs that do terminal output properly have to account for automatic right margins. In addition to the slight possibility that the terminal might not have them (viz the auto_right_margin field in terminfo), the behaviour of terminals that do have automatic right margins often follows the DEC VT precedent of pending line wrap. GNU grep doesn't account for this, naïvely expecting immediate line wrap, and its coloured output goes wrong.

Coloured output is not a simple thing.

Further reading

• Thomas E. Dickey (2016). "grep --color does not show the right output". xterm FAQ. Invisible Island.
• Jonathan de Boyne Pollard (2016). Italics and colour in manual pages on a nosh user-space virtual terminal. The nosh package.