common environment variable to set dark or light terminal background

There is no such convention. Furthermore, an environment variable is not a good way to report information about a terminal, because the value can get stale if a program starts another terminal emulator which doesn't update this variable, or if a program connects to multiple terminals.

(The TERM environment variable doesn't suffer from these problems because it's universal: every terminal emulator sets it and every program is aware of it. The problems only arise when a variable is partially supported.)

The right way to obtain the information would be to query the terminal. In the unix world, this is done by writing an escape sequence which the terminal interprets as “send back some data that answers my query”. As Thomas Dickey explains, xterm has such a control sequence, OSC 11 ; ? BEL (set text parameters, parameter 11 = text background color, value ? means query instead of set). Write \e]11;?\a to the terminal (where \e is the escape character (^[) and \a is the bell character (^G)), and xterm replies with a string like \e]11;rgb:0000/0000/0000\a (that's a black background). Unfortunately, few other terminal emulators support this escape sequence.

Rxvt sets the environment variable COLORFGBG to a string like 7;0 where 7 is the foreground color (7 is light gray) and 0 is the background color (black). Konsole also supports this.

Emacs attempts to detect whether the terminal has a light or dark background, in order to set the background-mode terminal parameter. As of Emacs 24.5, there are three methods to set the background mode automatically:

• On xterm, Emacs uses the OSC 11 escape sequence as explained above.
• On rxvt, Emacs uses the COLORFGBG environment variable as explained above.
• On DOS and Windows consoles, Emacs uses OS-specific interfaces to obtain information about the terminal; these interfaces play the same role as the OSC 11 escape sequence.

This leaves out many terminals, however there is some progress: the vte library, which powers many terminal emulators such as gnome-terminal, guake, terminator, xfce4-terminal, …, implements OSC 11 reporting like xterm since version 0.35.2. You can detect VTE-based terminals by checking the environment variable VTE_VERSION; the value is a number, you want 3502 and above.

If you want to standardize on a way report the information to applications, then support on the terminal side might not matter: after all you know whether you prefer light or dark backgrounds. Then you might as well align with rxvt and use COLORFGBG, since it's the only interface that somebody is already using and that you can adopt independently of any terminal support. The COLORFGBG interface is limited: it was designed for a world with only 16 colors, and everybody agreeing on a mapping from color numbers to colors (at least approximately, exact hues differ). Konsole supports more than 16 colors, but it uses an approximation when reporting COLORFGBG: it approximates the foreground and background colors by one of the 16 standard colors. If all you care about is light vs dark, that's not a problem, just set COLORFGBG to 15;0 for light text on a dark background or 0;15 for dark text on a light background.

There is none. You are free to make your own convention.

A few programs (such as xterm) can provide an application with currently-set foreground and background colors (see "dynamic colors"). But aside from applications such as xtermcontrol, very few make use of it. Emacs queries the current background color to set its background-mode terminal parameter, which determines whether to apply the “dark” or “light” version of color theme. On the other hand, Vim, which may use the xterm control sequences for the number of colors and the strings sent by special keys (see "Request Termcap/Terminfo String" in XTerm Control Sequences), does not detect the sense of foreground/background colors.

Based on the well-thought-out answers by Thomas Dickey and Gilles and what to do for OSX in the comment to this answer, I have culled together this script in github.

The crux is to use xterm-compatible color querying, and based on the RGB color values of background versus the foreground. If that doesn't work, then try to interpret COLORFGBG and failing that use preset defaults for various terminals. For example, xterm is a light background by default.

The code for the preset defaults and RGB dark/light interpolation was derived from GNU emacs code (as suggested by Gilles).

I haven't added any MS Windows specific code, but folks should feel free to submit pull requests to handle that or to add any other corrections.