"System" color for warnings (red)

"Using only custom colors and thus avoiding the problem is unacceptable behavior. Users expect programs to respect their settings."

But where does the user specify a color for warnings?

If you want an eye-catchy color dependent to system colors, you can take the color of selected item and saturate it a bit, or make it more red (e.g. make it's RGB red component to be 0xFF, and it will depend on user settings through the means of the other two components).

The closest that I have come to having this kind of problem was in a project were we implemented some color manipulation to ensure that elements in a CAD program were not "accidentally" hidden by the user changing the background color. For example, if the background color was black and there were some white elements in the CAD file, everything was good. The white elements are obviously visible against the black background. However, if the user changed to a white background, the white elements were no longer visible. We implemented some logic that manipulated (pretty heavy handedly) the element color (during draw) to ensure that it was visible. Typically, this logic only kicked in if the element color was exactly equal to the background color. Later, we extended the check to change the element color if it was "near" the background color. This was reasonbly successful in that no elements were accidentally hidden. However, the resulting colors were sometimes pretty hideous. That is just some background for when I have had to deal with colors programmatically.

Here is a post that describes how to pick a good text color for a given background color. This is probably of limited use because don't just want a color that is visible, you would probably prefer to use Red (or maybe some other color) and only change it if it is not easily readable (or not obviously indicative of a problem).

Here is an article that describes how to modify a color by manipulating brightness and saturation rather than fooling around with the RGB values.

You could make the problem items in the list have a White background and then use Red text against the background. I have not done much UI programming, so I can't tell you how easy or hard this is in practice (to change the background color of a single item in a ListBox).

Using the first link that I posted as a starting point, you could test Red to see if it is "easily readable" by some color math. How far is Red from the background color? If it is "too close", give up and get a better contrasting color (which is NOT the same as the color that you would have drawn with if no error).

You could define a "warning" color (or colors) for each of several standard Windows color schemes (by switching to each of these schemes, defining a warning color that satisfies you for a given text color and background color). If the user happens to be using one of those schemes (or at least if the background color and text color match one of the background/text color combinations that you defined), just use one of your predefined warning colors. If the user is not, try to find a color that might work by comparing the user's background color and text color against your predefined colors and use the one that is the best match. If no good matches (i.e. all possible predefined warning colors are not "easily readable" given some criteria), then try computing a color from scratch that is visible but is NOT the same as the text color you are replacing.

You could define several discrete warning colors that you feel are easily readable against certain color ranges. Maybe you define 16 colors. You might choose to use Color1 if the background color falls within ColorRange1, Color2 for BackgroundColor2, etc. I don't know how many colors you would have to define to ensure that you would have a "good" choice for all possible color combinations. By defining the warning colors "by hand", you might have a better chance at achieving relatively "aesthetically pleasing" colors than if you tried to generate a warning color at runtime.

If you are able to predefine a single warning color for a given background color (say unselected), you might be able to interpolate corresponding warning colors for the selected focused and unfocused cases by using the ratio between the background colors - e.g. unselected vs selected focused - and applying that ratio (or the opposite) to the "base" warning color.

I don't claim that many (or any) of these suggestions are particularly good. They are just some ideas that I had as I read and thought about your question.

Possibly a new approach is worth consideration. Using colored icons such as the windows event log. Keep all text consistent and use colored icons to visually distinguish different types of data.