Algorithm to check similarity of colors

I would recommend using CIE94 (DeltaE-1994), it's said to be a decent representation of the human color perception. I've used it quite a bit in my computer-vision related applications, and I am rather happy with the result.

It's however rather computational expensive to perform such a comparison:

1. RGB to XYZ for both colors
2. XYZ to LAB for both colors
3. Diff = DeltaE94(LABColor1,LABColor2)

Formulas (pseudocode):

• Color-conversion formulas: http://www.easyrgb.com/?X=MATH
• Delta-E formulas: http://www.easyrgb.com/index.php?X=DELT

There's an excellent write up on the subject of colour distances here: http://www.compuphase.com/cmetric.htm

In case that resource disappears the author's conclusion is that the best low-cost approximation to the distance between two RGB colours can be achieved using this formula (in C code).

typedef struct {
   unsigned char r, g, b;
} RGB;

double ColourDistance(RGB e1, RGB e2)
{
  long rmean = ( (long)e1.r + (long)e2.r ) / 2;
  long r = (long)e1.r - (long)e2.r;
  long g = (long)e1.g - (long)e2.g;
  long b = (long)e1.b - (long)e2.b;
  return sqrt((((512+rmean)*r*r)>>8) + 4*g*g + (((767-rmean)*b*b)>>8));
}

RGB distance in the euclidean space is not very similar to "average human perception".

You can use YUV color space, it takes into account this factor :

 |  Y' |     |  0.299     0.587    0.114   | | R |
 |  U  |  =  | -0.14713  -0.28886  0.436   | | G |
 |  V  |     |  0.615    -0.51499 -0.10001 | | B |

You can also use the CIE color space for this purpose.

EDIT:

I shall mention that YUV color space is an inexpensive approximation that can be computed via simple formulas. But it is not perceptually uniform. Perceptually uniform means that a change of the same amount in a color value should produce a change of about the same visual importance. If you need a more precise and rigourous metric you must definitely consider CIELAB color space or an another perceptually uniform space (even if there are no simple formulas for conversion).

Human perception is weaker in chroma than intensity.

For example, in commercial video, the YCbCr/YPbPr color spaces (also called Y'UV) reduces the resolution of the chroma info but preserves the luma (Y). In digital video compression such as 4:2:0 and 4:2:2 reduces the chroma bitrate due to relatively weaker perception.

I believe that you can calculate a distance function giving higher priority over luma (Y) and less priority over chroma.

Also, under low intensity, human vision is practically black-and-white. Therefore, the priority function is non-linear in that for low luma (Y) you put less and less weight on chroma.

More scientific formulas: http://en.wikipedia.org/wiki/Color_difference