Color Logic Algorithm

Most answers for this question will suggest calculating the distance between two colors when mapping the RGB values into a 3D space. The problem with this technique is that two colors with similar hues, but different saturation or brightness levels may map further away from each other in 3D RGB space than two colors with different hues, but very similar saturation and brightness levels. In other words, a blue and a green may be closer in 3D RGB space than two shades of a Red. In this application, ensuring team colors differ, hue differences should weigh much more heavily than brightness and saturation.

So I would convert the color mapping from RGB to hue, saturation, and brightness levels, and then check just the hue values for sufficient distance.

Wikipedia has an explanation for converting RGB to HSV. LiteratePrograms has some sample code.

Here is a theoretical explanation

And the algo in C:

typedef struct {
    unsigned char r, g, b;
} RGB;

double ColourDistance(RGB e1, RGB e2)
{
    long rmean = ( (long)e1.r + (long)e2.r ) / 2;
    long r = (long)e1.r - (long)e2.r;
    long g = (long)e1.g - (long)e2.g;
    long b = (long)e1.b - (long)e2.b;
    return sqrt((((512+rmean)*r*r)>>8) + 4*g*g + (((767-rmean)*b*b)>>8));
}

Here is pgras' algorithm in Java:

public double ColourDistance(Color c1, Color c2)
{
    double rmean = ( c1.getRed() + c2.getRed() )/2;
    int r = c1.getRed() - c2.getRed();
    int g = c1.getGreen() - c2.getGreen();
    int b = c1.getBlue() - c2.getBlue();
    double weightR = 2 + rmean/256;
    double weightG = 4.0;
    double weightB = 2 + (255-rmean)/256;
    return Math.sqrt(weightR*r*r + weightG*g*g + weightB*b*b);
}