"invalid comparator" : error when overloading the "<" operator

Your implementation is not correct.

bool outlierScore::operator<(const outlierScore& other) const {
    return (score < other.score) ||
           (score == other.score && coreDistance < other.coreDistance) ||
           (score == other.score && coreDistance == other.coreDistance && id < other.id);
}

Or

bool outlierScore::operator<(const outlierScore& other) const {
    return std::tie(score, coreDistance, id) < std::tie(other.score, other.coreDistance, other.id);
}

Other than the method is not const, this operator does not satisfy strict weak ordering.

bool outlierScore::operator<(const outlierScore& other) {
    if (score < other.score)
        return score < other.score;
    else if (coreDistance < other.coreDistance)
        return coreDistance < other.coreDistance;
    else if (id < other.id)
        return id < other.id;
    else
        return false;
}

For example, if all fields are are integers.

a.score = 0 ; a.coreDistance = 1
b.score = 1 ; b.coreDistance = 0

For these values, the following should be false, but a true is returned:

a < b && b < a

You should also check equality:

bool outlierScore::operator<(const outlierScore& other) {
    if (score != other.score)
        return score < other.score;
    else if (coreDistance != other.coreDistance)
        return coreDistance < other.coreDistance;
    else 
        return id < other.id;
}