Comparison of IPEndPoint objects not working

Both IPEndPoint and IPAddress don't implement the == operator. By default, the == operator compares if the two objects are the same reference, not if they represent the same value.

Use the IPAddress.Equals / IPEndPoint.Equals methods instead.

IPAddress does not define an overload for == however it does override Object.Equals, so your equality check should be:

public static bool AreEqual(IPEndpoint e1, IPEndpoint e2)
{
    return e1.Port == e2.Port && e1.Address.Equals(e2.Address);
}

If you are using linq, it is probably a good idea to create your own IEqualityComparer<IPEndpoint> to encapsulate this, since various linq methods take one to compare items.