Collection that allows only unique items in .NET?

How about just an extension method on HashSet?

public static void AddOrThrow<T>(this HashSet<T> hash, T item)
{
    if (!hash.Add(item))
        throw new ValueExistingException();
}

HashSet<T> is what you're looking for. From MSDN (emphasis added):

The HashSet<T> class provides high-performance set operations. A set is a collection that contains no duplicate elements, and whose elements are in no particular order.

Note that the HashSet<T>.Add(T item) method returns a bool -- true if the item was added to the collection; false if the item was already present.