how to write write a set for unordered pair in Java

Well, it depends on the hashCode() and equals() method of your Pair class. They need to ignore order.

Set itself is a good example of a class which ignores order for equality--you can look at the code of AbstractSet. If the order of the pair doesn't matter even outside of equality comparison, you can just store HashSets (each with two elements) in your set. It would be best to wrap it in a datatype:

 public class UnorderedPair<T> {
     private final Set<T> set;

     public UnorderedPair(T a, T b) {
          set = new HashSet<T>();
          set.add(a);
          set.add(b);
     }

     public boolean equals(Object b) {
         //...delegate to set
     }

     public int hashCode() {
         return set.hashCode();
     }
}

As none of the answers mentions this approach, I'd like to add this approach:

public class UnorderedPair<T> {
    final T first, second;

    public UnorderedPair(T first, T second) {
        this.first = first;
        this.second = second;
    }

    @Override
    public boolean equals(Object o) {
        if (!(o instanceof UnorderedPair))
            return false;
        UnorderedPair<T> up = (UnorderedPair<T>) o;
        return (up.first == this.first && up.second == this.second) ||
                (up.first == this.second && up.second == this.first);
    }

    @Override
    public int hashCode() {
        int hashFirst = first.hashCode();
        int hashSecond = second.hashCode();
        int maxHash = Math.max(hashFirst, hashSecond);
        int minHash = Math.min(hashFirst, hashSecond);
        // return Objects.hash(minHash, maxHash);
        // the above line also works but I tend to avoid this to avoid unnecessary auto-boxing
        return minHash * 31 + maxHash;
    }
}

Note that the general contract of the hashCode() and equals() should be followed:

• If you are overriding either hashCode() or equals() you usually have to also override the other method.
• If the equals return true for 2 objects, then the hashCode() method must return the same int for both the objects.
• If the hashCode() returns same int for 2 objects, then it's not necessarily true that the equals() method return true.

The above implementation of equals() and hashCode() method ensures this.

Define a class Pair whose equals and hashCode methods are based on both a and b in the way that the order of a and b does not matter and use a HashSet.

final class Pair<T> {
  private final Set<T> elements = new LinkedHashSet<T>();
  Pair(T a, T b) {
    elements.add(a);
    if (!elements.add(b))
      throw new IllegalArgumentException();
  }
  @Override
  public int hashCode() {
    return elements.hashCode();
  }
  @Override
  public boolean equals(Object obj) {
    if (obj == this)
      return true;
    if (!(obj instanceof Pair<?>))
      return false;
    return elements.equals(((Pair<?>) obj).elements);
  }
}