Why an EnumSet or an EnumMap is likely to be more performant than their hashed counterparts?

EnumSet is backed by a bit array. Since the number of different items you can put in EnumSet is known in advance, we can simply reserve one bit for each enum value. You can imagine similar optimization for Set<Byte> or Set<Short>, but it's not feasible for Set<Integer> (you'd need 0.5 GiB of memory for 2^32 bits) or in general.

Thus basic operations like exists or add ar constant time (just like HashSet) but they simply need to examine or set one bit. No hashCode() computation. This is why EnumSet is faster. Also more complex operations like union or easily implemented using bit manipulation techniques.

In OpenJDK there are two implementations of EnumSet: RegularEnumSet capable of handling enum with up to 64 values in long and JumboEnumSet for bigger enums (using long[]). But it's just an implementation detail.

EnumMap works on similar principles, but it uses Object[] to store values while key (index) is implicitly inferred from Enum.ordinal().