@UniqueConstraint and @Column(unique = true) in hibernate annotation

As said before, @Column(unique = true) is a shortcut to UniqueConstraint when it is only a single field.

From the example you gave, there is a huge difference between both.

@Column(unique = true)
@ManyToOne(optional = false, fetch = FetchType.EAGER)
private ProductSerialMask mask;

@Column(unique = true)
@ManyToOne(optional = false, fetch = FetchType.EAGER)
private Group group;

This code implies that both mask and group have to be unique, but separately. That means that if, for example, you have a record with a mask.id = 1 and tries to insert another record with mask.id = 1, you'll get an error, because that column should have unique values. The same aplies for group.

On the other hand,

@Table(
   name = "product_serial_group_mask", 
   uniqueConstraints = {@UniqueConstraint(columnNames = {"mask", "group"})}
)

Implies that the values of mask + group combined should be unique. That means you can have, for example, a record with mask.id = 1 and group.id = 1, and if you try to insert another record with mask.id = 1 and group.id = 2, it'll be inserted successfully, whereas in the first case it wouldn't.

If you'd like to have both mask and group to be unique separately and to that at class level, you'd have to write the code as following:

@Table(
        name = "product_serial_group_mask",
        uniqueConstraints = {
                @UniqueConstraint(columnNames = "mask"),
                @UniqueConstraint(columnNames = "group")
        }
)

This has the same effect as the first code block.

From the Java EE documentation:

public abstract boolean unique

(Optional) Whether the property is a unique key. This is a shortcut for the UniqueConstraint annotation at the table level and is useful for when the unique key constraint is only a single field. This constraint applies in addition to any constraint entailed by primary key mapping and to constraints specified at the table level.

See doc

In addition to Boaz's answer ....

@UniqueConstraint allows you to name the constraint, while @Column(unique = true) generates a random name (e.g. UK_3u5h7y36qqa13y3mauc5xxayq).

Sometimes it can be helpful to know what table a constraint is associated with. E.g.:

@Table(
   name = "product_serial_group_mask", 
   uniqueConstraints = {
      @UniqueConstraint(
          columnNames = {"mask", "group"},
          name="uk_product_serial_group_mask"
      )
   }
)