Unitary groups over number fields

If $E$ is not CM, then the action of complex conjugation on $E$ depends on how it is embedded into $\mathbb{C}$. In particular, it could have different real subfields depending on which embedding you are using. When $E$ is CM, so that it is a totally imaginary quadratic extension of the totally real subfield $F$, then it has a unique complex conjugation that commutes with all automorphisms of $E$, such that $F$ contains precisely the elements that are fixed by complex conjugation in all embeddings. This lets you talk about unitarity and hermiticity for the abstract field $E$, and not just for some particular embedding.

Or, to put it another way, there can be more than one way to consistently define a complex conjugation on your field, unless it is CM, in which case there is a unique way to define it.

Just to add to Jon Yard's answer: when one defines a unitary group as described in the OP, the group that one gets after extending scalars from $F$ to $F_v = \mathbb R$ for each archimedean place $v$ of $F$ is indeed a unitary group in the usual sense, i.e. the set of complex matrices preserving some non-degenerate Hermitian form.

If one were to apply the defintion in a more general setting, say to a quadratic extension $E$ over $F$ which is not a CM extension of a totally real field, the groups at infinity could end up being $GL_n(\mathbb R)$ or $GL_n(\mathbb C)$, which are not unitary groups.