Why can't compact symplectic groups $Sp(n)\equiv USp(2n)\equiv U(2n)\cap Sp(2n,\mathbb{C})$ be gauge groups in Yang-Mills theory?

The structure of standard model $SU(3)\times SU(2)\times U(1)$ is chiral which basically tells you the necessity of chiral fermions. If left-handed fermions transform under a representation $R$ of the symmetry group then due to charge-conjugation relating left-handed and right-handed fermions as $$\psi_{Right}=C(\bar{\psi^C})^T_{Left}$$ and so, right handed fermions should transform under the complex conjugate representation $R^*$. If $R$ is real or pseudoreal, then left-handed and right-handed fermions transform in same representation of the group and the theory is known to be a vector like theory (QCD). To have chiral structure of fermions, one has to have $R \ne R^*$ which demands $R$ to be complex.

Even though QCD is vector like and $2=2^*$, the whole standard model is chiral as can be seen by writing $R$ for left-handed fermions as, $$R=(3,2)_{\frac{1}{6}}+(3^*,1)_{\frac{-2}{3}}+(3^*,1)_{\frac{1}{3}}+(1,2)_{\frac{-1}{2}}+(1,1)_{1}$$ the complex conjugate to which is not same as $R$.

It is known that $USp(2n)$ for $n>2$ admits real and pseudo-real representations (Weinberg Vol. 2, chapter 22) and $USp(4)$ is not big enough to contain standard model.

Moreover using a $Sp(n)$ like gauge group demands even number of fermion multiplets otherwise the gauge theory will show a non-perturbative anomaly$^1$ involving fourth homotopy group of $Sp(n)$.

1- Ed Witten, nucl. phys. B223 (1983),433-444.

Well the answer of your question is not so trivial, I guess. Here is my try. I want to give a glimpse why a symplectic group is not a good choice for model building from a phenomenological point of view.

Now look at the symplectic group closely.

• $Sp(1)$ is isomorphic to $SU(2)$
• $Sp(4)$ is isomorphic to $SO(5)$ (which is due to a deeper connection between $SO(2n+1)$ and $Sp(2n)$)

The standard model gauge group is $SU(3)_{C}\times SU(2)_{L}\times U(1)_{Y}$. If we have a closer look then $SU(3)$ has complex representation (fundamental and anti-fundamental representation does not mix with each other), $SU(2)$ has pseudo real representation. That simply says particles belongs to standard model (also belongs to real world!) gauge group has complex representations.

Most strikingly, the symplectic group does not have complex representations. For example $USp(2n)$ with $n\geq 3$ has only real and pseudo-real representations. So, any gauge theory which is can not accommodate complex representation is not a good choice for model building.

For a more rigorous perspective one can consult with, Group theory for unified model building by Slansky.