Why is $U(1)$ special when defining global charges?

The reason is confinement. Yang Mills theories with $SU(2)$ and $SU(3)$ gauge groups exhibit confinement, while for example $U(1)$ electrodynamics does not. Whether a theory is confining or not can be found out by studying the properties of Wilson loops.

The premise of this question is wrong: U(1) is not special. There are conserved charges associated to the global symmetry group $G$ of any gauge theory.

In the Standard Model, for example, we have the weak isospin charge, which is the conserved Noether charge of the electroweak $SU(2)$. (At low energies, this conservation is obscured by the Higgs mechanism, but at high enough energies, it becomes more plain.) There is also a conserved color charge associated to the $SU(3)$ color symmetry.

As Frederic Brunner correctly points out, in our world, the color force is confining, so the value of this conserved color charge in normal physics is always 'zero'. But this is not true above QCD's Hagedorn temperature, where the system deconfines. Nor is this entirely a theoretical issue: RHIC has produced collisions intense enough to reach this deconfined state.