Why is the Yang-Mills gauge group assumed compact and semi-simple?

As Lubos Motl and twistor59 explain, a necessary condition for unitarity is that the Yang Mills (YM) gauge group $G$ with corresponding Lie algebra $g$ should be real and have a positive (semi)definite associative/invariant bilinear form $\kappa: g\times g \to \mathbb{R}$, cf. the kinetic part of the Yang Mills action. The bilinear form $\kappa$ is often chosen to be (proportional to) the Killing form, but that need not be the case.

If $\kappa$ is degenerate, this will induce additional zeromodes/gauge-symmetries, which will have to be gauge-fixed, thereby effectively diminishing the gauge group $G$ to a smaller subgroup, where the corresponding (restriction of) $\kappa$ is non-degenerate.

When $G$ is semi-simple, the corresponding Killing form is non-degenerate. But $G$ does not have to be semi-simple. Recall e.g. that $U(1)$ by definition is not a simple Lie group. Its Killing form is identically zero. Nevertheless, we have the following YM-type theories:

1. QED with $G=U(1)$.

2. the Glashow-Weinberg-Salam model for electroweak interaction with $G=U(1)\times SU(2)$.

I recommend that you to read the chapter 15.2 in "The Quantum Theory of Fields" Volume 2 by Steven Weinberg, he answers precisely your question.

Here a short summary
In a gauge theory with algebra generators satisfying $$ [t_\alpha,t_\beta]=iC^\gamma_{\alpha\beta}t_\gamma $$ it can be checked that the field strength tensor $F^\beta_{\mu\nu}$ transforms as follows $$ \delta F^\beta_{\mu\nu}=i\epsilon^\alpha C^\beta_{\gamma\alpha} F^\gamma_{\mu\nu} $$ We want to construct Lagrangians. A free-particle kinetic term must be a quadratic combination of $F^\beta_{\mu\nu}$ and Lorentz invariance and parity conservation restrict its form to $$ \mathcal{L}=-\frac{1}{4}g_{\alpha\beta}F^\alpha_{\mu\nu}F^{\beta\mu\nu} $$ where $g_{\alpha\beta}$ may be taken symmetric and must be taken real for the Lagrange density to be real as well. The Lagrangian above must be gauge-invariant thus it must satisfy $$ \delta\mathcal{L}=\epsilon^\delta g_{\alpha\beta}F^\alpha_{\mu\nu}C^\beta_{\gamma\delta}F^{\gamma\mu\nu}=0 $$ for all $\epsilon^\delta$. In order not to impose any functional restrictions for the field strengths $F$ the matrix $g_{\alpha\beta}$ must satisfy the following condition $$ g_{\alpha\beta}C^\beta_{\gamma\delta}=-g_{\gamma\beta}C^\beta_{\alpha\delta} $$ In short, the product $g_{\alpha\beta}C^\beta_{\gamma\delta}$ is anti-symmetric in $\alpha$ and $\gamma$.
Furthermor the rules of canonical quantization and the positivity properties of the quantum mechanical scalar product require that the matrix $g_{\alpha\beta}$ must be positive-definite. Finally one can prove that the following statements are equivalent

1. There exists a real symmetric positive-definite matrix $g_{\alpha\beta}$ that satisfies the invariance condition above.
2. There is a basis for the Lie algebra for which the structure constants $C^\alpha_{\beta\gamma}$ are anti-symmetric not only in the lower indices $\beta$ and $\gamma$ but in all three indices $\alpha$, $\beta$ and $\gamma$.
3. The Lie algebra is the direct sum of commuting compact simple and $U(1)$ subalgebras.

The proof for the equivalence of these statements as well as a more in-detail presentation of the material can be found in the aforementioned book by S. Weinberg.

A proof for the equivalence for $g_{\alpha\beta}=\delta_{\alpha\beta}$ (actually the most common form) was given by M. Gell-Mann and S. L. Glashow in Ann. Phys. (N.Y.) 15, 437 (1961)

It's because you want the kinetic part of the Yang Mills action $$ \int Tr({\bf{F^2}}) dV$$ to be positive definite. To guarantee this, the Lie algebra inner product you're using (Killing form) needs to be positive definite. This is guaranteed if the gauge group is compact and semi-simple. (I'm not sure if it's only if G is compact and semi simple though. Maybe someone else could fill in this detail).