Condition for the roots of $x^n+a_1x^{n-1}+a_2x^{n-2}+\ldots+a_n=0$ not all to be real

Consider $n$ reals $(\alpha_1,\dots,\alpha_n)$. By Cauchy-Schwarz, $$(1.\alpha_1+\dots+1.\alpha_n)^2\le (1^2+\dots+1^2)(\alpha_1^2+\dots+\alpha_n^2)$$ so $$\left(\sum \alpha_i\right)^2\le n\sum \alpha_i^2$$ What you proved with your second try is that for your polynomial, this inequality doesn't stand. So the hypothesis must be wrong, and not all $\alpha_i$ are real.

You were close :-)

The inequality should be in the other direction, i.e. the roots cannot be all real if $(n - 1)a_1^2 - 2na_2 < 0$. For example, the equation $x^2 - 3x + 2$ has the two real solutions $x = 1$ and $x = 2$, but $1 \cdot 3^2 - 2\cdot2\cdot2 = 1 > 0$.

Your second attempt leads to the solution.

Let $\alpha_1, \ldots, \alpha_n$ denote the roots of the polynomial (according to their multiplicity). Then $a_1 = -\sum \limits_{i} \alpha_i$ and $a_2 = \sum \limits_{i < j} \alpha_i \alpha_j$. Now if all $\alpha_i$ are real, then by Jensen's inequality

$$\left( \sum \limits_{i} \alpha_i\right)^2 = n^2 \left( \sum \limits_i\frac{1}{n} \alpha_i\right)^2 \le n^2 \sum \limits_i \frac{1}{n} \alpha_i^2 = n \sum \limits_{i} \alpha_i^2$$ holds, but this means $$(n - 1)a_1^2 - 2n a_2 = n \sum_i \alpha_i^2 - \left(\sum \limits_i \alpha_i\right)^2 \ge 0.$$