How can I prove mathematically the reflection matrix has only the eigenvalues 1 or -1?

$M=I-2P$. Then $M^2=(I-2P)(I-2P)=I^2-4P+4P^2=I-4P+4P=I$ and $M^2x=λ^2x=Ix=x$. Then $λ^2=1$. Hence $λ=±1$

If you're reflecting in the subspace $S$ then $S$ is the $1$-eigenspace and $S^\perp$ is the $(-1)$-eigenspace. Since $S \oplus S^\perp = \mathbf{R}^n$ there are no additional possible eigenvalues.

This, to me, is the definition of reflection in the space $S$: it fixes $S$ and it sends a vector orthogonal to $S$ to its negative. To write this in terms of $P_S$ we use that $P_S$ is the identity operator on $S$ and $1 - P_S$ is the identity operator on $S^\perp$ (and they are zero on the complementary space). So the reflection map is $P_S - (I - P_S)$ (identity on $S$ + the negative identity on $S^\perp$). That gives you $-I + 2P_S$.

Now, for completeness, you can verify: if $x \in S$ (so $P_S x = x$) then $(-I + 2P_S)x = -x + 2x = x$ and if $x \in S^\perp$ (so $P_S x = 0$) then $(-I + 2P_S)x = -x$.

The matrix you gave I do not consider to be reflection in $S$. But, if my matrix has eigenvalues $-1$ and $+1$, then yours—being the negative of mine—has eigenvalues $-(-1)$ and $-(+1)$.

Let $\lambda_S\in${$0,1$} denote the eigenvalue of projection matrix $P_S$. The given relation $M=I-2P_S=f(P_S)$ where $f(P_S)$ is matrix polynomial in $P_S$. If $\lambda_M$ is the eigenvalue of $M$ then $\lambda_M=1-2\lambda_S=1-2(0)$ or $1-2(1)$ i.e. $1$ or $-1$.