If $\alpha,a,b$ are integers and $b\neq-1$, then prove that, if $\alpha$ satisfies the equation $x^2+ax+b+1=0$, $a^2+b^2$ must be composite.

If $u=\alpha$ and $v$ are the roots $$ \begin{align} x^2+ax+b+1 &=(x-u)(x-v)\\ &=x^2-(u+v)x+uv \end{align} $$ Then $a=-(u+v)$ and $b=uv-1$. Since $u=\alpha\in\mathbb{Z}$, and $a\in\mathbb{Z}$, we know that $v\in\mathbb{Z}$. Therefore, $$ \begin{align} a^2+b^2 &=(u+v)^2+(uv-1)^2\\ &=u^2+v^2+u^2v^2+1\\ &=\left(u^2+1\right)\left(v^2+1\right) \end{align} $$

Hint: $a^2-4(b+1)=a^2+b^2-(b^2+4b+4)=c^2$ for some integer $c$.

Do you know any facts about expressing primes as the sum of two squares?

Let the roots be $r,s$.

By hypothesis, at least one of $r,s$ is an integer.

By Vieta's formulas \begin{align*} r + s &= -a\\[4pt] rs &= b + 1\\[4pt] \end{align*} From the first of the above equations, since one of $r,s$ is an integer, and $a$ is an integer, $r,s$ must both be integers.

From the second of the above equations, since $b \ne -1$, it follows that $r,s$ are both nonzero. \begin{align*} \text{Then}\; a^2+b^2&=(r+s)^2 + (rs-1)^2\\[4pt] &=(r^2 + 2rs + s^2) + (r^2s^2 - 2rs + 1)\\[4pt] &=r^2s^2 + r^2 + s^2 + 1\\[4pt] &=(r^2+1)(s^2+1)\\[4pt] \end{align*} Since $r,s$ are nonzero integers, it follows that $a^2+b^2$ is composite.