How did Euler realize $x^4-4x^3+2x^2+4x+4=(x^2-(2+\alpha)x+1+\sqrt{7}+\alpha)(x^2-(2-\alpha)x+1+\sqrt{7}-\alpha)$?

Euler lived a century after Isaac Newton and Blaise Pascal, so he must have been familiar with the former's binomial theorem and the latter's triangle. Indeed, the polynomial you presented looks quite similar to the binomial expansion of $(x-1)^4$, whose coefficients are found on the fourth row of Pascal's triangle. By subtracting the two, we are left with $4x^2-8x-3$, whose roots are $1\pm\dfrac{\sqrt7}2$ which is a quarter of $\alpha$. So, $$P(x)=(x-1)^4-4\bigg[(x-1)-\dfrac{\sqrt7}2\bigg]\bigg[(x-1)+\dfrac{\sqrt7}2\bigg],$$ which, after substituting $u=(x-1)^2$, becomes $u^2-4u+7$. Then, by completing the square, we arrive at the desired result.

I like this rather old question. Here is a yet another possible way Euler could have taken:

Note that $\displaystyle x^4+ax^2+b$ can be factorized easily if $\displaystyle a^2-4b\geq 0$. If, however, $\displaystyle a^2-4b\leq 0$, then \begin{align} x^4+ax^2+b&=(x^2+\sqrt{b})^2-(x\sqrt{2\sqrt{b}-a})^2\\ &=(x^2+\sqrt{b}-x\sqrt{2\sqrt{b}-a})(x^2+\sqrt{b}+x\sqrt{2\sqrt{b}-a}). \end{align}

Now in $P(x)=x^4-4x^3+2x^2+4x+4$, use $x=y+1$, and proceed: \begin{align} P(y+1)&=y^4-4y^2+7\\ &=(y^2+\sqrt{7}-y\sqrt{2\sqrt{7}+4})(y^2+\sqrt{7}+y\sqrt{2\sqrt{7}+4}). \end{align}

substituting $y=x-1$ we arrive at Euler's result.

Once it is noticed that $$ x^4-4x^3+2x^2+4x+4=(x-1)^4-4(x-1)^2+7 $$ the square can be completed to get $$ \left((x-1)^2-2\right)^2+3=\color{#00A000}{\left((x-1)^2-2-i\sqrt3\right)}\color{#0000FF}{\left((x-1)^2-2+i\sqrt3\right)} $$ Solving for $(\alpha+i\beta)^2=2+i\sqrt3$, we get $\alpha^2-\beta^2=2$ and $2\alpha\beta=\sqrt3$. Adding the squares and taking the square root gives $\alpha^2+\beta^2=\sqrt7$. Solving for $\alpha$ and $\beta$ yields $$ \color{#00A000}{\left({\small\sqrt{\frac{2+\sqrt7}2}}+i\,{\small\sqrt{\frac{-2+\sqrt7}2}}\right)^2=2+i\sqrt3} $$ $$ \color{#0000FF}{\left({\small\sqrt{\frac{2+\sqrt7}2}}-i\,{\small\sqrt{\frac{-2+\sqrt7}2}}\right)^2=2-i\sqrt3} $$ The full factorization is $$ \overset{\underbrace{\color{#0000FF}{\left[x-1-\sqrt{\frac{2+\sqrt7}2}+i\,\sqrt{\frac{-2+\sqrt7}2}\right]}\color{#00A000}{\left[x-1-\sqrt{\frac{2+\sqrt7}2}-i\,\sqrt{\frac{-2+\sqrt7}2}\right]}}_{}} {\left[\left(x-1-\sqrt{\frac{2+\sqrt7}2}\right)^2+\frac{-2+\sqrt7}2\right]} \overset{\underbrace{\color{#00A000}{\left[x-1+\sqrt{\frac{2+\sqrt7}2}+i\,\sqrt{\frac{-2+\sqrt7}2}\right]}\color{#0000FF}{\left[x-1+\sqrt{\frac{2+\sqrt7}2}-i\,\sqrt{\frac{-2+\sqrt7}2}\right]}}_{}} {\left[\left(x-1+\sqrt{\frac{2+\sqrt7}2}\right)^2+\frac{-2+\sqrt7}2\right]} $$ This gives the factorization sought.