Matrices with $A^2+B^2=2AB$

You are almost there. You've obtained that $2Y^2=[Y,X]$ and that $Y$ is nilpotent. Hence \begin{aligned} 0=2Y^3 &=Y[Y,X]\\ &=YYX-YXY\\ &=YYX-([Y,X]+XY)Y\\ &=YYX-(2Y^2+XY)Y\\ &=YYX-XYY. \end{aligned} Thus $Y^2$ commutes with $X$. Hence $X^2$ and $Y^2$ are simultaneously triangulable.

The hypothesis $\det(A+B)^2=8\det(A^2+B^2)$ can be rewritten as $\det(2X)^2 = 8\det(2(X^2+Y^2))$. Pulling out the constants, it is equivalent to $\det(X^2) = \det(X^2+Y^2)$. The result now follows because $X^2$ and $Y^2$ are simultaneously triangulable and $Y^2$ is nilpotent.

With what you've done you are almost there, you just need to dirty your hands a bit. From the fact the $Y^2$ is nilpotent, $Y$ is as well. Since $Y^2=0$ is the easy commuting case, we may assume that $Y$ has a single Jordan block of size $3$. Then by explicit computation, the fact that $X$ commutes with $Y$ up to positive powers of $Y$ forces $X$ to be simultaneously triangulisable with $Y$. Changing to a basis of simultaneous triangulisation, determinants are just products of diagonal coefficients, and what you need to prove becomes obvious.