Studying the extrema of $f(x,y) = x^4 + y^4 -2(x-y)^2$

To solve the system you can add: $$\begin{cases} x^3-x+y=0 \\ y^3-y+x=0 \end{cases} \Rightarrow x^3+y^3=0 \Rightarrow x=-y \Rightarrow \\ x^3-x-x=0 \Rightarrow x(x^2-2)=0 \Rightarrow x=0;\pm \sqrt{2}.$$

The matrix $H(0,0)$ is negative semidefinite since

$$\left\langle \begin{bmatrix}-4 & 4 \\ 4 & -4\end{bmatrix}\begin{bmatrix}x \\ y\end{bmatrix}, \begin{bmatrix}x \\ y\end{bmatrix}\right\rangle = -4(x-y)^2 \le 0$$

with equality iff $x = y$.

This is a necessary condition for a local maximum, but not sufficient. Therefore, the test is inconclusive.

Indeed, $(0,0)$ is a saddle point. If we approach $(0,0)$ along $x = y$ then we have

$$f(x,x) = 2x^4 > 0$$

However, if we approach along $x = -y$, then

$$f(x,-x) = 2x^4-8x^2 < 0$$

near $(0,0)$.

as mentioned below, $(0,0)$ is a saddle point, but this function still has extrema: you forgot the (possible) extrema at $x=(-\sqrt{2},\sqrt{2})$ and $y=(\sqrt{2},-\sqrt{2})$. Inserting them in the Hessian $H$ gives a matrix with pos det. so you get two minima at $x$ and $y$.