Finding the minimum or maximum of a bivariate function when $f_{xx}\times f_{yy}-f_{xy}^2=0$.

I'd like to mention the role of the eigenvalues of the hessian and relate this to the other answers.

At a critical point $p_0=(x_0,y_0)$ of a function $f(x,y)$ (i.e. $\nabla f(p_0)=0$) the local shape of the graph of $f$ around $p_0$ is determined by the hessian $$ \mathrm{Hess}(p_0)=\begin{bmatrix} f_{xx}(p_0) & f_{xy}(p_0) \\ f_{xy}(p_0) & f_{yy}(p_0) \end{bmatrix}.$$

This is a symmetric matrix and thus diagonalisable. The two eigenvectors of the hessian point in the direction in which the function increases or decreases the most. The eigenvalues $\lambda_1$, $\lambda_2$ tell us whether the function rises of decreases in the direction of these eigenvectors. If the eigenvalues are non-zero, we get

• $\lambda_1, \lambda_2 > 0$: a local (isolated) minimum.
• $\lambda_1, \lambda_2 < 0$: a local (isolated) maximum.
• $\lambda_1 >0, \lambda_2 < 0$ or vice versa: a saddlepoint.

These conditions can be expressed in terms of $\det \mathrm{Hess}(p_0)$, but the eigenvalue approach is more general: it also works for functions with more than two variables.

If $f_{xx}(p_0) f_{yy}(p_0)-f_{xy}^2(p_0) =0$, that is $\det \mathrm{Hess}(p_0) = 0$, then at least one eigenvalue is zero. In the case the other eigenvalue is non-zero, the extremum is not isolated and we can still determine the type.

• If the other eigenvalue is positive, there is a curve of local minima. Example: $f(x,y)=x^2$.
• If the other eigenvalue is negative, there is a curve of local maxima. Example: $f(x,y)=-x^2$.

If both eigenvalues are zero, then there is no general conclusion, as shown in the examples of smcc. You should then look at the higher order terms of the Taylor expansion of $f$ as Cesareo mentioned in his answer.