How to determine whether a function of many variables is convex or non-convex?

If the function is twice differentiable and the Hessian is positive semidefinite in the entire domain, then the function is convex. Note that the domain must be assumed to be convex too. If the Hessian has a negative eigenvalue at a point in the interior of the domain, then the function is not convex.

This is a proof that if $f:U \rightarrow \mathbb{R}$, $U$ is an open and convex set of $\mathbb{R}^n$ and f is twice differentiable, then $f''(x)$ positive definite imply $f$ convex.

Let $f''(x)(h²) \geq 0 $ $\forall x \in U, \forall h \in \mathbb{R}^n. $

If $f$ isn't convex, $\exists x_{1},x_{2} \in U , \exists \lambda $ st. $ f((1- \lambda)x_{1} + \lambda x_{2}) > (1-\lambda)f(x_{1}) + \lambda f(x_{2})$

Let $\phi:[0,1] \rightarrow \mathbb{R}, \phi(t) = f((1-t)x_{1} + tx_{2})$

Then $ \phi(\lambda) > (1- \lambda) \phi(0) + \lambda \phi(1)$

Using the Mean Value Theorem, $\exists t_{1}, t_{2}$ with $0<t_{1}<\lambda<t_{2}<1$ st.

$\phi(\lambda) - \phi(0) = \lambda \frac{d}{dt} \phi(t_{1}) $ and $\phi(1)-\phi(\lambda) = (1-\lambda) \frac{d}{dt} \phi(t_{2}) $

Observe that

$\lambda ( \phi(1) - \phi(0)) < \phi(\lambda) - \phi(0) = \lambda \frac{d}{dt} \phi(t_{1})$

and that

$ (1-\lambda) \frac{d}{dt} \phi(t_{2}) = \phi(1) - \phi(\lambda) < (1-\lambda) [ \phi(1) - \phi(0) ]$

Then $ \frac{d}{dt} \phi (t_{2}) < \phi(1) - \phi(0) < \frac{d}{dt} \phi(t_{1}).$

Again, using MVT,

$\exists t_{3} \in (t_{1}, t_{2})$ st. $\frac{d²}{dt²} \phi(t_3) < 0$

This is an absurd, since by hypothesis $\frac{d²}{dt²} \phi(t) \geq 0, \frac{d²}{dt²} \phi(t) = f''((1-t)x_{1} + tx_{2})(x_{2}-x_{1})²$

$\square$