How to prove that a Hamiltonian system is *not* Liouville integrable?

Explicitly proving non-integrability of an arbitrary Hamiltonian system is an open problem.

For some classes of Hamiltonian systems (e.g systems on a plane) is possible to prove explicitly the non-integrability of the system, using theorems of Poincare, Burns, Ziglin and Yoshida (and generalizations).

For example there is a theorem of Poincare:

For a hamiltonian of the form:

1:    $ H = \frac{p_x^2 + p_y^2}{2} + V(x,y)$

If the hamiltonian (1) can have an isolated periodic solution, then the system is not integrable (specificaly there does not exist a second integral of motion that is independent of the $H$)

For the meaning of isolated periodic solution in relation to Poincare's method see for example here and here

The theorem of Ziglin has more extensive applications:

If the Hamiltonian system (1) is integrable, and there is a monodromy matrix $\Delta$ from the monodromy group of the vertical variations equation, then any other monodromy matrix $\Delta'$ must commute with $\Delta$ or its eigenvalues must be $i, -i$

Yoshida's theorem involves hamiltonian systems with homogeneopus potentials (for example see here for a generalisation)

Related approaches involve the Painleve properties and characterization of the equations of motion (e.g here, here and here).

Furthermore there are approaches to integrability which involve Differential Galois theory (i.e Galois theory for differential equations) where one has the analogy solvability -> integrability. This approach can also unify various other approaches (e.g here and here)

User Nikos M has already given a good answer. Here we would like to mention the following Poincare theorem, which can be used to prove non-existence of integrals of motion, cf. e.g. this Phys.SE post.

Poincare theorem (1892):

• Consider an autonomous Hamiltonian system in a $2n$-dimensional symplectic manifold $({\cal M}, \{\cdot,\cdot\})$ equipped with Hamiltonian function $H:{\cal M}\to \mathbb{R}$.

• Let there be given with a periodic solution $\Gamma(t)=\Gamma(t+T)$ with start & end point $p=\Gamma(0)=\Gamma(T)\in {\cal M}$.

• Let there be given $r$ functionally independent, Poisson-commuting integrals of motion $F_1,\ldots, F_r$ in a tubular neighborhood ${\cal U}\subseteq {\cal M}$ of $\Gamma$, where $r\leq n$, and where the Hamiltonian $H|_{\cal U}$ is a function of $F_1,\ldots, F_r$ only.

• Let $\sigma: {\cal M} \times \mathbb{R} \to {\cal M}$ denote the Hamiltonian flow corresponding to the Hamiltonian vector field $X_{-H}=\{-H,\cdot\}$.

Then the monodromy map $\sigma_{\ast}(p,T): T_p{\cal M}\to T_p{\cal M}$ at the point $p$ has $r$ eigenvectors $X_{-F_1}(p), \ldots, X_{-F_r}(p),$ and $r$ co-eigenvectors $\mathrm{d}F_1(p), \ldots, \mathrm{d}F_r(p),$ all with eigenvalue $1$. In fact, the monodromy map $\sigma_{\ast}(p,T)$ has (generalized) eigenvalue 1 with multiplicity $2r$.

Sketched proof of Poincare theorem:

1. In a neighborhood of $p\in {\cal M}$, we can use the Caratheodory–Jacobi–Lie theorem to construct local Darboux coordinates $$ z^I ~=~(\varphi^1, \ldots, \varphi^r, q^1,\ldots, q^{n-r}, p_1,\ldots, p_{n-r}, F_1, \ldots, F_r ), $$ with non-zero canonical Poisson brackets $$ \{\varphi^i, F_j\} ~=~\delta^i_j, \qquad i,j\in\{1, \ldots, r\},$$ $$ \{q^a, p_b\} ~=~\delta^a_b, \qquad a,b\in\{1, \ldots, n-r\}. $$

2. The monodromy matrix $$ M^I{}_J ~=~ \frac{\partial z^I(T)}{\partial z^J(0)}, \qquad I,J\in\{1, \ldots, 2n\}, $$ at the point $p$ has $r$ co-eigenvectors $$\begin{align} \begin{bmatrix} \vec{0}^T{}_{1\times r}& \vec{0}^T{}_{1\times 2(n-r)}&\vec{e}_i^T{}_{1\times r}\end{bmatrix}_{1\times 2n} &~=~ \frac{\partial F_i(z(0))}{\partial z^I(0)} ~=~\frac{\partial F_i(z(T))}{\partial z^I(0)} \cr &~=~\frac{\partial F_i(z(T))}{\partial z^J(T)}\frac{\partial z^J(T)}{\partial z^I(0)}~=~\frac{\partial F_i(z(0))}{\partial z^J(0)} M^J{}_I,\end{align} $$ $i\in\{1, \ldots, r\}$, with eigenvalue 1.

3. Since the flow $\sigma$ is Hamiltonian, the monodromy matrix $M$ must be symplectic $$ \{z^I(T),z^J(T)\}~=~\{z^I(0),z^J(0)\}, \qquad I,J\in\{1, \ldots, 2n\}, $$ or equivalently, $$M^T\omega M~=~\omega, \qquad \omega ~=~ \begin{bmatrix} \mathbb{0}_{n\times n} & -\mathbb{1}_{n\times n}\cr \mathbb{1}_{n\times n} & \mathbb{0}_{n\times n} \end{bmatrix}_{2n\times 2n} .$$ Therefore the monodromy matrix has $r$ eigenvectors $$\begin{align} \begin{bmatrix} \vec{e}_i{~}_{r\times 1} \cr \vec{0}{~}_{2(n-r)\times 1} \cr \vec{0}{~}_{r\times 1}\end{bmatrix}_{2n\times 1} &~=~X^I_{-F_i} ~=~\{z^I(0),F_i \} ~=~\{z^I(T),F_i \} \cr &~=~\frac{\partial z^I(T)}{\partial z^J(0)} \{z^J(0),F_i \}~=~M^I{}_J \{z^J(0),F_i \} ~=~M^I{}_J X^J_{-F_i},\end{align}$$ $i\in\{1, \ldots, r\}$, with eigenvalue 1.

4. Altogether, we deduce that the monodromy matrix $M$ has the following block-triangular form $$ M ~=~ \begin{bmatrix} \mathbb{1}_{r\times r} & \ast & \ast \cr \mathbb{0}_{2(n-r)\times r} & \ast & \ast \cr \mathbb{0}_{r\times r} &\mathbb{0}_{r\times 2(n-r)}& \mathbb{1}_{r\times r}\end{bmatrix}_{2n\times 2n} .$$ Hence the characteristic polynomial $\det(M-\lambda \mathbb{1}_{2n\times 2n} )$ has generalized eigenvalue 1 with multiplicity $2r$. $\Box$

Poincare corollary: If an autonomous Hamiltonian Liouville-integrable system has a periodic solution $\Gamma(t)=\Gamma(t+T)$, then the monodromy matrix for the linearized system along $\Gamma$ can only have 1 as a (generalized) eigenvalue.

References:

1. H. Poincare, Les Methodes Nouvelles de la Mecanique Celeste, Vol. I, (1892); p. 192-198.

2. A. Chenciner, Poincare and the Three-Body Problem, (2012); p. 87. (Hat tip: Nikos M.)

3. J.J. Morales-Ruiz, Differential Galois Theory and Non-Integrability of Hamiltonian Systems, Progress in Math. 179 (1999); p. 3-4 & p. 57.