What is an integrable system

I'll take off from the questioner's suggesting that maybe it's better to say what is a NON-integrable system is.

The Newtonian planar three body problem, for most masses, has been proven to be non-integrable.

Before Poincare, there seemed to be a kind of general hope in the air that every autonomous Hamiltonian system was integrable. One of Poincare's big claims to fame, proved within his Les Methodes Nouvelles de Mecanique Celeste, was that the planar three-body problem is not completely integrable. It is the dynamical systems equivalent to Galois' work on quintics. Specifically, Poincare proved that besides the energy, angular momentum and linear momentum there are no other ANALYTIC functions on phase space which Poisson commute with the energy. (To be more careful: any 'other' such function is a function of energy, angular momentum, and linear momentum. And his proof, or its extensions, only holds in the parameter region where one of the mass dominates the other two. It is still possible that for very special masses and angular momenta/ energies the system is integrable. No one believes this.) As best I can tell, existence of additional smooth integrals (with fractal-like level sets) is still open, at least in most cases.

Poincare's impossibitly proof is based on his discovery of what is nowadays called a "homoclinic tangle" embedded within the restricted three body problem, viewed in a rotating frame. In this tangle, the unstable and stable manifold of some point (an orbit in the non-rotating inertial frame) cross each other infinitely often, these crossing points having the point in its closure.
Roughly speaking, an additional integral would have to be constant along this complicated set. Now use the fact that if the zeros of an analytic function have an accumulation point then that function is zero to conclude that the function is zero.

Before Poincare (and I suppose since) mathematicians and in particular astronomers spent much energy searching for sequences of changes of variables which made the system "more and more integrable". Poincare realized the series defining their transformations were divergent -- hence his interest in divergent series. This divergence problem is the "small denominators problem" and getting around it by putting number theoretic conditions on frequencies appearing is at the heart of the KAM theorem.

This is, of course, a very good question. I should preface with the disclaimer that despite having worked on some aspects of integrability, I do not consider myself an expert. However I have thought about this question on and (mostly) off.

I will restrict myself to integrability in classical (i.e., hamiltonian) mechanics, since quantum integrability has to my mind a very different flavour.

The standard definition, which you can find in the wikipedia article you linked to, is that of Liouville. Given a Poisson manifold $P$ parametrising the states of a mechanical system, a hamiltonian function $H \in C^\infty(P)$ defines a vector field $\lbrace H,-\rbrace$, whose flows are the classical trajectories of the system. A function $f \in C^\infty(P)$ which Poisson-commutes with $H$ is constant along the classical trajectories and hence is called a conserved quantity. The Jacobi identity for the Poisson bracket says that if $f,g \in C^\infty(P)$ are conserved quantities so is their Poisson bracket $\lbrace f,g\rbrace$. Two conserved quantities are said to be in involution if they Poisson-commute. The system is said to be classically integrable if it admits "as many as possible" independent conserved quantities $f_1,f_2,\dots$ in involution. Independence means that the set of points of $P$ where their derivatives $df_1,df_2,\dots$ are linearly independent is dense.

I'm being purposefully vague above. If $P$ is a finite-dimensional and symplectic, hence of even dimension $2n$, then "as many as possible" means $n$. (One can include $H$ among the conserved quantities.) However there are interesting infinite-dimensional examples (e.g., KdV hierarchy and its cousins) where $P$ is only Poisson and "as many as possible" means in practice an infinite number of conserved quantities. Also it is not strictly necessary for the conserved quantities to be in involution, but one can allow the Lie subalgebra of $C^\infty(P)$ they span to be solvable but nonabelian.

Now the reason that integrability seems to be such a slippery notion is that one can argue that "locally" any reasonable hamiltonian system is integrable in this sense. The hallmark of integrability, according to the practitioners anyway, seems to be coordinate-dependent. I mean this in the sense that $P$ is not usually given abstractly as a manifold, but comes with a given coordinate chart. Integrability then requires the conserved quantities to be written as local expressions (e.g., differential polynomials,...) of the given coordinates.

The simple answer is that a $2n$-dimensional Hamiltonian system of ODE is integrable if it has $n$ (functionally) independent constants of the motion that are "in involution". (Functionally independent means none of them can be written as a function of the others. And "in involution" means that their Poisson Brackets all vanish -- a somewhat technical condition I won't define carefully (* but see below), but instead refer you to: http://en.wikipedia.org/wiki/Poisson_bracket). The simplest and the motivating example is the $n$-dimensional Harmonic Oscillator. What makes integrable systems remarkable and interesting is that one can find so-called "action angle variables" for them, in terms of which the time-evolution of any orbit becomes transparent.

For a more detailed and modern discussion you may find an expository article I wrote in the Bulletin of The AMS useful. It is called "On the Symmetries of Solitons", and you can download it as pdf here:

http://www.ams.org/journals/bull/1997-34-04/S0273-0979-97-00732-5/

It is primarily about the infinite dimensional theory of integrable systems, like SGE (the Sine-Gordon Equation), KdV (Korteweg deVries) , and NLS (non-linear Schrodinger equation), but it starts out with an exposition of the classic finite dimensional theory.

• Here is a little bit about what the Poisson bracket of two functions is that explains its meaning and why two functions with vanishing Poisson bracket are said to "Poisson commute". Recall that in Hamiltonian mechanics there is a natural non degenerate two-form $\omega = \sum_i dp_i \wedge dq_i $. This defines (by contraction with $\omega$) a bijective correspondence between vector fields and differential 1-forms. OK then -- given two functions $f$ and $g$, let $F$ and $G$ be the vector fields corresponding to the 1-forms $df$ and $dg$. Then the Poisson bracket of $f$ and $g$ is the function $h$ such that $dh$ corresponds to the vector field $[F,G]$, the usual commutator bracket of the vector fields $F$ and $G$. Thus two functions Poisson commute iff the vector fields corresponding to their differentials commute, i.e., iff the flows defined by these vector fields commute. So if a Hamiltonian vector field (on a compact $2n$-dimensional symplectic manifold $M$) is integrable, then it belongs to an $n$-dimensional family of commuting vector fields that generate a torus action on $M$. And this is where the action-angle variables come from: the level surfaces of the action variables are the torus orbits and the angle variables are the angles coordinates for the $n$ circles whose product gives a torus orbit.